When upgrading an existing CDAP installation from a previous version, you will need run an upgrade tool to make sure the CDAP table definitions in HBase are up-to-date.
To upgrade existing pipeline applications created using the 4.2.x versions of system artifacts, there are separate instructions on doing so. Run these after upgrading CDAP.
These steps will upgrade from CDAP 4.2.x to 6.5.0. If you are on an earlier version of CDAP, please follow the upgrade instructions for the earlier versions and upgrade first to 4.2.x before proceeding.
Upgrading between patch versions of CDAP refers to upgrading from one 6.5.x version to a higher 6.5.x version.
Upgrading between major/minor versions of CDAP refers to upgrading from a 4.2.x version to 6.5.x. Upgrades between multiple major/minor versions must be done consecutively, and a version cannot be skipped unless otherwise noted.
These steps will stop CDAP, update the installation, run an upgrade tool for the table definitions, and then restart CDAP:
Stop all services, and other programs in all your applications.
Stop all CDAP processes:
$ for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i stop ; done |
Update the CDAP repository definition by running either of these methods:
On RPM using Yum:
$ sudo curl -o /etc/yum.repos.d/cask.repo https://repository.cdap.io/centos/6/x86_64/cdap/6.2/cask.repo |
This will create the file /etc/yum.repos.d/cask.repo
with:
[cask] name=Cask Packages baseurl=https://repository.cdap.io/centos/6/x86_64/cdap/6.2 enabled=1 gpgcheck=1 |
Add the Cask Public GPG Key to your repository:
$ sudo rpm --import https://repository.cdap.io/centos/6/x86_64/cdap/6.2/pubkey.gpg |
Update your Yum cache:
$ sudo yum makecache |
On Debian using APT:
$ sudo curl -o /etc/apt/sources.list.d/cask.list https://repository.cdap.io/ubuntu/precise/amd64/cdap/6.2/cask.list |
This will create the file /etc/apt/sources.list.d/cask.list
with:
deb [ arch=amd64 ] https://repository.cdap.io/ubuntu/precise/amd64/cdap/6.2 precise cdap |
Add the Cask Public GPG Key to your repository:
$ curl -s https://repository.cdap.io/ubuntu/precise/amd64/cdap/6.2/pubkey.gpg | sudo apt-key add - |
Update your APT-cache:
$ sudo apt-get update |
Update the CDAP packages by running either of these methods:
On RPM using Yum:
$ sudo yum upgrade 'cdap*' |
On Debian using APT:
$ sudo apt-get install --only-upgrade '^cdap.*' |
Run the upgrade tool, as the user that runs CDAP Master (the CDAP user, indicated by <cdap-user>
):
$ sudo -u <cdap-user> /opt/cdap/master/bin/cdap run io.cdap.cdap.data.tools.UpgradeTool upgrade |
Note that once you have upgraded an instance of CDAP, you cannot reverse the process; down-grades to a previous version are not possible. Also, note that authorization is disabled in the UpgradeTool so that the cdap
user can upgrade all users' data.
The UpgradeTool will produce output similar to the following, prompting you to continue with the upgrade:
UpgradeTool - version 6.5.0-<build timestamp>. upgrade - Upgrades CDAP to 6.5.0 The upgrade tool upgrades the following: 1. User Datasets - Upgrades the coprocessor jars for tables - Migrates the metadata for PartitionedFileSets 2. System Datasets 3. UsageRegistry Dataset Type Note: Once you run the upgrade tool you cannot rollback to the previous version. Do you want to continue (y/n) y Starting upgrade ... |
You can run the tool in a non-interactive fashion by using the force
flag, in which case it will run unattended and not prompt for continuing:
$ sudo -u <cdap-user> /opt/cdap/master/bin/cdap run io.cdap.cdap.data.tools.UpgradeTool upgrade force |
Restart the CDAP processes:
$ for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i start ; done |
To upgrade existing pipeline applications created using the 4.2.x versions of system artifacts, there are separate instructions on doing so.
Once CDAP has restarted, you can check the status of the upgrade using the Monitor Microservices:
$ curl -w"\n" -X GET "http://<cdap-host>:11015/v3/system/upgrade/status" |
Returning:
{"metadata":true } |
To upgrade Hadoop, contact your remote Hadoop provider.
These steps cover what to do when upgrading the version of Hadoop of an existing CDAP installation. As the different versions of Hadoop can use different versions of HBase, upgrading from one version to the next can require that the HBase coprocessors be upgraded to the correct version. The steps below will, if required, update the coprocessors appropriately.
It is important to perform these steps as described, otherwise the coprocessors may not get upgraded correctly and HBase regionservers may crash.
Upgrade CDAP to a version that will support the new Hadoop version, following the usual CDAP upgrade procedure for packages.
After upgrading CDAP, start CDAP and check that it is working correctly.
Stop all CDAP applications and services:
$ for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i stop ; done |
Disable all CDAP tables; from an HBase shell, run the command:
> disable_all 'cdap.*' |
Upgrade to the new version of Hadoop.
Run the Post-Hadoop Upgrade Tasks—to upgrade CDAP for the new version of Hadoop—by running the CDAP Upgrade Tool, as the user that runs CDAP Master (the CDAP user, indicated by <cdap-user>
):
$ sudo -u <cdap-user> /opt/cdap/master/bin/cdap run io.cdap.cdap.data.tools.UpgradeTool upgrade_hbase |
Enable all CDAP tables; from an HBase shell, run this command:
> enable_all 'cdap.*' |
Restart CDAP:
$ for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i start ; done |