...
These steps will upgrade from CDAP 4.2.x to 6.25.0. If you are on an earlier version of CDAP, please follow the upgrade instructions for the earlier versions and upgrade first to 4.2.x before proceeding.
...
Upgrading between patch versions of CDAP refers to upgrading from one 6.25.x version to a higher 6.25.x version.
Upgrading CDAP Major/Minor Release Versions
Upgrading between major/minor versions of CDAP refers to upgrading from a 4.2.x version to 6.25.x. Upgrades between multiple major/minor versions must be done consecutively, and a version cannot be skipped unless otherwise noted.
...
Stop all services, and other programs in all your applications.
Stop all CDAP processes:
Code Block $ for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i stop ; done
Update the CDAP repository definition by running either of these methods:
On RPM using Yum:
Code Block $ sudo curl -o /etc/yum.repos.d/cask.repo https://repository.cdap.io/centos/6/x86_64/cdap/6.2/cask.repo
This will create the file
/etc/yum.repos.d/cask.repowith:Code Block [cask] name=Cask Packages baseurl=https://repository.cdap.io/centos/6/x86_64/cdap/6.2 enabled=1 gpgcheck=1
Add the Cask Public GPG Key to your repository:
Code Block $ sudo rpm --import https://repository.cdap.io/centos/6/x86_64/cdap/6.2/pubkey.gpg
Update your Yum cache:
Code Block $ sudo yum makecache
On Debian using APT:
Code Block $ sudo curl -o /etc/apt/sources.list.d/cask.list https://repository.cdap.io/ubuntu/precise/amd64/cdap/6.2/cask.list
This will create the file
/etc/apt/sources.list.d/cask.listwith:Code Block deb [ arch=amd64 ] https://repository.cdap.io/ubuntu/precise/amd64/cdap/6.2 precise cdap
Add the Cask Public GPG Key to your repository:
Code Block $ curl -s https://repository.cdap.io/ubuntu/precise/amd64/cdap/6.2/pubkey.gpg | sudo apt-key add -
Update your APT-cache:
Code Block $ sudo apt-get update
Update the CDAP packages by running either of these methods:
On RPM using Yum:
Code Block $ sudo yum upgrade 'cdap*'
On Debian using APT:
Code Block $ sudo apt-get install --only-upgrade '^cdap.*'
Run the upgrade tool, as the user that runs CDAP Master (the CDAP user, indicated by
<cdap-user>):Code Block $ sudo -u <cdap-user> /opt/cdap/master/bin/cdap run io.cdap.cdap.data.tools.UpgradeTool upgrade
Note that once you have upgraded an instance of CDAP, you cannot reverse the process; down-grades to a previous version are not possible. Also, note that authorization is disabled in the UpgradeTool so that the
cdapuser can upgrade all users' data.The UpgradeTool will produce output similar to the following, prompting you to continue with the upgrade:
Code Block UpgradeTool - version 6.25.0-<build timestamp>. upgrade - Upgrades CDAP to 6.25.0 The upgrade tool upgrades the following: 1. User Datasets - Upgrades the coprocessor jars for tables - Migrates the metadata for PartitionedFileSets 2. System Datasets 3. UsageRegistry Dataset Type Note: Once you run the upgrade tool you cannot rollback to the previous version. Do you want to continue (y/n) y Starting upgrade ...
You can run the tool in a non-interactive fashion by using the
forceflag, in which case it will run unattended and not prompt for continuing:Code Block $ sudo -u <cdap-user> /opt/cdap/master/bin/cdap run io.cdap.cdap.data.tools.UpgradeTool upgrade force
Restart the CDAP processes:
Code Block $ for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i start ; done
To upgrade existing pipeline applications created using the 4.2.x versions of system artifacts, there are separate instructions on doing so.
Once CDAP has restarted, you can check the status of the upgrade using the Monitor Microservices:
Code Block $ curl -w"\n" -X GET "http://<cdap-host>:11015/v3/system/upgrade/status"
Returning:
Code Block {"metadata":true }
Upgrading Hadoop
These tables list different versions of CDAP and the Hadoop distributions for which they are supported. If your particular distribution is not listed here, you can determine its components and from that determine which version of CDAP may be compatible. Our blog lists the different components of the common Hadoop distributions.
...
Supported Cloudera Manager (CM) and Cloudera Distribution of Apache Hadoop (CDH) Distributions
...
CM Version
...
CDH Version
...
CDAP Parcel / CSD Version
...
5.10
...
5.9.x through 5.10.x
...
6.2.x
...
5.10
...
5.8.x
...
3.5.x through 6.2.x
...
5.10
...
5.7.x
...
3.4.x through 6.2.x
...
5.10
...
5.5.x through 5.6.x
...
3.3.x through 6.2.x
...
5.10
...
5.4.x
...
3.1.x through 6.2.x
...
5.10
...
no greater than 5.3.x
...
3.0.x through 6.2.x
...
...
5.9
...
5.9.x
...
6.2.x
...
5.9
...
5.8.x
...
3.5.x through 6.2.x
...
5.9
...
5.7.x
...
3.4.x through 6.2.x
...
5.9
...
5.5.x through 5.6.x
...
3.3.x through 6.2.x
...
5.9
...
5.4.x
...
3.1.x through 6.2.x
...
5.9
...
no greater than 5.3.x
...
3.0.x through 6.2.x
...
...
5.8
...
5.8.x
...
3.5.x through 6.2.x
...
5.8
...
5.7.x
...
3.4.x through 6.2.x
...
5.8
...
5.5.x through 5.6.x
...
3.3.x through 6.2.x
...
5.8
...
5.4.x
...
3.1.x through 6.2.x
...
5.8
...
no greater than 5.3.x
...
3.0.x through 6.2.x
...
...
5.7
...
5.7.x
...
3.4.x through 6.2.x
...
5.7
...
5.5.x through 5.6.x
...
3.3.x through 6.2.x
...
5.7
...
5.4.x
...
3.1.x through 6.2.x
...
5.7
...
no greater than 5.3.x
...
3.0.x through 6.2.x
...
...
5.6
...
5.5.x through 5.6.x
...
3.3.x through 3.6.x
...
5.6
...
5.4.x
...
3.1.x through 3.6.x
...
5.6
...
no greater than 5.3.x
...
3.0.x through 3.6.x
...
...
5.5
...
5.5.x
...
3.3.x through 3.6.x
...
5.5
...
5.4.x
...
3.1.x through 3.6.x
...
5.5
...
no greater than 5.3.x
...
3.0.x through 3.6.x
...
...
5.4
...
5.4.x
...
3.1.x through 3.6.x
...
5.4
...
no greater than 5.3.x
...
3.0.x through 3.6.x
...
...
5.3
...
no greater than 5.3.x
...
3.0.x through 3.1.x
...
5.2
...
no greater than 5.2.x
...
3.0.x through 3.1.x
...
5.1
...
no greater than 5.1.x
...
Not supported
...
Supported Hortonworks Data Platform (HDP) Distributions
...
CDAP Series or Release
...
Hadoop Distributions
...
CDAP 4.1.1, 4.2.x
...
HDP 2.0 through HDP 2.6
...
CDAP 4.1.0
...
HDP 2.0 through HDP 2.5
...
CDAP 4.0.x
...
HDP 2.0 through HDP 2.5
...
CDAP 3.6.x
...
HDP 2.0 through HDP 2.4
...
CDAP 3.5.x
...
HDP 2.0 through HDP 2.4
...
CDAP 3.4.x
...
HDP 2.0 through HDP 2.4
...
CDAP 3.3.x
...
HDP 2.0 through HDP 2.3
...
CDAP 3.2.x
...
HDP 2.0 through HDP 2.3
...
CDAP 3.1.x
...
HDP 2.0 through HDP 2.2
...
CDAP 3.0.x
...
To upgrade Hadoop, contact your remote Hadoop provider.
Upgrade Steps
These steps cover what to do when upgrading the version of Hadoop of an existing CDAP installation. As the different versions of Hadoop can use different versions of HBase, upgrading from one version to the next can require that the HBase coprocessors be upgraded to the correct version. The steps below will, if required, update the coprocessors appropriately.
...
Upgrade CDAP to a version that will support the new Hadoop version, following the usual CDAP upgrade procedure for packages.
After upgrading CDAP, start CDAP and check that it is working correctly.
Stop all CDAP applications and services:
Code Block $ for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i stop ; done
Disable all CDAP tables; from an HBase shell, run the command:
Code Block > disable_all 'cdap.*'
Upgrade to the new version of Hadoop.
Run the Post-Hadoop Upgrade Tasks—to upgrade CDAP for the new version of Hadoop—by running the CDAP Upgrade Tool, as the user that runs CDAP Master (the CDAP user, indicated by
<cdap-user>):Code Block $ sudo -u <cdap-user> /opt/cdap/master/bin/cdap run io.cdap.cdap.data.tools.UpgradeTool upgrade_hbase
Enable all CDAP tables; from an HBase shell, run this command:
Code Block > enable_all 'cdap.*'
Restart CDAP:
Code Block $ for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i start ; done