Hadoop Compatibility

Before installing the CDAP components, you must first install (or have access to) a Hadoop cluster with HBase, HDFS, Spark, YARN, and ZooKeeper. 

All CDAP components can be installed on the same boxes as your Hadoop cluster, or on separate boxes that can connect to the Hadoop services.

CDAP depends on these services being present on the cluster. There are core dependencies, which must be running for CDAP system services to operate correctly.

The host(s) running the CDAP Master service must have the HBase, HDFS, and YARN clients installed, as CDAP uses the command line clients of these for initialization and their connectivity information for external service dependencies. If Hadoop system services are also running on the same hosts as the CDAP services, they will already have these clients installed.

Core Dependencies

• HBase: For system runtime storage and queues

• HDFS: The backing file system for distributed storage

• Spark: For running Spark programs within CDAP applications

• YARN: For running system services in containers on cluster NodeManagers

• MapReduce2: For batch operations in workflows and data exploration (included with YARN)

• ZooKeeper: For service discovery and leader election

Hadoop/HBase Environment

For a Distributed CDAP cluster, version 6.2.0 and later, you must install these Hadoop components (see notes following the tables):

Note 1: Component versions shown in these tables are those that we have tested and are confident of their suitability and compatibility. Later versions of components may work, but have not necessarily been either tested or confirmed compatible.

Note 2: Certain CDAP components need to reference your Hadoop, YARN, and HBase cluster configurations by adding those configurations to their class paths.