CDAP Setup (Replication)

CDAP requires that you provide an extension that will perform HBase-related DDL operations on both clusters instead of only on a single cluster. To create the extension, you must implement the HBaseDDLExecutor class. Details on implementing this class, a sample implementation, and example files are available in the Appendix: HBaseDDLExecutor.

CDAP must have invalid transaction list pruning disabled, as this cannot be used with replication.

To deploy your extension (once compiled and packaged as a JAR file, such as my-extension.jar), run these steps on both your master and slave clusters. These steps assume <cdap-home> is /opt/cdap:

1. Create an extension directory, such as:

   $ mkdir -p /opt/cdap/master/ext/hbase/repl

2. Copy your JAR to the directory:

   $ cp my-extension.jar /opt/cdap/master/ext/hbase/repl/

3. Modify cdap-site.xml to use your implementation of HBaseDDLExecutor:

   <property> <name>hbase.ddlexecutor.extension.dir</name> <value>/opt/cdap/master/ext/hbase</value> </property>

4. Modify cdap-site.xml with any properties required by your executor. Any property prefixed with cdap.hbase.spi.hbase. will be available through the HBaseDDLExecutorContext object passed into your executor's initialize method:

   <property> <name>cdap.hbase.spi.hbase.zookeeper.quorum</name> <value>slave.example.com:2181/cdap</value> </property> <property> <name>cdap.hbase.spi.hbase.zookeeper.session.timeout</name> <value>60000</value> </property> <property> <name>cdap.hbase.spi.hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>cdap.hbase.spi.hbase.bulkload.staging.dir</name> <value>/tmp/hbase-staging</value> </property> <property> <name>cdap.hbase.spi.hbase.replication</name> <value>true</value> </property>

5. Modify cdap-site.xml to disable invalid transaction list pruning, as it cannot be used with replication:

   <property> <name>data.tx.prune.enable</name> <value>false</value> <description> Enable invalid transaction list pruning </description> </property>

6. Before starting CDAP on the master cluster, run a command on the slave cluster to load the HBase coprocessors required by CDAP onto the slave's HDFS:

   [slave] $ cdap setup coprocessors

7. Start CDAP on the master cluster:

   [master] $ cdap master start