{"type":"doc","content":[{"type":"paragraph","content":[{"text":"Th","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"a1f26e55-5eb0-483d-ae8c-6ff9e4398973"}}]},{"text":"e overall installation process is:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Prepare the cluster.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Download and distribute packages.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Install CDAP services.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Start CDAP services.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Verify the installation.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"This section describes installing CDAP on Hadoop clusters that are:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Generic Apache Hadoop distributions;","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"CDH (Cloudera Distribution of Apache Hadoop) clusters ","type":"text"},{"text":"not managed","type":"text","marks":[{"type":"em"}]},{"text":" with Cloudera Manager; or","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"HDP (Hortonworks Data Platform) clusters ","type":"text"},{"text":"not managed","type":"text","marks":[{"type":"em"}]},{"text":" with Apache Ambari.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Cloudera Manager (CDH) and Apache Ambari (HDP) distributions should be installed with our other ","type":"text"},{"text":"distribution instructions","type":"text","marks":[{"type":"link","attrs":{"href":"https://cdap.atlassian.net/wiki/spaces/DOCS/pages/480477829/Installation"}}]},{"text":".","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"As CDAP depends on HDFS, YARN, HBase, ZooKeeper, and (optionally) Hive and Spark, it must be installed on cluster host(s) with full client configurations for these dependent services.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The CDAP Master Service must be co-located on a cluster host with an HDFS client, a YARN client, an HBase client, and, optionally, Hive or Spark clients.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Note that these clients are redundant if you are co-locating the CDAP Master on a cluster host (or hosts, in the case of a deployment with high availability) with actual services, such as the HDFS NameNode, the YARN resource manager, or the HBase Master.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"You can download the ","type":"text"},{"text":"Hadoop client","type":"text","marks":[{"type":"link","attrs":{"href":"http://hadoop.apache.org/releases.html#Download"}}]},{"text":" and ","type":"text"},{"text":"HBase client","type":"text","marks":[{"type":"link","attrs":{"href":"http://www.apache.org/dyn/closer.cgi/hbase/"}}]},{"text":" libraries, and then install them on the hosts running CDAP services. No Hadoop or HBase services need be running.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"All services run as the ","type":"text"},{"text":"cdap","type":"text","marks":[{"type":"code"}]},{"text":" user installed by the package manager. See “Create the cdap User” below.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"If you are installing CDAP with the intention of using ","type":"text"},{"text":"replication,","type":"text","marks":[{"type":"em"}]},{"text":" see these instructions on ","type":"text"},{"text":"CDAP Replication","type":"text","marks":[{"type":"link","attrs":{"href":"https://cdap.atlassian.net/wiki/spaces/DOCS/pages/480379650/CDAP+Replication"}}]},{"text":" ","type":"text"},{"text":"before","type":"text","marks":[{"type":"em"}]},{"text":" installing or starting CDAP.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Advanced Topics","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Enabling Security","type":"text"}]},{"type":"paragraph","content":[{"text":"Cask Data Application Platform (CDAP) supports securing clusters using perimeter security, authorization, impersonation and secure storage.","type":"text"}]},{"type":"paragraph","content":[{"text":"Network (or cluster) perimeter security limits outside access, providing a first level of security. However, perimeter security itself does not provide the safeguards of authentication, authorization and service request management that a secure Hadoop cluster provides.","type":"text"}]},{"type":"paragraph","content":[{"text":"Authorization provides a way of enforcing access control on CDAP entities.","type":"text"}]},{"type":"paragraph","content":[{"text":"Impersonation ensures that programs inside CDAP are run as configured users at the namespace level. When enabled, it guarantees that all actions on datasets, streams and other resources happen as the configured user.","type":"text"}]},{"type":"paragraph","content":[{"text":"We recommend that in order for CDAP to be secure, CDAP security should always be used in conjunction with secure Hadoop clusters. In cases where secure Hadoop is not or cannot be used, it is inherently insecure and any applications running on the cluster are effectively \"trusted”. Although there is still value in having perimeter security, authorization enforcement and secure storage in that situation, whenever possible a secure Hadoop cluster should be employed with CDAP security.","type":"text"}]},{"type":"paragraph","content":[{"text":"For instructions on enabling CDAP Security, see ","type":"text"},{"text":"CDAP Security","type":"text","marks":[{"type":"link","attrs":{"href":"https://cdap.atlassian.net/wiki/spaces/DOCS/pages/480347025/Security"}}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Enabling Kerberos","type":"text"}]},{"type":"paragraph","content":[{"text":"When running CDAP on top of a secure Hadoop cluster (using Kerberos authentication), the CDAP processes will need to obtain Kerberos credentials in order to authenticate with Hadoop, HBase, ZooKeeper, and (optionally) Hive. In this case, the setting for ","type":"text"},{"text":"hdfs.user","type":"text","marks":[{"type":"code"}]},{"text":" in ","type":"text"},{"text":"cdap-site.xml","type":"text","marks":[{"type":"code"}]},{"text":" will be ignored and the CDAP processes will be identified by the default authenticated Kerberos principal.","type":"text"}]},{"type":"paragraph","content":[{"text":"Note:","type":"text","marks":[{"type":"strong"}]},{"text":" CDAP support for secure Hadoop clusters is limited to the latest versions of CDH, HDP, and Apache BigTop; currently, Amazon EMR is not supported on secure Hadoop clusters.","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"In order to configure ","type":"text"},{"text":"CDAP for Kerberos authentication:","type":"text","marks":[{"type":"strong"}]}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Create a Kerberos principal for the user running CDAP. The principal name should be in the form ","type":"text"},{"text":"username/hostname@REALM","type":"text","marks":[{"type":"code"}]},{"text":", creating a separate principal for each host where a CDAP service will run. This prevents simultaneous login attempts from multiple hosts from being mistaken for a replay attack by the Kerberos KDC.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Generate a keytab file for each CDAP Master Kerberos principal, and place the file as ","type":"text"},{"text":"/etc/security/keytabs/cdap.keytab","type":"text","marks":[{"type":"code"}]},{"text":" on the corresponding CDAP Master host. The file should be readable only by the user running the CDAP Master service.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Edit ","type":"text"},{"text":"/etc/cdap/conf/cdap-site.xml","type":"text","marks":[{"type":"code"}]},{"text":" on each host running a CDAP service, substituting the Kerberos primary (user) for ","type":"text"},{"text":"","type":"text","marks":[{"type":"code"}]},{"text":", and your Kerberos authentication realm for ","type":"text"},{"text":"EXAMPLE.COM","type":"text","marks":[{"type":"code"}]},{"text":", when adding these two properties:","type":"text"}]},{"type":"codeBlock","content":[{"text":"\n cdap.master.kerberos.keytab\n /etc/security/keytabs/cdap.service.keytab\n\n\n\n cdap.master.kerberos.principal\n

/_HOST@EXAMPLE.COM

\n","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The ","type":"text"},{"text":"","type":"text","marks":[{"type":"code"}]},{"text":" is shown in the commands that follow as ","type":"text"},{"text":"cdap","type":"text","marks":[{"type":"code"}]},{"text":"; however, you are free to use a different appropriate name.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The ","type":"text"},{"text":"/cdap","type":"text","marks":[{"type":"code"}]},{"text":" directory needs to be owned by the ","type":"text"},{"text":"","type":"text","marks":[{"type":"code"}]},{"text":"; you can set that by running the following command as the ","type":"text"},{"text":"hdfs","type":"text","marks":[{"type":"code"}]},{"text":" user (change the ownership in the command from ","type":"text"},{"text":"cdap","type":"text","marks":[{"type":"code"}]},{"text":" to whatever is the ","type":"text"},{"text":"","type":"text","marks":[{"type":"code"}]},{"text":"):","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ |su_hdfs| && hadoop fs -mkdir -p /cdap && hadoop fs -chown cdap /cdap","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"When running on a secure HBase cluster, as the ","type":"text"},{"text":"hbase","type":"text","marks":[{"type":"code"}]},{"text":" user, issue the command:","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ echo \"grant 'cdap', 'RWCA'\" | hbase shell","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"When CDAP Master is started, it will login using the configured keytab file and principal.","type":"text"}]}]}]}]}]},{"type":"paragraph","content":[{"text":"2. In order to configure ","type":"text"},{"text":"YARN for secure Hadoop:","type":"text","marks":[{"type":"strong"}]},{"text":" the ","type":"text"},{"text":"","type":"text","marks":[{"type":"code"}]},{"text":" user must be able to launch YARN containers, either by adding it to the YARN ","type":"text"},{"text":"allowed.system.users","type":"text","marks":[{"type":"code"}]},{"text":" whitelist (preferred) or by adjusting the YARN ","type":"text"},{"text":"min.user.id","type":"text","marks":[{"type":"code"}]},{"text":" to include the ","type":"text"},{"text":"","type":"text","marks":[{"type":"code"}]},{"text":" user.","type":"text"}]},{"type":"paragraph","content":[{"text":"3. In order to configure ","type":"text"},{"text":"CDAP Explore Service for secure Hadoop:","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","marks":[{"type":"indentation","attrs":{"level":1}}],"content":[{"text":"a. To allow CDAP to act as a Hive client, it must be given ","type":"text"},{"text":"proxyuser","type":"text","marks":[{"type":"code"}]},{"text":" permissions and allowed from all hosts. For example: set the following properties in the configuration file ","type":"text"},{"text":"core-site.xml","type":"text","marks":[{"type":"code"}]},{"text":", where ","type":"text"},{"text":"cdap","type":"text","marks":[{"type":"code"}]},{"text":" is a system group to which the ","type":"text"},{"text":"cdap","type":"text","marks":[{"type":"code"}]},{"text":" user is a member:","type":"text"}]},{"type":"codeBlock","content":[{"text":"\n hadoop.proxyuser.hive.groups\n cdap,hadoop,hive\n\n\n hadoop.proxyuser.hive.hosts\n *\n","type":"text"}]},{"type":"paragraph","marks":[{"type":"indentation","attrs":{"level":1}}],"content":[{"text":"b. To execute Hive queries on a secure cluster, the cluster must be running the MapReduce ","type":"text"},{"text":"JobHistoryServer","type":"text","marks":[{"type":"code"}]},{"text":" service. Consult your distribution documentation on the proper configuration of this service.","type":"text"}]},{"type":"paragraph","marks":[{"type":"indentation","attrs":{"level":1}}],"content":[{"text":"c. To execute Hive queries on a secure cluster using the CDAP Explore Service, the Hive MetaStore service must be configured for Kerberos authentication. Consult your distribution documentation on the proper configuration of the Hive MetaStore service.","type":"text"}]},{"type":"paragraph","content":[{"text":"With all these properties set, the CDAP Explore Service will run on secure Hadoop clusters.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Enabling CDAP HA","type":"text"}]},{"type":"paragraph","content":[{"text":"In addition to having a cluster architecture that supports HA (high availability), these additional configuration steps need to be followed and completed:","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"CDAP Components","type":"text"}]},{"type":"paragraph","content":[{"text":"For each of the CDAP components listed below (Master, Router, Kafka, UI, Authentication Server), these comments apply:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Sync the configuration files (such as ","type":"text"},{"text":"cdap-site.xml","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"cdap-security.xml","type":"text","marks":[{"type":"code"}]},{"text":") on all the nodes.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"While the default ","type":"text"},{"text":"bind.address","type":"text","marks":[{"type":"em"}]},{"text":" settings (","type":"text"},{"text":"0.0.0.0","type":"text","marks":[{"type":"code"}]},{"text":", used for ","type":"text"},{"text":"app.bind.address","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"data.tx.bind.address","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"router.bind.address","type":"text","marks":[{"type":"code"}]},{"text":", and so on) can be synced across hosts, if you customize them to a particular IP address, as a result, they will be different on different hosts.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Starting services is described in ","type":"text"},{"text":"Starting CDAP Services","type":"text","marks":[{"type":"link","attrs":{"href":"https://cdap.atlassian.net/wiki/spaces/DOCS/pages/596476329/Step+4.+Starting+CDAP+Services+Manual"}}]},{"text":".","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"CDAP Master","type":"text"}]},{"type":"paragraph","content":[{"text":"The CDAP Master service primarily performs coordination tasks and can be scaled for redundancy. The instances coordinate amongst themselves, electing one as a leader at all times.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Install the ","type":"text"},{"text":"cdap-master","type":"text","marks":[{"type":"code"}]},{"text":" package on different nodes.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Ensure they are configured identically (","type":"text"},{"text":"/etc/cdap/conf/cdap-site.xml","type":"text","marks":[{"type":"code"}]},{"text":").","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Start the ","type":"text"},{"text":"cdap-master","type":"text","marks":[{"type":"code"}]},{"text":" service on each node.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"CDAP Router","type":"text"}]},{"type":"paragraph","content":[{"text":"The CDAP Router service is a stateless API endpoint for CDAP, and simply routes requests to the appropriate service. It can be scaled horizontally for performance. A load balancer, if desired, can be placed in front of the nodes running the service.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Install the ","type":"text"},{"text":"cdap-gateway","type":"text","marks":[{"type":"code"}]},{"text":" package on different nodes.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The ","type":"text"},{"text":"router.bind.address","type":"text","marks":[{"type":"code"}]},{"text":" may need to be customized on each box if it is not set to the default wildcard address (","type":"text"},{"text":"0.0.0.0","type":"text","marks":[{"type":"code"}]},{"text":").","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Start the ","type":"text"},{"text":"cdap-router","type":"text","marks":[{"type":"code"}]},{"text":" service on each node.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"CDAP Kafka","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Install the ","type":"text"},{"text":"cdap-kafka","type":"text","marks":[{"type":"code"}]},{"text":" package on different nodes.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Two properties need to be set in the ","type":"text"},{"text":"cdap-site.xml","type":"text","marks":[{"type":"code"}]},{"text":" files on each node:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The ","type":"text"},{"text":"Kafka seed brokers list","type":"text","marks":[{"type":"strong"}]},{"text":" is a comma-separated list of hosts, followed by ","type":"text"},{"text":"/${root.namespace}","type":"text","marks":[{"type":"code"}]},{"text":":","type":"text"}]},{"type":"paragraph","content":[{"text":"kafka.seed.brokers","type":"text","marks":[{"type":"code"}]},{"text":": ","type":"text"},{"text":"myhost.example.com:9092,.../${root.namespace}","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"Substitute appropriate addresses for ","type":"text"},{"text":"myhost.example.com","type":"text","marks":[{"type":"code"}]},{"text":" in the above example.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The ","type":"text"},{"text":"replication factor","type":"text","marks":[{"type":"strong"}]},{"text":" is used to replicate Kafka messages across multiple machines to prevent data loss in the event of a hardware failure:","type":"text"}]},{"type":"paragraph","content":[{"text":"kafka.server.default.replication.factor","type":"text","marks":[{"type":"code"}]},{"text":": 2","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The recommended setting is to run at least two Kafka brokers with a minimum replication factor of two; set this property to the maximum number of tolerated machine failures plus one (assuming you have that number of machines). For example, if you were running five Kafka brokers, and would tolerate two of those failing, you would set the replication factor to three. The number of Kafka brokers listed should always be equal to or greater than the replication factor.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Start the ","type":"text"},{"text":"cdap-kafka","type":"text","marks":[{"type":"code"}]},{"text":" service on each node.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"CDAP UI","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Install the ","type":"text"},{"text":"cdap-ui","type":"text","marks":[{"type":"code"}]},{"text":" package on different nodes.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Start the ","type":"text"},{"text":"cdap-ui","type":"text","marks":[{"type":"code"}]},{"text":" service on each node.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"CDAP Authentication Server","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Install the ","type":"text"},{"text":"cdap-security","type":"text","marks":[{"type":"code"}]},{"text":" package (the CDAP Authentication Server) on different nodes.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Start the ","type":"text"},{"text":"cdap-security","type":"text","marks":[{"type":"code"}]},{"text":" service on each node.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Note that when an unauthenticated request is made in a secure HA setup, a list of all running authentication endpoints will be returned in the body of the request.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Hive Execution Engines","type":"text"}]},{"type":"paragraph","content":[{"text":"CDAP Explore has support for additional execution engines such as ","type":"text"},{"text":"Apache Spark","type":"text","marks":[{"type":"link","attrs":{"href":"http://spark.apache.org/"}}]},{"text":" and ","type":"text"},{"text":"Apache Tez","type":"text","marks":[{"type":"link","attrs":{"href":"http://tez.apache.org/"}}]},{"text":". Details on specifying these engines and configuring CDAP are in the Developer Manual section on Date Exploration, ","type":"text"},{"text":"Hive Execution Engines","type":"text","marks":[{"type":"link","attrs":{"href":"https://cdap.atlassian.net/wiki/spaces/DOCS/pages/480314227/Hive+Execution+Engines"}}]},{"text":".","type":"text"}]}],"version":1}

Browser not supported