CDAP Components and Functional Responsibilities

Infrastructure components used by Cask Data Application Platform (CDAP)

Following are the underlying infrastructure components used by CDAP and/or CDAP Applications running in CDAP.  The components presented below are in no priority order. 

• HDFS

• HBase

• Hive

• Kafka

• YARN and

• Zookeeper

• KMS

• Sentry ???
  
  

Functional use of infrastructure components

This section provides information about how and for what the components underneath are used. 

HDFS

• CDAP Stream

• Apache Tephra WAL

• Deployed Application Artifact and Dataset Artifact

• Aggregated Logs

• CDAP Fileset Dataset

• YARN distributed cache 

• Coprocessor jars 

HBase

• CDAP System data/metadata (ex: Preferences, Application, Namespace, Artifact…)

• Metrics Cube

• Lineage

• Workflow Statistics

• Run Record and Statistics

• Checkpoint information

• CDAP Table Dataset

Kafka

• Logs

• Metrics

• Audit Logs (Will be moved to HBase in 4.0)

• Metadata updates (Will be moved to HBase in 4.0)

• Notifications (Will be moved to HBase in 4.x)

YARN

• System Services

• User applications

Zookeeper

• Routing Tables

• Coordination

• Secret keys 

  • Auth keys

Hive

• Dataset integration 

  • Schema

  • Properties

  • Serde

KMS

• User Secrets (Ex: Password, access tokens etc..)