Introduction

From time to time, applications have need to access to small piece of sensitive data, such as pass-phases, cryptographic keys or access tokens. Those information should be stored and managed by CDAP in a secure way. At runtime, authorized applications should be able to have access to those information.

Design

API

Programatic API

New API will be introduce to CDAP to allow applications accessing sensitive data in a secure way.

// Represents the meta data about the secure data
interface SecureStoreMetaData {
  String getName();
  String getDescription();
  long getLastModifiedTime();
  Map<String, String> getProperties();
}
 
// Represents the secure data
interface SecureStoreData {
  // Returns the meta data about the secure data
  SecureStoreMetaData getMetaData();
 
  // Returns the secure data
  byte[] get();
}
 
// Provides read-only access to secure store
interface SecureStore {
  // Returns a list of available secure data in the secure store.
  List<SecureStoreMetaData> list();
 
  // Gets the secure data
  SecureStoreData get(String name);
}

// Manager interface for managing secure data
interface SecureStoreManager {
  
  // Stores the secure data
  void put(String name, byte[] data, Map<String, String> properties);
 
  // Remove the secure data
  void delete(String name);
}

REST API

New REST APIs will be provide to mirror the list, get, put and delete capability as exposed through the SecureStore and SecureStoreManager interfaces as shown above.

The REST API can only runs on HTTPS and only authorized user can access them. Permissions will also be enforced based on the following roles:

• MANAGE - Can perform put and delete operations
• READ_ONLY - Can perform list and get operations
• ALL - Can perform all operations

Permission is enforced at per key level. For example, a user can only list or get keys that he has access to.

Implementations

SDK

The SecureStore and SecureStoreManager will be implemented using the standard JKS or JCEKS keystore to store the sensitive data. The keystore can be protected with a key in the CDAP master keystore, which CDAP already requires the user to provide in order to have SSL enabled. Since program will be executed in the same JVM as the SDK process, accessing to the sensitive data directly through the proper Guice binding that binds the SecureStore interface to the actual implementation.

Cluster

Hadoop with KMS

On Hadoop cluster with KMS enabled, CDAP can provide an implementation of SecureStore and SecureStoreManager through the Hadoop KeyProvider API, assuming the cluster is configured to use KMS implementation of KeyProvider.

The CDAP master process will host the HTTPS server for providing the REST API support. The master process is also responsible for acquiring and refreshing the KMS delegation token when launch programs to be run on YARN.

Implementation of the SecureStore used by the program at runtime can be implemented by adapting calls to the underlying Hadoop KeyProvider.

Hadoop without KMS

When KMS is not available, CDAP can provide an implementation of SecureStore with an architecture similar to KMS.

Integration with Hydrator

UI

The Hydrator UI can use the REST API to get list of names and provides a dropbox/auto-complete box for the user to pick which to use when configuring the plugin.

Plugin

Plugin can access to the secure data store through the SecureStore API exposed through the context object. Plugin will get the name through the configuration and gets the actual sensitive data at runtime.