Secure Storage

Owned by Robin Rielley
Last updated: Dec 09, 2020
2 min read

Applications can need controlled access to sensitive data such as passphrases, cryptographic keys, access tokens, and passwords. This data is usually small in size, but needs to be stored and managed in a secure manner. Secure Storage allows users to store such sensitive information in a secure and encrypted manner. Data is encrypted upon submission to CDAP (via Microservices or programmatic APIs) and is decrypted upon retrieval.

Secure Storage Format

An entry in secure storage consists of:

  • Key: An alias for the entry, also referred to as a secure key. Data is stored against the provided key and can be retrieved using the same key. Key must be of the Alphanumeric Character Set, contain only lowercase characters, and should start with a letter.

  • Data: The data which is to be stored in a secure and encrypted manner. This could be a passphrase, cryptographic key, access token, or any other data that needs to be stored securely.

  • Description: A description for the secure store entry.

  • Properties: A string map of properties for the secure storage entry. A creationTime property is added for all secure store entries by default. Optionally, you can add additional properties (key-value pairs) to describe the secure storage entries.

CDAP provides two different implementations of secure storage, depending on the runtime:

  • File-back secure storage (CDAP Sandbox and in-memory CDAP)

  • Hadoop Key Management Server-backed Secure Storage (Distributed CDAP)

File-backed Secure Storage

File-backed secure storage is available for use with in-memory CDAP (unit-test) and CDAP Sandbox modes. It uses the Sun JCEKS implementation for storing secure keys. This implementation is not available in Distributed CDAP because it stores the secure data in the local file system, and thus is not available on all nodes of a distributed cluster.

To use the file-backed secure storage mode, set these properties:

  • In cdap-site.xml, set security.store.provider to file:

    <property> <name>security.store.provider</name> <value>file</value> <description> Backend provider for the secure store </description> </property>

  • To protect the secure storage file, in cdap-security.xml, set security.store.file.password to a password:

    <property> <name>security.store.file.password</name> <value>your password</value> <description> Password to access the key store </description> </property>

  • You can also configure the path and filename of the backing file in cdap-site.xml by adding these optional settings:

    <property> <name>security.store.file.path</name> <value>${local.data.dir}/store</value> <description> Location of the encrypted file which holds the secure store entries </description> </property> <property> <name>security.store.file.name</name> <value>securestore</value> <description> Name of the secure store file </description> </property>

Hadoop Key Management Server-backed Secure Storage

Hadoop KMS (Key Management Server)-backed secure storage is available for use with Distributed CDAP.

To use this mode, set this property:

  • In cdap-site.xml, set security.store.provider to kms:

For additional information on integration with Hadoop KMS, see Apache Hadoop Key Management Server (KMS).

Accessing the Secure Storage

The Secure Storage Microservices has endpoints for the management and creation, retrieval, and deletion of secure keys.

Created in 2020 by Google Inc.