CDAP Release 6.7.1

Owned by Robin Rielley
Last updated: Jan 17, 2023 by robin

Release Date: August 11, 2022

Enhancements

CDAP-19050: Enhanced the Dataproc provisioner to avoid making unneeded Compute Engine calls depending on the configuration settings. 

CDAP-18336: For new Dataproc compute profiles, changed the default value of Master Machine Type and Worker Machine Type from n1 to n2.

Bug Fixes

CDAP-19381: Fixed an issue in CDAP that created duplicate entries in file cache map, which resulted in multiple attempts to delete the same cache file.

CDAP-19379: 

  • Fixed an issue where the Log service left empty folders, which made the mounting of Persistent Disk slow. This caused the Log service to fail to start in a timely manner.  

  • Fixed an issue that caused pipelines to take a long time to launch or get stuck. This was linked to I/O throttling that occurred on the underlying Persistent Disk. 

CDAP-19366: Fixed an issue that caused pipelines to fail when two or more pipelines were scheduled to start simultaneously on a static Dataproc cluster. This was due to a file upload race condition. 

CDAP-19353: Fixed an issue in flow control that caused Appfabric to return 5xx error code in rare scenarios instead of 429 (Too Many Requests Error) if the number of concurrently launching or running pipelines were above certain thresholds. 

CDAP-19276: Fixed an issue that resulted in an error when a compute profile was exported from the default namespace after switching from a custom namespace. 

CDAP-19216: Fixed an issue when you started a pipeline multiple times and then stopped the pipeline before it completed, which resulted in the following  UI error: Program <program name> is not running. 

CDAP-19211: Removed verbose logs from the BigQuery client libraries in pipeline logs.

PLUGIN-1256: Fixed an issue that caused the BigQuery Execute action plugin configured with an Encryption Key Name (CMEK) to fail when the SQL query contained DDL Statements.

PLUGIN-954: In the BigQuery Execute action plugin, added a property Store Results in a BigQuery Table in the UI, which hides the destination table related properties by default.

Known Issues

Replication

CDAP-19622: Upgrade for replication jobs is broken. We recommend not upgrading replication jobs to 6.7.1.

Secure Macros

CDAP-20271: Pipelines fail when they use a connection that includes a secure macro and the secure macro has JSON as the value (for example, the Service Account property).

Workarounds

Use one of the following workarounds:

(1) For existing, running pipelines, create a new secure key for the connection and escape all the quotes in the secure macro JSON. Then edit the connection to use the new secure key JSON. Note: If you use this approach, browsing and sampling in Wrangler, and other places that directly use the secure macro will start to fail.

(2) Do not use the connection with the secure macro when you run the pipeline. 

To remove the connection, follow these steps:

  1. Duplicate the deployed pipeline.

  2. In the Pipeline Studio, for each plugin that uses the connection, turn off the connection and then edit the Service Account property to include the secure macro. 

 

Created in 2020 by Google Inc.