CDAP Release 6.5.1

Release Date: November 2, 2021

Enhancements

PLUGIN-883, PLUGIN-897: Added Encryption Key Name property to the following plugins so users can encrypt any new resources created by these plugins with Customer Managed Encryption Keys (CMEK):

PLUGIN-898: Added Location property to GCS Copy and GCS Move action plugins to auto-create destination buckets if they do not exist before running the pipeline. Previously, the bucket had to exist before running the pipeline.

CDAP-18566: The File connection now browses the file system. For example, on a Hadoop cluster, the File connection now browses the HDFS file system. For CDAP Sandbox, the File connection still browses the local file system.

CDAP-18532: Added the following optional cdap-site.xml configs:

  • If a config router.block.request.enabled is true in conf, the request router should respond with a specific response (provided through config) to every user request, hence blocking all the user requests.

  • If a status code is provided using config router.block.request.status.code, the server should respond with this status code, the default value should be 503.

  • If a response message is provided using config router.block.request.message, the server should respond with this response body; otherwise the response body should be empty.

CDAP-18384: Added metrics for authorization in CDAP.

Bug Fixes

CDAP-18571: Fixed an issue where messages could not be retrieved for Kafka topics. This broke in 6.5.0 and is now fixed in 6.5.1.

CDAP-18538, CDAP-184254: Fixed an issue where you could not create a profile for an existing Dataproc cluster.

CDAP-18529: Fixed an issue that caused pipelines to fail when Transformation Pushdown was enabled and used macros as properties.

CDAP-18446: Fixed an issue that caused long running programs, like Replication, to fail within the default Hadoop delegation token timeout. Now, these tokens get renewed so that the job keeps running.

CDAP-18439: Fixed an issue in Replication that caused caused an error when you clicked Configure.

CDAP-18428: Fixed an issue that caused pipelines to fail with an Access Denied error when the pipeline had BigQuery plugins or Transformation Pushdown configuration that included a Dataset Project ID that was in a different project than the specified Project ID:

Example error in the logs (might differ depending on the plugin you are using):

1 2 3 4 5 6 7 8 9 10 11 POST https://bigquery.googleapis.com/bigquery/v2/projects/PROJECT_ID/jobs { "code" : 403, "errors" : [ { "domain" : "global", "message" : "Access Denied: Project xxxx: User does not have bigquery.jobs.create permission in project PROJECT_ID", "reason" : "accessDenied" } ], "message" : "Access Denied: Project PROJECT_ID: User does not have bigquery.jobs.create permission in project PROJECT_ID.", "status" : "PERMISSION_DENIED" }

In this example, PROJECT_ID is the Project ID that you specified in the plugin. The service account for the project specified in the plugin does not have permission to do at least one of the following:

  • Run a BigQuery job

  • Read a BigQuery dataset

  • Create a temporary bucket

  • Create a BigQuery dataset

  • Create the BigQuery table

CDAP-18423: Fixed an issue in the GCS connection that prevented browsing and parsing files stored in folders under buckets.

CDAP-18335: Fixed an issue where the UI was unusable until an error displayed in the UI was closed by clicking the x icon.

CDAP-18318: Fixed an issue where users did not need permission to restart system services, reset system service log levels, get system service statuses, etc. Now, if authorization is enabled on the cluster, users will need to have the corresponding permissions for these system services in order to access them.

CDAP-18249: Fixed an issue where the Upload window didn’t close after uploading a user-defined directive due to missing properties in the user-defined directive json.

PLUGIN-899: Fixed an issue that caused custom formats to be unusable in the GCS source and sink.