CDAP Release 6.11.0

Release date: Mar 17, 2025

Features

CDAP-21058: When a pipeline run fails, you can now retrieve detailed error information in the UI. CDAP classifies pipeline errors by category, reason, and message. This classification speeds up resolution and reduces the need to examine complex logs.

CDAP-21096: Improved scalability and availability of App Fabric service by splitting it into stateless App Fabric server and stateful App Fabric processor. App Fabric server will be responsible for handling API requests and Appfabric processor will be responsible for running messaging subscriber services and other singleton services.

CDAP-18796: Added support for using Cloud Spanner as metadata storage.

NOTE: Cloud spanner has strict limits of 10MB of data / 2,621,440 Unicode characters (for a STRING cell). As a workaround, data compression can be enabled for such columns that may exceed the size limit using the config property data.storage.properties.gcp-spanner.compression.config.

NOTE: 6.11 onwards there will be a soft limit of 2MB on pipeline json size when using Cloud Spanner storage extension. Deployments for pipelines with sizes greater than 2MB might fail.

CDAP-21079: Added support for using Cloud Spanner as message storage provider, to improve scalability and availability of message processing in CDAP services.

Changes

CDAP-21072: The maximum concurrent runs limit for triggers is displayed in the console.

CDAP-21027: Hadoop version for CDAP has been upgraded to 3.3.6.

CDAP-14950: The ProgramHeartbeatTable now gets cleaned up periodically.

PLUGIN-1438: Added support for destination table write preference in the BigQuery Execute plugin.

CDAP-21133: Dataproc 2.2 is the default version starting 6.11.0 for when dataproc provisioner is used.

Fixes

PLUGIN-1843: Fixed a null pointer exception in the BigQuery multi-sink plugin when used without a reference name.

CDAP-21075: Fixed Joiner plugin failures observed on Dataproc 2.2 instances.

PLUGIN-1742: Fixed an issue that prevented pipelines from accepting empty input in Amazon S3 and Google Cloud Storage batch source plugins.

CDAP-21069: Fixed an issue in CDAP instances where internal authentication is enabled and pipeline details API returned an incorrect author name.

Deprecated

• The ability to list applications without providing pageSize is deprecated in 6.11.0.