CDAP Release 6.8.1

Release Date: February 23, 2023

Features

CDAP-19729: Added support to upgrade realtime pipelines created in CDAP 6.8.0 with a Kafka Consumer Streaming source to CDAP 6.8.1. After the CDAP platform is upgraded to 6.8.1, you can use the Lifecycle microservices to upgrade these pipelines. 

Changes

CDAP-20110: When running CDAP on Kubernetes, Spark program types are now run as Kubernetes jobs instead of deployments.

CDAP-20201: CDAP now sets Spark Kubernetes connect/read timeouts based on the CDAP Kubernetes timeout settings. Previously, CDAP did not set Spark Kubernetes connection/read timeouts. Spark used its default timeout setting.a

PLUGIN-1119: The Oracle Batch Source (version 1.9.1) reads Oracle NUMBER data type with undefined precision and scale as CDAP string. Previously, CDAP read Oracle NUMBER data type with undefined precision and scale as decimal(38,0), which could have resulted in data loss. For more information, see Oracle batch source plugin (versions 1.9.1 and 1.8.5) converts Oracle NUMBER data type with undefined precision and scale to CDAP string.

Bug Fixes

CDAP-20394: Fixed an issue where the replication source plugin's event reader was not stopped by the Delta worker in case of errors, leading to leakage of the plugin's resources. 

CDAP-20392: Fixed an issue that occurred in certain upgrade scenarios, where pipelines that didn’t have the Use Connection property set, but the plugin the connection properties (such as Project ID and Service Account Type) were not displayed in the plugin UI. 

CDAP-20271: Fixed an issue that caused pipelines to fail when they used a connection that included a secure macro that had JSON as the value (for example, the Service Account JSON property).

CDAP-20257: For Oracle by Datastream replication sources, fixed an issue where the Review Assessment page would freeze for a long time when the selected or manually entered table did not exist in the source database.

CDAP-20235: For Database plugins, fixed a security issue where the database username and password were exposed in App Fabric logs.

CDAP-20199: For Oracle by Datastream replication sources, fixed an issue where the Select tables and transformations page timed out, failed to load the list of tables, and displayed the error deadline exceeded when the source database contained a large number of tables. 

CDAP-20146: Fixed an error in security-enabled instances that caused pipeline launch to fail and return token expired to the user when evaluating secure macros in provisioner properties.

CDAP-20121: For MySQL Replication sources, fixed an issue that caused replication jobs to fail during initial snapshotting when the job included a runtime argument with the Debezium property binary-handling-mode.

CDAP-20028: For Replication jobs, increased retry duration for API calls to update state/offsets in Replication jobs.

CDAP-20013: Fixed upgrade for Oracle by Datastream replication jobs. You can now upgrade Oracle by Datastream replication jobs from CDAP 6.6.0 and 6.7.x to CDAP 6.8.1.

CDAP-19622: Fixed upgrade for MySQL and SQL Server replication jobs. You can now upgrade MySQL and SQL Server replication jobs from CDAP 6.7.x to CDAP 6.8.1.

Known Issues

CDAP-20431: In CDAP versions 6.8.0 and 6.8.1, there's a known issue that might cause pipelines to fail when running pipelines on GCP Dataproc with the following error: Unsupported program type: Spark. The first time a pipeline that only contains actions runs on a newly created or upgraded instance, it succeeds. However, following pipeline runs, which include sources or sinks, might fail with this error. For updated settings, see Troubleshooting.

Created in 2020 by Google Inc.