CDAP Release 6.1.2

Summary

This release primarily focuses on bug fixes and performance improvements. Some of the highlights include:

Performance improvements

• Improve preview performance & limits concurrent preview runs to 10 by default

• Shift in polling logic to UI to avoid polling leaks in Nodejs server

• Batch API usage in UI to reduce the load on backend services

Pipeline and Plugin fixes

• Support Field Level Lineage for Streaming pipelines

• Improve Field Level Lineage computation algorithm

• Add support for Spark 2.4

• Improves memory consumption during pipeline execution

New Features

• CDAP-15579 - Added the ability for SparkCompute and SparkSink to record field lineage.

• CDAP-16107 - Adds support for Spark 2.4

• CDAP-13643 - Added the ability to record field lineage for streaming pipelines.

Bug Fixes

• CDAP-16002 - Fixed a bug that caused errors when Wrangler's parse-as-csv with header was used when reading multiple small files.

• CDAP-16526 - Fixed the BigQuery sink to properly allow certain types as clustering fields.

• CDAP-16471 - Fixed a bug that would cause zombie processes when using the Remote Hadoop Provisioner

• CDAP-16472 - Fixed a bug that getSchema is not working for database plugins.

• CDAP-16453 - Fixed a bug that made DBSource plugin fail in preview mode

• CDAP-16309 - Fixed a race condition bug that can cause failure when running Spark program

Improvements

• CDAP-16517 - Added an option to skip header in the files in delimited, csv, tsv and text formats.

• CDAP-16525 - Added an option for database source to replace the characters in the field names.

• CDAP-16308 - Reduces preview startup by 60%. Also adds limit to max concurrent preview runs (10 by default).

• CDAP-16509 - Reduce memory footprint for StructureRecord which improves overall memory consumption for pipeline execution

• CDAP-16339 - Introduced a new REST endpoint for fetching scheduled time for multiple programs