Important: CDAP 6.1.2 is deprecated. |
This release primarily focuses on bug fixes and performance improvements. Some of the highlights include:
Improve preview performance & limits concurrent preview runs to 10 by default
Shift in polling logic to UI to avoid polling leaks in Nodejs server
Batch API usage in UI to reduce the load on backend services
Support Field Level Lineage for Streaming pipelines
Improve Field Level Lineage computation algorithm
Add support for Spark 2.4
Improves memory consumption during pipeline execution
CDAP-15579 - Added the ability for SparkCompute and SparkSink to record field lineage.
CDAP-16107 - Adds support for Spark 2.4
CDAP-13643 - Added the ability to record field lineage for streaming pipelines.
CDAP-16002 - Fixed a bug that caused errors when Wrangler's parse-as-csv with header was used when reading multiple small files.
CDAP-16526 - Fixed the BigQuery sink to properly allow certain types as clustering fields.
CDAP-16471 - Fixed a bug that would cause zombie processes when using the Remote Hadoop Provisioner
CDAP-16472 - Fixed a bug that getSchema is not working for database plugins.
CDAP-16453 - Fixed a bug that made DBSource plugin fail in preview mode
CDAP-16309 - Fixed a race condition bug that can cause failure when running Spark program
CDAP-16517 - Added an option to skip header in the files in delimited, csv, tsv and text formats.
CDAP-16525 - Added an option for database source to replace the characters in the field names.
CDAP-16308 - Reduces preview startup by 60%. Also adds limit to max concurrent preview runs (10 by default).
CDAP-16509 - Reduce memory footprint for StructureRecord which improves overall memory consumption for pipeline execution
CDAP-16339 - Introduced a new REST endpoint for fetching scheduled time for multiple programs