Versions Compared

Old Version 18

changes.mady.by.user Robin Rielley

Saved on

compared with

New Version Current

changes.mady.by.user Robin Rielley

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Note

Important: CDAP 6.2.2 is deprecated.

Summary

This release introduces a number of improvements and bug fixes to CDAP. Some of the main highlights of the release are:

...

  • CDAP-16668 - Added support for creating autoscale Dataproc clusters.

  • CDAP-16682 - When the system is experiencing slowness, users now see a message saying there's a delay.

  • CDAP-16712 - Improved the scalability of the preview system when running in Kubernetes environment by separating out preview runs in their own individual pods. Preview manager pod now only responsible for handling preview REST api.

  • CDAP-17015 - Updated Preview to show number of preview runs pending before current run (if there are any runs pending). The number of pending runs is shown under the timer in the UI.

  • CDAP-17077 - Changed the auto-caching strategy in Spark pipelines to default to using disk only caching instead of memory due to common out of memory failures. Also changed the caching strategy to only cache at places that would prevent sources from being recomputed instead of the more aggressive caching previously done.

  • CDAP-17078 - Added an experimental setting to consolidate multiple pipeline branches into single operations in Spark pipelines. This can improve performance in pipelines by avoiding recomputation. This can be turned on by setting a preference or runtime argument forspark.cdap.pipeline.consolidate.stages to true.

  • CDAP-17095 - Added Distribution to AutoJoiner API to increase performance for skewed joins.

  • CDAP-17123 - Make records.updated metric available for GCS Batch Sink plugin.

  • CDAP-17130 - Added joiner distribution support to MapReduce and streaming pipelines.

  • CDAP-17179 - Added new properties Filesystem properties and Output File Prefix for GCS Sink.

  • CDAP-17182 - Enable traffic compression in runtime service.

  • CDAP-17198 - Added Runtime service to the system service statues.

  • PLUGIN-303 - Added distribution settings to Joiner plugin for increased performance in skewed joins.

  • PLUGIN-386 - Added support for BigQuery Views and Materialized Views to Wrangler.

...