Disabling Spark Auto-caching

In some cases, you might want to disable Spark auto-caching at the Engine config level to improve pipeline performance.

For example, it is highly recommended that you disable Spark auto-caching at the Engine config level before using the Data Cacher plugin in pipelines.

To disable Spark auto-caching:

  1. In Pipeline Studio, click Configure.

  2. Click Engine config > Show Custom Config.

  3. In the Name field, enter spark.cdap.pipeline.autocache.enable and enter false as the Value.

     

  4. Click Save.

Created in 2020 by Google Inc.