Disabling Spark Auto-caching

In some cases, you might want to disable Spark auto-caching at the Engine config level to improve pipeline performance.

For example, it is highly recommended that you disable Spark auto-caching at the Engine config level before using the Data Cacher plugin in pipelines.

To disable Spark auto-caching:

1. In Pipeline Studio, click Configure.

2. Click Engine config > Show Custom Config.

3. In the Name field, enter spark.cdap.pipeline.autocache.enable and enter false as the Value.

   Open

   

    

4. Click Save.