Exporting and importing data pipelines

You can export and import data pipelines in the Pipeline Studio.

You might want to export pipelines to share them with other developers, add them to version control, or deploy them from a development environment to a test or production environment. For example, you finish developing and debugging a set of pipelines and want to deploy them to production. In this case, you export the pipelines, check them into version control, and use the Lifecycle Microservices to deploy the pipelines to production.

Exporting data pipelines

You can export pipelines from the Pipeline Studio, List Pipelines page, or Deploy Pipeline page.

Want to watch the video?

Pipeline Studio

List Pipelines page

Deploy Pipeline page

When you export a pipeline, CDAP saves the pipeline configuration to your local drive as a JSON file. The JSON includes all of the plugin configuration and pipeline configuration, including comments.

All pipelines are exported in the DRAFT state. This means that an exported pipeline does not include any triggers, schedules, or pipeline configuration that were created after the pipeline was deployed. If they are required, you need to recreate them.

Note: To export all deployed data pipelines for all namespaces, see Lifecycle Microservices, “Export All Application Details”.

To export a single data pipeline in a namespace, see Lifecycle Microservices, “Details of an Application”.

To export all pipelines in a namespace, see Lifecycle Microservices, “List Applications”.

Importing data pipelines

In the Pipeline Studio, click Import.
Upload the JSON file that you saved when you exported the pipeline.
(Optional) You might be prompted to upgrade the version of your plugins. Click Fix all. If a particular plugin is not found, follow the prompts to install it from the Hub.
(Optional) If the pipeline uses an earlier version of a plugin and if that version of the plugin exists in the current CDAP instance, you can continue using the earlier version of the plugin, or upgrade to the current version of the plugin (that matches the instance/namespace version).
(Optional) If you use a custom plugin in your pipeline and you receive a prompt in the upgrade window that the plugin was not found, you must deploy that custom plugin first, and then import the pipeline again.
The pipeline opens in the Pipeline Studio page.
Click Save to save the pipeline as Draft.
The pipeline appears under Drafts in the List Pipelines page.

Note: You cannot use CDAP Lifecycle Microservices to import pipelines.