Editing a deployed pipeline

Owned by robin
Last updated: Apr 13, 2023Version comment
6 min read

Starting in CDAP 6.9.0, you can edit a deployed pipeline. When you edit a deployed pipeline, CDAP creates a new version of the pipeline with the same name and marks it as the latest version. This lets you develop pipelines iteratively rather than duplicating pipelines, which creates a new pipeline with a different name.

Note: You can edit deployed batch pipelines. You cannot edit deployed streaming pipelines or deployed replication jobs.

When you edit a pipeline and deploy it, CDAP retains all triggers, pipeline configuration, runtime arguments, metadata, comments, and the schedule added to the older pipeline version.

When you edit a deployed pipeline, you can:

  • Incrementally add features to a deployed pipeline instead of duplicating it. For example, you can add plugins, remove plugins, or modify plugin properties in a pipeline and then deploy it again. Editing a pipeline creates a new version of the same pipeline. Duplicating a pipeline creates a new pipeline with a different name.

  • Maintain a full history of all versions of the deployed pipeline created by the edit functionality. 

  • View and restore old versions of a deployed pipeline from the History list.

  • Edit the entire specification of the pipeline including its structure, configuration, metadata, pipeline preferences, and comments. You can import edited JSON for a deployed pipeline.

Note: Starting in CDAP 6.9.0, you can also edit deployed pipelines using the Lifecycle Microservices. 

To edit a deployed pipeline, follow these steps:

  1. From the Pipeline List page, click Deployed.Locate the pipeline you want to edit.

  2. Click the three dots and click Edit.


    A new draft version of the pipeline (with the same name) appears in the Pipeline Studio canvas and shows the draft pipeline status.

  3. Make changes to the pipeline, such as adding plugins or editing plugin properties.

  4. To continue editing the pipeline later, click Save.

  5. When you finish editing the pipeline, click Deploy.
    The Enter Change Summary window appears.

  6. Enter a description of the changes you made to the pipeline:

  7. Click Deploy.
    The pipeline appears on the Deployed Pipeline page.

  8. Optional. Edit any existing triggers or add new triggers.

  9. Optional. Edit the pipeline configuration and runtime arguments.
    Notice that CDAP auto-generates runtime arguments:

  10. Optional. Edit or add a Schedule.

Note: You can also edit a deployed pipeline from the Deployed Pipeline page. Click Actions > Edit.

Viewing deployed pipeline version history

When you view the pipeline version history, you can view the pipeline or restore an older version of the pipeline to the latest version. The version history also includes version change summaries.

To view the version history for an edited pipeline, follow these steps:

  1. From the Pipeline List page, click Deployed.

  2. Click the pipeline for which you want to view version history. The pipeline appears on the Deployed Pipeline page.

  3. Click History.


    The version history window lists each version of the edited pipeline, sorted from latest to oldest.

  4. Optional. Click View to view the pipeline on the Deployed Pipeline page.

  5. Optional. To restore an older version of the pipeline to the latest version, click Restore. For more information, see “Restoring an older version of a deployed pipeline”.

Exporting an edited version of a deployed pipeline

You can export the latest version of the pipeline and older versions of the pipeline. CDAP saves the pipeline configuration to your local drive as a JSON file.

You might want to export pipelines to share them with other developers, add them to version control, or deploy them from a development environment to a test or production environment. For example, you finish developing and debugging a set of pipelines and want to deploy them to production. In this case, you export the pipelines, check them into version control, and use the Lifecycle Microservices to deploy the pipelines to production.

Exporting the latest version of the deployed pipeline

On the Pipeline List page, CDAP lists only the latest version of the pipeline. When you export the pipeline, the JSON is based on the latest version. 

To export the latest version of the pipeline, follow these steps:

  1. On the Pipeline List page, click Deployed.

  2. Click the pipeline you want to export.

  3. Click the three dots and click Export.
    CDAP saves the pipeline configuration to your local drive as a JSON file. 

Exporting an older version of the deployed pipeline

To export older versions of the pipeline, follow these steps:

  1. From the Pipeline List page, click Deployed.

  2. Click the latest version of the pipeline.The pipeline appears on the Deployed pipeline page.

  3. Click History.

  4. Locate the version of the pipeline you want to export.

  5. Click View.

  6. Click Actions > Export.

Importing an edited version of a deployed pipeline

All pipelines are exported in DRAFT mode. Edited versions of a deployed pipeline are also imported in DRAFT mode.

For more information, see Importing data pipelines.

Deleting an edited deployed pipeline

When you delete the latest version of a deployed pipeline, all versions of the deployed pipeline are deleted. Draft versions are not deleted.

The draft version of the pipeline will have an Orphaned status:

To resolve this status, deploy the pipeline. CDAP creates a new pipeline, which is the latest version. 

Restoring an older version of a deployed pipeline

When you restore an older version of a pipeline, CDAP creates a new pipeline, which is the latest version.

To restore an older version of a pipeline, follow these steps:

  1. From the Pipeline List page, click Deployed.

  2. Click the deployed pipeline for which you want to restore an older version.
    The pipeline appears on the Deployed Pipeline page.

  3. Click History.

  4. Find the pipeline version you want to restore and click Restore.

    CDAP creates a new version of the pipeline and opens it on the Deployed Pipeline page.

    The new pipeline is the latest version.

  5. Optional. On the Deployed Pipeline page, you can modify triggers, pipeline configuration, and runtime arguments. You can also click Actions > Edit and make changes to the pipeline structure and then deploy it.

Edited deployed pipeline status

Pipeline List Draft page statuses

On the Pipeline List pages, edited deployed pipelines can have the following are the statuses:

  • In-Progress. You are editing a pipeline version in DRAFT mode and click Save in the pipeline canvas. This status appears on the Pipeline List page:

  • Orphaned. The latest pipeline was deleted. Pipeline drafts are marked as Orphaned. For example, a teammate deletes the pipeline with the endpoint  DELETE /v3/namespaces/<namespace-id>/apps/<app-id>
    This status appears on the Pipeline List page.

  • Obsolete. A teammate deployed a newer version while you were editing the pipeline. This status appears on the Pipeline List page. This is the same as Draft Out of Date that appears on the Pipeline canvas.

Pipeline Studio canvas statuses

On the Pipeline Studio canvas, pipelines can have the following statuses while you are editing them:

  • Editing in-Progress. You are editing a pipeline version in DRAFT mode. This status appears in the Pipeline Studio canvas toolbar:

  • Orphaned draft. A teammate deployed a newer version while you were editing the pipeline. This status appears in the Pipeline Studio canvas toolbar:

 

Created in 2020 by Google Inc.