Deploying a data pipeline

After you finish designing and debugging a data pipeline and are satisfied with the data you see in Preview, you are ready to deploy the pipeline.

Note: A deployed pipeline name must be unique in the namespace. You might be prompted to enter a unique name.

Deploying converts the pipeline into an immutable entity, which is not versioned.

After deployment, the pipeline can be either run manually using the Run button, scheduled on a time basis using the Schedule button, or triggered based on the completion of another pipeline using Incoming Triggers. The Pipeline Studio saves a run history of the pipeline each time it runs. You can toggle between different runtime versions of the pipeline:

When you deploy the pipeline, the Pipeline Studio creates the workflow and corresponding Spark jobs in the background.

Once you deploy a pipeline, you cannot edit it. You can duplicate it or export it and then import it into the Pipeline Studio.

After you deploy a pipeline, you can configure runtime arguments if the pipeline includes macros.