Running a data pipeline

After you deploy a data pipeline, you can run it. You can create a schedule to run a deployed pipeline or you can run it on demand.

To run a pipeline on demand, open a deployed pipeline and click Run.

If the pipeline has macros, set the runtime arguments for each macro.

You can see the status change during the phases of the pipeline run, for example provisioning, starting, running, and succeeded. You can also stop the pipeline at any time.

You can also view basic and advanced logs while the pipeline runs.

After the pipeline run completes

After the pipeline run completes, you can view and download the logs, including the advanced logs:

If you enabled Instrumentation, you can click the Properties button on any node (source, transformation, sink) in the pipeline and explore the metrics generated by the pipeline.

To see more details about the pipeline runs, you can click the Summary button.

Validating output

After the data pipeline run succeeds, you can validate the output in each sink to ensure the pipeline run produced the desired results.