Reusable data pipelines

In the Pipeline Studio, you can use macros to add dynamic variables to plugin configurations so that you can specify the variable substitutions at runtime. 

With reusable data pipelines, you can have a single pipeline that can apply a data integration pattern to a variety of use cases and datasets.

Reusable pipelines are give you better manageability. They let you set most of the configuration of a pipeline at execution time, instead of hard-coding it at design time.

Fields that can be made dynamic using macros have a blue M icon on the far right in the Plugin Configuration section:

You can also use the Argument Setter plugins to build a reusable pipelines:

• BigQuery Argument Setter

• Database Argument Setter

• GCS Argument Setter. To learn more, see Creating a reusable pipeline with the GCS Argument Setter.

• HTTP Argument Setter