Reusable data pipelines

With reusable data pipelines, you can have a single pipeline that can apply a data integration pattern to a variety of use cases and datasets. Reusable pipelines give you better manageability. They let you set most of the configuration of a pipeline at execution time, instead of hard-coding it at design time.

In the Pipeline Studio, you can use macros to add dynamic variables to plugin configurations so that you can specify the variable substitutions at runtime. 

That pipeline behavior will be different based on the namespace or the instance it is executed in. Macros let you design the pipeline once, and run it in different environments in various forms.

You can add macros to any plugin, including Argument Setters.

You can set the values for macros in Pipeline Studio in the following areas:

• Argument setters

• Runtime arguments

• Application preferences

• Namespace preferences

• System preferences

When you run a pipeline, CDAP evaluates the macro values in the same order listed above. For example, if you set the value for a macro as a namespace preference and also in a runtime preference, CDAP uses the value in the runtime preference.