Page Comparison

To create a reusable pipeline, you can use macros and macro functions.

...

A common use of macros is in Path fields. Instead of using hard coded paths, you can use dynamic paths. For example, in a GCS source plugin, you might replace the path using multiple macros to split the bucket, folder, and file portions as follows: gs://${bucket.name}/${folder}/${file.name}. Or, if you want to ingest data from a bucket that is static and a file with a name that isn’t static, enter the name of the bucket and use a macro for the file name. The Path field will look like this:  

...

To add a macro to a field in a plugin, follow these steps:

1. Click the M button next to the field where you want to use a macro.

2. Enter the key for the macro. For example, if you want to use a macro in a File source for the Format field, enter ${format.type}.

Setting macro values

You set Set values for macros before you preview data for a pipeline and before you run a pipeline.

You can set macro values in the following places:

1. Argument Setter plugins

2. Runtime arguments

3. Application preferences

4. Namespace preferences

5. System preferences

When you run a pipeline with macros, CDAP first checks if the pipeline includes an Argument Setter plugin. If it does, CDAP uses the values for macros in the Argument Setter. If there isn’t an Argument Setter plugin or if there are macros that are not assigned in the Argument Setter, CDAP uses the values in the pipeline runtime arguments.

...

The following table lists the optional parameters for logicalStartTime:

For example, suppose the logical start time of a pipeline run is 2020-01-01T00:00:00 and this macro is provided:

...