| Table of Contents | ||||
|---|---|---|---|---|
|
To create a reusable pipeline, you can use macros and macro functions.
...
A common use of macros is in Path fields. Instead of using hard coded paths, you can use dynamic paths. For example, in a GCS source plugin, you might replace the path using multiple macros to split the bucket, folder, and file portions as follows: gs://${bucket.name}/${folder}/${file.name}. Or, if you want to ingest data from a bucket that is static and a file with a name that isn’t static, enter the name of the bucket and use a macro for the file name. The Path field will look like this:
...
| Info |
|---|
Want to watch the video? |
To add a macro to a field in a plugin, follow these steps:
Click the M button next to the field where you want to use a macro.
Enter the key for the macro. For example, if you want to use a macro in a File source for the Format field, enter ${format.type}.
Note: The M button is a toggle. Click it again to reset the default value of the plugin property.
...
You can set macro values in the following places:
Argument Setter plugins
Runtime arguments
Application preferences
Namespace preferences
System preferences
When you run a pipeline with macros, CDAP first checks if the pipeline includes an Argument Setter plugin. If it does, CDAP uses the values for macros in the Argument Setter. If there isn’t an Argument Setter plugin or if there are macros that are not assigned in the Argument Setter, CDAP uses the values in the pipeline runtime arguments.
...
The following table lists the optional parameters for logicalStartTime:
Parameter | Description |
|---|---|
timeFormat | Optional. Time format pattern, in the format of a Java SimpleDateFormat. |
offset | Optional. Offset from before the logical start time. |
timezone | Optional. Timezone to be used for the logical start time. |
For example, suppose the logical start time of a pipeline run is 2020-01-01T00:00:00 and this macro is provided:
...