...

• BigQuery Argument Setter

• Database Argument Setter

• GCS Argument Setter. To learn more, see Creating a reusable pipeline with the GCS Argument Setter.

• HTTP Argument Setter

There are two types of macros:

• Property lookups

• Macro functions

Property lookups

Property lookups are macros defined as variables wrapped inside of a ${ }. To separate the words of a placeholder or key, you can use dot notation, camel case, dashes, or underscores, but whatever you choose, make sure that you are consistent in your use of the parameters so that there is no ambiguity or confusion when you come across macros later.  

A typical use case for macros is to allow for variable substitution when the pipeline is run, so instead of using hard coded paths, you can use dynamic paths. This way you’re able to specify which file you want to process when the pipeline is invoked.  

For example, in a GCS source plugin, you might replace the path using multiple macros to split the bucket, folder, and file portions as follows: gs://${bucket.name}/${folder}/${file.name}. Or, if you want to ingest data from a file with a name that isn’t static, replace the file name portion with ${file.name}. The Path field will look like this:  

...

When you deploy a pipeline with macros, click Run to open Runtime Arguments and set the values for each macro. You can also use wildcards (*) in the values. For example, to read all .txt files that start with POS, enter POS*.txt. To read all .txt files in the directory, enter *.txt. If the folder contains only .txt files, enter / to read all files in the directory.

...

Macro functions

In addition to property lookups, you can use the following predefined macro functions:

• logicalStartTime

• secure

Logical Start Time Function

The logicalStartTime macro function returns the logical start time of a run of the pipeline.

If no parameters are supplied, it returns the start time in milliseconds. All parameters are optional. The function takes a time format, an offset, and a timezone as arguments and uses the logical start time of a pipeline to perform the substitution:

${logicalStartTime([timeFormat[,offset [,timezone])}

The following table lists the optional parameters for logicalStartTime:

...

Parameter

...

Description

...

timeFormat

...

Optional. Time format string, in the format of a Java SimpleDateFormat.

...

offset

...

Optional. Offset from before the logical start time.

...

timezone

...

Optional. Timezone to be used for the logical start time.

For example, suppose the logical start time of a pipeline run is 2020-01-01T00:00:00 and this macro is provided:

${logicalStartTime(yyyy-MM-dd'T'HH-mm-ss,1d-4h+30m)}

The format is yyyy-MM-dd'T'HH-mm-ss and the offset is 1d-4h+30m before the logical start time. This means the macro will be replaced with 2019-12-31T03:30:00, since the offset translates to 20.5 hours. The entire macro evaluates to 20.5 hours before midnight of January 1, 2020.

Including the today’s date in a filename

You can also use the logicalStartTime macro function in a filename to capture the current date. For example, if you want to capture today's date in an S3 filename that looks like this:

s3a://sales-data/sales_20210204.csv

in an Amazon S3 sink, you would enter this in the Path field:

s3a://sales-data/sales_${logicalStartTime(yyyyMMdd)}.csv

Secure Function

The secure macro function takes in a single key as an argument and looks up the key's associated string value from the Secure Store. In order to perform the substitution, the key provided as an argument must already exist in the secure store. This is useful for performing a substitution with sensitive data.

For example, for a plugin that connects to a MySQL database, you could configure the password property field with:

${secure(password)}

which will pull the password from the Secure Store at runtime.

Recursive Macros

Macros can be referential (refer to other macros), up to 10 levels deep. Macro arguments are evaluated from the innermost to the outermost. For example, you might have a server that refers to a hostname and port, and supply these runtime arguments, one of which is a definition of a macro that uses other macros:

hostname: my-demo-host.example.com
port: 9991
server-address: ${hostname}:${port}

In a pipeline configuration, you could use an expression such as:

server-address: ${server-address}

expecting that it would be replaced with:

my-demo-host.example.com:9991