Plugin version: 4.89.10
Wrangler is an interactive tool that lets you perform transformations on a subset of your data. It allows you to apply directives and create recipes using UI or JEXL commands. This plugin applies data transformation directives on your data records. The directives are generated either through an interactive user interface or by manual entry into the plugin.
BigQuery ELT Pushdown for Preconditions
The Precondition step of a Wrangler stage in a pipeline is now eligible to execute in BigQuery when BigQuery ELT Transformation Pushdown is enabled in a pipeline. This is only supported when the Precondition Language is set to SQL.
Configuration
Property | Macro Enabled? | Description |
|---|---|---|
Input field name | Yes | Required. The name of the input field (or * for all fields). Default is * (asterisk). |
Precondition Language | Yes | Required. This is a language selector for preconditions (JEXL/SQL). Default is |
Precondition (JEXL) | Yes | Required. A JEXL filter to be applied before a record is passed to data prepthe directives are executed. Default is falseFalse. |
Directives (Recipe) | Yes | Required. The series of data prep directives to be applied on the input records. |
User Defined Directives (UDD) | No | Optional. List of User Defined Directives (UDD) that must be loaded. |
Error Handling | Yes | Required. How Strategy to handle errors in record processing: Skip recorderroneous records.
Default is fail For example, if there are string values in a column for certain rows where the directive, set-type :col_name integer is used, this results in an error. Default is Fail pipeline. |
Output Schema | Yes | Required. The output schema for the data. |
...
All input record fields are made available to the data prep directives when when * is is used as the field to be data preppedtransformed. They are in the record in the same order as they appear.Note that if the transform doesn’t operate on all of the input record fields or a field is not configured as part of the output schema, and you are using the set columns directive, you may see inconsistent behavior. Use the drop directive to drop any fields that are not used in the data prep
Precondition Language is set to JEXL by default. It can be switched between SQL and JEXL.
If Precondition Language is set to SQL, the Directive and UDD fields must be blank. If these fields have values, plugin validation fails. In addition, Wrangler doesn't support multiple input stages when the Precondition Language is set to SQL.
A precondition filter is useful to apply filtering on filter records before the records are delivered for data prepdirectives are applied to the records. To filter a record, specify a condition that will result in boolean a Boolean state of of true.
For example, to filter out all records that are a header record from a CSV file where the header record is at the start of the filehave a value of under 18 for an age field, you could use this filter:
| Code Block |
|---|
offset == 0 |
This will filter out records that have an offset of zero.
This plugin uses the emiterror capability to emit records that fail parsing into a separate error stream, allowing the aggregation of all errors. However, if the Failure Threshold is reached, then the pipeline will fail.age < 18