...
Wrangler is an interactive tool that lets you perform transformations on a subset of your data. It allows you to apply directives and create recipes using UI or JEXL commands. This plugin applies data transformation directives on your data records. The directives are generated either through an interactive user interface or by manual entry into the plugin.
BigQuery ELT Pushdown for Preconditions (6.9.0+)
The Precondition step of a Wrangler stage in a pipeline is now eligible to execute in BigQuery when BigQuery ELT Transformation Pushdown is enabled in a pipeline. This is only supported when the Precondition Language is set to SQL.
Configuration
Property | Macro Enabled? | Version Introduced | Description |
|---|---|---|---|
Input field name | Yes | Required. The name of the input field (or * for all fields). Default is * (asterisk). | |
Precondition Language | Yes | 6.9.0/4.9.0 | Required. This is a language selector for preconditions (JEXL/SQL). Default is |
Precondition (JEXL) | Yes | 6.9.0/4.9.0 | Required. A JEXL filter to be applied before the directives are executed. Default is False. |
Directives (Recipe) | Yes | Required. The series of directives to be applied on the input records. | |
User Defined Directives (UDD) | No | Optional. List of User Defined Directives (UDD) that must be loaded. | |
Error Handling | Yes | Required. Strategy to handle erroneous records.
For example, if there are string values in a column for certain rows where the directive, set-type :col_name integer is used, this results in an error. Default is Fail pipeline. | |
Output Schema | Yes | Required. The output schema for the data. |
...