Plugin version: 4.9.0
Wrangler is an interactive tool that lets you perform transformations on a subset of your data. It allows you to apply directives and create recipes using UI or JEXL commands. This plugin applies data transformation directives on your data records. The directives are generated either through an interactive user interface or by manual entry into the plugin.
The Precondition step of a Wrangler stage in a pipeline is now eligible to execute in BigQuery when BigQuery ELT Transformation Pushdown is enabled in a pipeline. This is only supported when the Precondition Language is set to SQL.
Property | Macro Enabled? | Version Introduced | Description |
---|---|---|---|
Input field name | Yes | Required. The name of the input field (or * for all fields). Default is * (asterisk). | |
Precondition Language | Yes | 6.9.0/4.9.0 | Required. This is a language selector for preconditions (JEXL/SQL). Default is |
Precondition (JEXL) | Yes | 6.9.0/4.9.0 | Required. A JEXL filter to be applied before the directives are executed. Default is False. |
Directives (Recipe) | Yes | Required. The series of directives to be applied on the input records. | |
User Defined Directives (UDD) | No | Optional. List of User Defined Directives (UDD) that must be loaded. | |
Error Handling | Yes | Required. Strategy to handle erroneous records.
For example, if there are string values in a column for certain rows where the directive, Default is | |
Output Schema | Yes | Required. The output schema for the data. |
There are numerous directives and variations supported by CDAP. See Directives.
For information about working with decimals and BigDecimal types, see Working with Decimal types in Wrangler.
All input record fields are made available to the directives when * is used as the field to be transformed. They are in the record in the same order as they appear.
Precondition Language is set to JEXL
by default. It can be switched between SQL
and JEXL
.
If Precondition Language is set to SQL
, the Directive and UDD fields must be blank. If these fields have values, plugin validation fails. In addition, Wrangler doesn't support multiple input stages when the Precondition Language is set to SQL
.
A precondition filter is useful to filter records before the directives are applied to the records. To filter a record, specify a condition that will result in a Boolean state of true
.
For example, to filter out all records that have a value of under 18 for an age
field, you could use this filter:
age < 18