Wrangler Transformation

This plugin applies data transformation directives on your data records. The directives are generated either through an interactive user interface or by manual entry into the plugin.

Configuration

Property

Macro Enabled?

Description

Property

Macro Enabled?

Description

Input field name

Yes

Required. The name of the input field (or * for all fields).

Default is * (asterisk).

Precondition

Yes

Required. A filter to be applied before a record is passed to data prep.

Default is false.

Directives (Recipe)

Yes

Required. The series of data prep directives to be applied on the input records.

User Defined Directives (UDD)

No

Optional. List of User Defined Directives (UDD) that must be loaded.

Failure Threshold

Yes

Required. Maximum number of errors tolerated before exiting pipeline processing.

Default is 1.

Output Schema

Yes

Required. The output schema for the data.

Directives

There are numerous directives and variations supported by CDAP, which are documented at http://github.com/hydrator/wrangler.

Usage Notes

All input record fields are made available to the data prep directives when * is used as the field to be data prepped. They are in the record in the same order as they appear.

Note that if the transform doesn’t operate on all of the input record fields or a field is not configured as part of the output schema, and you are using the set columns directive, you may see inconsistent behavior. Use the drop directive to drop any fields that are not used in the data prep.

A precondition filter is useful to apply filtering on records before the records are delivered for data prep. To filter a record, specify a condition that will result in boolean state of true.

For example, to filter out all records that are a header record from a CSV file where the header record is at the start of the file, you could use this filter:

1 offset == 0

This will filter out records that have an offset of zero.

This plugin uses the emiterror capability to emit records that fail parsing into a separate error stream, allowing the aggregation of all errors. However, if the Failure Threshold is reached, then the pipeline will fail.