This plugin applies data transformation directives on your data records. The directives are generated either through an interactive user interface or by manual entry into the plugin.
Configuration
Property | Macro Enabled? | Description |
|---|---|---|
Input field name | Yes | Required. The name of the input field (or * for all fields). Default is * (asterisk). |
Precondition | Yes | Required. A filter to be applied before a record is passed to data prep. Default is false. |
Directives (Recipe) | Yes | Required. The series of data prep directives to be applied on the input records. |
User Defined Directives (UDD) | No | Optional. List of User Defined Directives (UDD) that must be loaded. |
Failure Threshold | Yes | Required. Maximum number of errors tolerated before exiting pipeline processing. Default is 1. |
Output Schema | Yes | Required. The output schema for the data. |
Directives
There are numerous directives and variations supported by CDAP, which are documented at http://github.com/hydrator/wrangler.
Usage Notes
All input record fields are made available to the data prep directives when * is used as the field to be data prepped. They are in the record in the same order as they appear.
Note that if the transform doesn’t operate on all of the input record fields or a field is not configured as part of the output schema, and you are using the set columns directive, you may see inconsistent behavior. Use the drop directive to drop any fields that are not used in the data prep.
A precondition filter is useful to apply filtering on records before the records are delivered for data prep. To filter a record, specify a condition that will result in boolean state of true.
For example, to filter out all records that are a header record from a CSV file where the header record is at the start of the file, you could use this filter:
| Code Block |
|---|
offset == 0 |
This will filter out records that have an offset of zero.
This plugin uses the emiterror capability to emit records that fail parsing into a separate error stream, allowing the aggregation of all errors. However, if the Failure Threshold is reached, then the pipeline will fail.