All input record fields are made available to the data prep directives when * is used as the field to be data prepped. They are in the record in the same order as they appear.
Note that if the transform doesn’t operate on all of the input record fields or a field is not configured as part of the output schema, and you are using the set columns directive, you may see inconsistent behavior. Use the drop directive to drop any fields that are not used in the data prep.
A precondition filter is useful to apply filtering on records before the records are delivered for data prep. To filter a record, specify a condition that will result in boolean state of true.
For example, to filter out all records that are a header record from a CSV file where the header record is at the start of the file, you could use this filter:
offset == 0
This will filter out records that have an offset of zero.
This plugin uses the emiterror capability to emit records that fail parsing into a separate error stream, allowing the aggregation of all errors. However, if the Failure Threshold is reached, then the pipeline will fail.