Directives

JSON Path

Uses a DSL (a JSON path expression) for parsing JSON records

Parse as AVRO File

Parses an AVRO data file

Parse as CSV

Parses an input record as comma-separated values

Parse as Currency

Parses a currency value that is a string representation of locale currency into a number.

Parse as Date (Deprecated)

Use Parse as Simple Date.

Parse as Datetime

6.4.0

Parses string as datetime with a given format.

Parse as Excel

Parsing Excel file.

Parse as Fixed Length

Parses as a fixed length record with specified widths

Parse as HL7

Parses Health Level 7 Version 2 (HL7 V2) messages

Parse as JSON

Parses a JSON object

Parse as Log

Parses access log files as from Apache HTTPD and nginx servers

Parse as Simple Date

Parses date strings

Parse as Timestamp

Parses column values representing unix timestamp as date

Parse XML To JSON

Parses an XML document into a JSON structure

Write as CSV

Converts a record into CSV format

Write as JSON Map

Converts the record into a JSON map

Write JSON Object

Composes a JSON object based on the fields specified.

Format as Currency

Formats a number as currency as specified by locale.

Changing Case

Changes the case of column values

Create Record

Creates a record column with nested values by copying values from source columns into a destination column

Cut Character

Selects parts of a string value

Set Column

Sets the column value to the result of an expression execution

Find and Replace

Transforms string column values using a "sed"-like expression

Index Split

(Deprecated)

Invoke HTTP

Invokes an HTTP Service (Experimental, potentially slow)

Quantization

Quantizes a column based on specified ranges

Extract Regex Groups

Extracts the data from a regex group into its own column

Set Charset

Sets the encoding and then converts the data to a UTF-8 String

Set Record Delim

Sets the record delimiter

Split by Separator (Deprecated)

(Deprecated)

Split Email

Splits an email ID into an account and its domain

Split URL

Splits a URL into its constituents

Text Distance (Fuzzy String Match)

Measures the difference between two sequences of characters

Text Metric (Fuzzy String Match)

Measures the difference between two sequences of characters

URL Decode

Decodes from the application/x-www-form-urlencoded MIME format

URL Encode

Encodes to the application/x-www-form-urlencoded MIME format

Trimming Spaces

Functions for trimming white spaces around string data

Decode

Decodes a column value as one of base32, base64, or hex

Encode

Encodes a column value as one of base32, base64, or hex

Generate UUID

Generates a universally unique identifier (UUID)

Diff Date

Calculates the difference between two dates

Format Date

Custom patterns for date-time formatting

Format UNIX Timestamp

Formats a UNIX timestamp as a date

Current Datetime

6.4.0

Generates current datetime with a given time zone

Datetime to Timestamp

6.4.0

Creates a timestamp from datetime and given time zone

Format Datetime

6.4.0

Formats datetime to a string of a given format

Timestamp to Datetime

6.4.0

Converts a timestamp to datetime

Catalog Lookup

Static catalog lookup of ICD-9, ICD-10-2016, ICD-10-2017 codes

Table Lookup

Performs lookups into Table datasets

Hash

Generates a message digest

Mask Number

Applies substitution masking on the column values

Mask Shuffle

Applies shuffle masking on the column values

Filter Rows

Filters records based on a condition.

Filter Rows On

Deprecated. Filters records based on a condition.

Filter Row if Matched

Filters rows that match a pattern for a column

Filter Row if True (Deprecated)

Filters rows if the condition is true.

Filter Row Empty of Null

(Obsolete?) Filters rows that are empty of null.

Flatten

Separates the elements in a repeated field

Fail on condition

Fails processing when the condition is evaluated to true.

Send to Error

Filtering of records to an error collector

Send to Error and Continue Processing

Filtering of records to an error collector and continues processing

Split to Rows

Splits based on a separator into multiple records

Change Column Case

Changes column names to either lowercase or uppercase

Changing Case

Changes the case of column values

Cleanse Column Names

Sanitizes column names, following specific rules

Columns Replace

Alters column names in bulk

Copy

Copies values from a source column into a destination column

Drop Column

Drops a column in a record

Fill Null or Empty

Fills column value with a fixed value if null or empty

Keep

Keeps specified columns from the record

Merge

Merges two columns by inserting a third column

Rename

Renames an existing column in the record

Set Headers

Sets the names of columns, in the order they are specified

Split to Columns

Splits a column based on a separator into multiple columns

Swap

Swaps column names of two columns

Set Type

Converts data type of a column

Stemming Tokenized Words

Applies the Porter stemmer algorithm for English words

Increment Variable

Increments a transient variable with a record of processing.

Set Variable

Sets a transient variable with a record of processing.