Directives

JSON Path

Uses a DSL (a JSON path expression) for parsing JSON records

Parse as AVRO File

Parses an AVRO data file

Parse as CSV

Parses an input record as comma-separated values

Parse as Currency

Parses a currency value that is a string representation of locale currency into a number.

Parse as Date (Deprecated)

Use Parse as Simple Date.

Parse as Datetime

6.4.0

Parses string as datetime with a given format.

Parse as Excel

Parsing Excel file.

Parse as Fixed Length

Parses as a fixed length record with specified widths

Parse as HL7

Parses Health Level 7 Version 2 (HL7 V2) messages

Parse as JSON

Parses a JSON object

Parse as Log

Parses access log files as from Apache HTTPD and nginx servers

Parse as Simple Date

Parses date strings

Parse as Timestamp

Parses column values representing unix timestamp as date

Parse XML To JSON

Parses an XML document into a JSON structure

Write as CSV

Converts a record into CSV format

Write as JSON Map

Converts the record into a JSON map

Write JSON Object

Composes a JSON object based on the fields specified.

Format as Currency

Formats a number as currency as specified by locale.

Changing Case

Changes the case of column values

Create Record

Creates a record column with nested values by copying values from source columns into a destination column

Cut Character

Selects parts of a string value

Set Column

Sets the column value to the result of an expression execution

Find and Replace

Transforms string column values using a "sed"-like expression

Index Split

(Deprecated)

Invoke HTTP

Invokes an HTTP Service (Experimental, potentially slow)

Quantization

Quantizes a column based on specified ranges

Extract Regex Groups

Extracts the data from a regex group into its own column

Set Charset

Sets the encoding and then converts the data to a UTF-8 String

Set Record Delim

Sets the record delimiter

Split by Separator (Deprecated)

(Deprecated)

Split Email

Splits an email ID into an account and its domain

Split URL

Splits a URL into its constituents

Text Distance (Fuzzy String Match)

Measures the difference between two sequences of characters

Text Metric (Fuzzy String Match)

Measures the difference between two sequences of characters

URL Decode

Decodes from the application/x-www-form-urlencoded MIME format

URL Encode

Encodes to the application/x-www-form-urlencoded MIME format

Trimming Spaces

Functions for trimming white spaces around string data

Decode

Decodes a column value as one of base32, base64, or hex

Encode

Encodes a column value as one of base32, base64, or hex