Send to Error directive

The SEND-TO-ERROR directive allows the filtering of records and directs the filtered records that match a given condition to an error collector. If the error collector is not connected as the next stage in a data pipeline, then the filtered records will be dropped.


send-to-error exp:{<condition>} [metric] ['message']

The condition is an EL specifying the condition that governs if the record should be sent to the error collector. Optionally, you can specify the metric name that should be registered every time a record is sent to error combined with the optional ability to specify an error message that should be recorded.

Usage Notes

The most common use of the SEND-TO-ERROR directive is to filter out records that are not part of clean data. This is a data cleansing directive to remove records that do not conform to specified rules.

The record is sent to the error collector (if connected) when the condition for the record evaluates to true. If the condition evaluates to false, the record is passed on untouched.


Assume a record that has these three fields:

  • Name

  • Age

  • DOB

As part of a data cleansing process, check that all the records being ingested follow these rules:

  • Name is not empty

  • Age is not empty and not less than 1 or greater 130

  • DOB is a valid date

These directives will implement these rules; any records that match any of these conditions will be sent to the error collector for further investigation:

send-to-error Name == null send-to-error Age.isEmpty() send-to-error Age < 1 || Age > 130 send-to-error !date:isDate(DOB) send-to-error Age.isEmpty age_empty 'Age field is empty' send-to-error Name == null name_null send-to-error Age < 1 || Age > 130 'Age not in range between 1 - 130'


Created in 2020 by Google Inc.