HTTP Sink

The HTTP sink plugin is available in the Hub.

Plugin version: 1.4.1

Sink plugin to send the messages from the pipeline to an external HTTP endpoint.

Configuration

General Properties

Property	Macro enabled	Required	Description
Reference Name	No	Yes	Required. Name used to uniquely identify this source for lineage, annotating metadata, etc.
URL	Yes	Yes	The URL to post data to. Additionally, a placeholder like #columnName can be added to the URL that can be substituted with column value at the runtime.
HTTP Method	Yes	Yes	The HTTP request method. Defaults to `POST`.
Batch Size	Yes	Yes	Number of records in each HTTP request.
Write JSON As Array	Yes	Yes	Whether to write the JSON as an array. Defaults to `false`.
Json Batch Key	Yes	No	Optional key to be used for wrapping JSON array as an object. Leave empty for no wrapping of the array. Ignored if Write JSON As Array is set to `false`.
Message Format	Yes	Yes	Format to send messages in. Options are JSON, Form, Custom. Defaults to JSON.
Custom Message	Yes	No	This is used to define a custom message format.
Delimiter For Batch Messages	Yes	No	Delimiter for messages in case of batch size more than 1. Defaults to “\n”
Request Headers	Yes	No	An optional string of header values to send in each request.
Charset for Text	Yes	Yes	Charset. Defaults to UTF-8.
Should Follow Redirects ?	Yes	Yes	Whether to automatically follow redirects. Defaults to true.
Disable SSL Validation	Yes	Yes	If a user enables SSL validation, they will be expected to add the certificate to the trustStore on each machine. Defaults to true.
Connection Timeout (milliseconds)	Yes	No	Connection timeout, defaults to 60000 (1 Minute) , use 0 for infinite.
Read Timeout (milliseconds)	Yes	No	Read timeout, defaults to 60000 (1 Minute) , use 0 for infinite.
Wait Time Between Request	Yes	No	The time duration between requests.
HTTP Errors Handling	No	No	Defines the error handling strategy for certain HTTP response codes. The left column contains a regular expression for HTTP status code. The right column contains an action to perform if there’s a match. If the HTTP status code matches multiple regular expressions, the first that’s specified in the mapping is matched.
Non-HTTP Error Handling	No	Yes	Error handling strategy to use when the HTTP response cannot be transformed to an output record. Possible values are: Stop on error: fails pipeline due to erroneous record Send to error: Sends erroneous record’s text to the error port and then continues Skip on error: Ignores erroneous records Default is Stop on error.
Retry Policy	No	Yes	Policy used to calculate delay between retries. Default is Exponential.
Linear Retry Interval	Yes	No	Interval between retries. This property only appears if Retry policy is Linear. Default is 30.
Max Retry Duration	Yes	No	Maximum time, in seconds, that retries can take. Default is 600.

Authentication

OAuth2 Properties

Property	Macro enabled	Required	Description
OAuth2 Enabled	No	No	If set to True, the plugin will perform QAuth2 authentication. Default is False.
Auth URL	Yes	No	Endpoint for the authorization server that’s used to retrieve the authorization code.
Token URL	Yes	No	Endpoint for the resource server, which exchanges the authorization code for an access token.
Client ID	Yes	No	Client identifier obtained during the Application registration process.
Client Secret	Yes	No	Client secret obtained during the Application registration process.
Scopes	Yes	No	Scope of the access request, which might have multiple space-separated values.
Refresh Token	Yes	No	Token used to receive accessToken, which is end product of OAuth2.

Service Account Properties

Property	Macro enabled	Required	Description
Service Account Type	No	No	Select one of the following options: File Path. File path where the service account is located. JSON. JSON content of the service account.
Service Account File Path	Yes	No	Path on the local file system of the service account key used for authorization. Can be set to `auto-detect` when running on a Dataproc cluster. When running on other clusters, the file must be present on every node in the cluster.
Service Account JSON	Yes	No	Service account JSON file contents.
Service Account Scope	Yes	No	The additional Google credential scopes required to access entered URL. `cloud-platform` is included by default. For more information, see OAuth 2.0 Scopes for Google APIs.

Basic Authentication Properties

Property	Macro enabled	Required	Description
Username	Yes	No	Username for basic authentication.
Password	Yes	No	Password for basic authentication.

HTTP Proxy Properties

Property	Macro enabled	Required	Description
Proxy URL	Yes	No	Proxy URL. Must contain a protocol, address, and port.

Example

This example performs an HTTP POST request to http://example.com/data.

Property	Value

Property	Value
URL	`http://example.com/data`
HTTP Method	`POST`
Batch Size	`1`
Message Format	`JSON`
Charset for Text	`UTF-8`
Should Follow Redirects?	`true`
Disable SSL Validation	`true`
Number of Retries	`0`
Connection Timeout (milliseconds)	`60000`
Read Timeout (milliseconds)	`60000`