...
Configuration
General Properties
Property | Macro enabled | Required | Description |
Reference Name | No | Yes | Required. Name used to uniquely identify this source for lineage, annotating metadata, etc. |
URL | Yes | Yes | The URL to post data to. Additionally, a placeholder like #columnName can be added to the URL that can be substituted with column value at the runtime. |
HTTP Method | Yes | Yes | The HTTP request method. Defaults to |
Batch Size | Yes | Yes | Number of records in each HTTP request. |
Write JSON As Array | Yes | Yes | Whether to write the JSON as an array. Defaults to |
Json Batch Key | Yes | No | Optional key to be used for wrapping JSON array as an object. Leave empty for no wrapping of the array. Ignored if Write JSON As Array is set to |
Message Format | Yes | Yes | Format to send messages in. Options are JSON, Form, Custom. Defaults to JSON. |
Custom Message | Yes | No | This is used to define a custom message format. |
Delimiter For Batch Messages | Yes | No | Delimiter for messages in case of batch size more than 1. Defaults to “\n” |
Request Headers | Yes | No | An optional string of header values to send in each request. |
Charset for Text | Yes | Yes | Charset. Defaults to UTF-8. |
Should Follow Redirects ? | Yes | Yes | Whether to automatically follow redirects. Defaults to true. |
Disable SSL Validation | Yes | Yes | If a user enables SSL validation, they will be expected to add the certificate to the trustStore on each machine. Defaults to true. |
Connection Timeout (milliseconds) | Yes | No | Connection timeout, defaults to 60000 (1 Minute) , use 0 for infinite. |
Read Timeout (milliseconds) | Yes | No | Read timeout, defaults to 60000 (1 Minute) , use 0 for infinite. |
Wait Time Between Request | Yes | No | The time duration between requests. |
HTTP Errors Handling | No | No | Defines the error handling strategy for certain HTTP response codes. The left column contains a regular expression for HTTP status code. The right column contains an action to perform if there’s a match. If the HTTP status code matches multiple regular expressions, the first that’s specified in the mapping is matched. |
Non-HTTP Error Handling | No | Yes | Error handling strategy to use when the HTTP response cannot be transformed to an output record. Possible values are:
Default is Stop on error. |
Retry Policy | No | Yes | Policy used to calculate delay between retries. Default is Exponential. |
Linear Retry Interval | Yes | No | Interval between retries. This property only appears if Retry policy is Linear. Default is 30. |
Max Retry Duration | Yes | No | Maximum time, in seconds, that retries can take. Default is 600. |
Authentication
OAuth2 Properties
Property | Macro enabled | Required | Description |
OAuth2 Enabled | No | No | If set to True, the plugin will perform QAuth2 authentication. Default is False. |
Auth URL | Yes | No | Endpoint for the authorization server that’s used to retrieve the authorization code. |
Token URL | Yes | No | Endpoint for the resource server, which exchanges the authorization code for an access token. |
Client ID | Yes | No | Client identifier obtained during the Application registration process. |
Client Secret | Yes | No | Client secret obtained during the Application registration process. |
Scopes | Yes | No | Scope of the access request, which might have multiple space-separated values. |
Refresh Token | Yes | No | Token used to receive accessToken, which is end product of OAuth2. |
Service Account Properties
Property | Macro enabled | Required | Description |
Service Account Type | No | No | Select one of the following options:
|
Service Account File Path | Yes | No | Path on the local file system of the service account key used for authorization. Can be set to |
Service Account JSON | Yes | No | Service account JSON file contents. |
Service Account Scope | Yes | No | The additional Google credential scopes required to access entered URL. |
Basic Authentication Properties
Property | Macro enabled | Required | Description |
Username | Yes | No | Username for basic authentication. |
Password | Yes | No | Password for basic authentication. |
HTTP Proxy Properties
Property | Macro enabled | Required | Description |
Proxy URL | Yes | No | Proxy URL. Must contain a protocol, address, and port. |
Example
This example performs an HTTP POST
request to http://example.com/data
.
...