Amazon SQS Streaming Source

The Amazon SQS Streaming source plugin is available in the Hub.

The source reads messages from SQS Queue using the long polling option of the API. SQS currently does provide ability for callbacks or streaming. The plugin will continuously issue a call to read messages off the queue. There are options to control the frequency of the poll interval.

Configuration

Property	Macro Enabled?	Description

Property	Macro Enabled?	Description
Reference Name	No	Required. Name used to uniquely identify this source for lineage, annotating metadata, etc.
Amazon Region	Yes	Required. The Amazon Region where the SQS queue is located. Default is us-west-1.
Queue Name	Yes	Required. The name of SQS queue to read messages from the plugin.
SQS Endpoint	Yes	Optional. The endpoint of the SQS server to connect to. For example, `https://sqs.us-east-1.amazonaws.com`
Authentication Method	Yes	Required. Authentication method to access SQS. The default value is Access Credentials. IAM can only be used if the plugin is run in an AWS environment, such as on EMR. Default is Access Credentials.
Access ID	Yes	Optional. Amazon access ID required for authentication.
Access Key	Yes	Optional. Amazon access key required for authentication.
Delete messages after read?	Yes	Required. Permanently delete messages from the queue after successfully reading. If set to `No`, the messages will stay on queue and will be received again leading to duplicates. Default is Yes.
Format	No	Optional. Optional format of the SQS message. Any format supported by CDAP is supported. For example, a value of `csv` will attempt to parse SQS message as comma-separated values. If no format is given, SQS messages will be treated as bytes. Default is bytes.
Interval (in seconds)	Yes	Required. The amount of time to wait between each poll in seconds. The plugin will wait for the duration specified before issuing a receive message call. Default is 0.
Wait Time (in seconds)	Yes	Required. The duration (in seconds) for which the call waits for a message to arrive in the queue before returning. This will be set as part of the ReceiveMessage API call action. The maximum wait time currently allowed by SQS 20 seconds. Default is 10.
Number of messages to return	Yes	Required. The maximum number of messages to return from a single API request. The maximum value allowed by SQS is 10. Default is 10.