Amazon SQS plugin
Introduction
Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. The SQS plugin in CDAP will enable ETL developers to create streaming pipelines that read events from SQS queues in realtime and process them.
Use case(s)
As a user, I would like to create a streaming pipeline that reads events from Amazon SQS, runs some transformations and aggregations on it and joins the data with other sources, so that I can generate real-time enrichments/insights based on telemetry data in SQS.
A web beacon is pushing log records to SQS and I want to read these log events in real-time
User Storie(s)
I want to specify credentials securely as Access Key and Access ID
I want to also specify credentials using IAM
I want to specify the queue and region in SQS to read events from
Plugin Type
Realtime Source
This section defines properties that are configurable for this plugin.
Section | User Facing Name | Type | Description | Constraints | Optional? | Default |
|---|---|---|---|---|---|---|
Credentials | Authentication method | Radio button | Either Access Credentials or IAM |
| N | Access Credentials |
Access ID | Textbox | AWS Access ID. Only shown when Authentication method is Access Credentials |
| Y |
| |
Access Key | Password | AWS Secret Access Key. Only shown when Authentication method is Access Credentials |
| Y |
| |
SQS properties | Region | Drop down | Select from a list of available regions where your SQS queue is located |
|
| us-west-1 |
Queue name | Textbox | Specifies the queue name to read from |
|
|
| |
Endpoint | Textbox | Endpoint of the SQS server to connect to. Omit this field to connect to AWS. |
| Yes |
| |
Delete Messages | Drop Down | Delete messages from SQS queue after successfully reading. |
| N | true | |
Wait Time | Number | SQS Long poll wait time. | Valid values 1-20 | Y | 10 | |
Interval | Number | The amount of time to wait between each poll in seconds. The plugin will wait for the duration specified |
| Y | 0 | |
Number of Messages to return | Number | Maximum number of messages to return for each API call. | Valid values 1-10 | Y | 10 |
Batch Sink
This section defines properties that are configurable for this plugin.
Section | User Facing Name | Type | Description | Constraints | Optional? | Default |
|---|---|---|---|---|---|---|
Credentials | Authentication method | Radio button | Either Access Credentials or IAM |
| N | Access Credentials |
Access ID | Textbox | AWS Access ID. Only shown when Authentication method is Access Credentials |
| Y |
| |
Access Key | Password | AWS Secret Access Key. Only shown when Authentication method is Access Credentials |
| Y |
| |
SQS properties | Region | Drop down | Select from a list of available regions where your SQS queue is located |
|
| us-west-1 |
Queue name | Textbox | Specifies the queue name to read from |
|
|
| |
Endpoint | Textbox | Endpoint of the SQS server to connect to. Omit this field to connect to AWS. |
| Yes |
| |
Message format | Select | Either CSV or JSON. Converts the structured record into a CSV or JSON to be sent to SQS |
|
| JSON | |
Delay seconds | Number | The length of time, in seconds, for which a specific message is delayed. | Valid values: 0 to 900. |
|
|
Design / Implementation Tips
Tip #1
Tip #2
Design
Approach(s)
Properties
Security
Limitation(s)
Future Work
Some future work – HYDRATOR-99999
Another future work – HYDRATOR-99999
Test Case(s)
Test case #1
Test case #2
Sample Pipeline
Please attach one or more sample pipeline(s) and associated data.
Pipeline #1
Pipeline #2
Table of Contents
Checklist