Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. The SQS plugin in CDAP will enable ETL developers to create streaming pipelines that read events from SQS queues in realtime and process them.
Use case(s)
As a user, I would like to create a streaming pipeline that reads events from Amazon SQS, runs some transformations and aggregations on it and joins the data with other sources, so that I can generate real-time enrichments/insights based on telemetry data in SQS.
A web beacon is pushing log records to SQS and I want to read these log events in real-time
User Storie(s)
I want to specify credentials securely as Access Key and Access ID
I want to also specify credentials using IAM
I want to specify the queue and region in SQS to read events from
Plugin Type
Batch Source
Batch Sink
Real-time Source
Real-time Sink
Action
Post-Run Action
Aggregate
Join
Spark Model
Spark Compute
Realtime Source
This section defines properties that are configurable for this plugin.
Section
User Facing Name
Type
Description
Constraints
Optional?
Default
Credentials
Authentication method
Radio button
Either Access Credentials or IAM
N
Access Credentials
Access ID
Textbox
AWS Access ID. Only shown when Authentication method is Access Credentials
Y
Access Key
Password
AWS Secret Access Key. Only shown when Authentication method is Access Credentials
Y
SQS properties
Region
Drop down
Select from a list of available regions where your SQS queue is located
us-west-1
Queue name
Textbox
Specifies the queue name to read from
Endpoint
Textbox
Endpoint of the SQS server to connect to. Omit this field to connect to AWS.
Yes
Batch Sink
This section defines properties that are configurable for this plugin.
Section
User Facing Name
Type
Description
Constraints
Optional?
Default
Credentials
Authentication method
Radio button
Either Access Credentials or IAM
N
Access Credentials
Access ID
Textbox
AWS Access ID. Only shown when Authentication method is Access Credentials
Y
Access Key
Password
AWS Secret Access Key. Only shown when Authentication method is Access Credentials
Y
SQS properties
Region
Drop down
Select from a list of available regions where your SQS queue is located
us-west-1
Queue name
Textbox
Specifies the queue name to read from
Endpoint
Textbox
Endpoint of the SQS server to connect to. Omit this field to connect to AWS.
Yes
Message format
Select
Either CSV or JSON. Converts the structured record into a CSV or JSON to be sent to SQS
JSON
Delay seconds
Number
The length of time, in seconds, for which a specific message is delayed.
Valid values: 0 to 900.
Design / Implementation Tips
Tip #1
Tip #2
Design
Approach(s)
Properties
Security
Limitation(s)
Future Work
Some future work – HYDRATOR-99999
Another future work – HYDRATOR-99999
Test Case(s)
Test case #1
Test case #2
Sample Pipeline
Please attach one or more sample pipeline(s) and associated data.