Introduction

Amazon Aurora is a Mysql and Postgres compatible database offered as a service. Users will have needs to write to AuroraDB or read from AuroraDB

Use case(s)

• Users would like to batch build a data pipeline to read complete table from Amazon Aurora DB instance and write to BigTable. 
• Users would like to batch build a data pipeline to perform upserts on AuroraDB tables in batch 
• Users should get relevant information from the tool tip while configuring the AuroraDB source and AuroraDB sink
  • The tool tip for the connection string should be customized specific to the database. 
  • The tool tip should describe accurately what each field is used for
• Users should get field level lineage for the source and sink that is being used
• Reference documentation be available from the source and sink plugins

Deliverables 

Relevant links 

• Data-integrations org: https://github.com/data-integrations/
• Field level lineage: https://docs.cdap.io/cdap/6.0.0-SNAPSHOT/en/developer-manual/metadata/field-lineage.html
• Integration test repos: https://github.com/caskdata/cdap-integration-tests

Plugin Type

• Batch Source
• Batch Sink 
• Real-time Source
• Real-time Sink
• Action
• Post-Run Action
• Aggregate
• Join
• Spark Model
• Spark Compute

Design / Implementation Tips

• Amazon's SDK for Relational Database Service (part of which Aurora DB is)  is only useful for programmatically managing database clusters and instances (creating and deleting databases, changing instance state, managing users, creating dumps to S3, etc). From client perspective, all you need to work with Aurora DB is JDBC driver.
• Since client can connect to AuroraDB cluster using JDBC driver it is rational to reuse existing database source and sink functionality from database-plugins project.

Design

• For better user experience it is suggested to create separate plugins for AuroraDB MySQL and PostgreSQL. 

Future Work

• Amazon AuroraDB MySQL plugin
• Amazon AuroraDB PostgreSQL plugin