Introduction

Data loading in Snowflake (https://docs.snowflake.net/manuals/user-guide/data-load-overview.html) is performed in two, separate steps. The first step is to load the files into a staging area, and the second step is to trigger the upload with a COPY INTO step. This action plugin will allow files to be written to the staging area for further processingtrigger the loading of data from a Snowflake staging area or s3 into snowflake.

Use case(s)

• As a financial services company, I would like to build an analytics pipeline to push data into the Snowflake. I would leverage Hydrator, and use this plugin as a sink to create files in the staging area. Finally, there would be an action plugin to trigger the loading of the data (this plugin).

User Storie(s)

• I would like to load processed data to from the correct Snowflake snowflake staging area in the proper format.into Snowflake.
• I would like to load data from s3 into snowflake
• I would like to provide IAM or keys for connecting to s3

Plugin Type

•  Batch Sink 
•  Real-time Sink
• Action

Configurables

This section defines properties that are configurable for this plugin. 

Design / Implementation Tips

• See this document for an overview: https://docs.snowflake.net/manuals/user-guide/data-load-overview.html
• See this for details: https://docs.snowflake.net/manuals/user-guide/data-load-copy-into.html

Design

Approach(s)

Properties

Security

Limitation(s)

Future Work

• Some future work – HYDRATOR-99999
• Another future work – HYDRATOR-99999

Test Case(s)

• Test case #1
• Test case #2

Sample Pipeline

Please attach one or more sample pipeline(s) and associated data. 

Pipeline #1

Pipeline #2