Google CloudSQL MySQL Sink

Writes records to a CloudSQL MySQL table. Each record will be written to a row in the table. For example, suppose you periodically build a recommendation model for products on your online store. The model is stored in a GCS bucket and you want to export the contents of the bucket to a CloudSQL MySQL table where it can be served to your users.

Column names would be autodetected from input schema.

Configuration

Property

Macro Enabled?

Version Introduced

Description

Property

Macro Enabled?

Version Introduced

Description

Use Connection

No

6.7.0/1.8.0

Optional. Whether to use an existing connection. If you use a connection, connection related properties do not appear in the plugin properties.

Connection

Yes

6.7.0/1.8.0

Optional. Name of the connection to use. Project and service account information will be provided by the connection. You can also use the macro function ${conn(connection_name)}

JDBC Driver Name

No

 

Required. Name of the JDBC driver to use.

Default is cloudsql-mysql.

CloudSQL Instance Type

No

 

Optional. Whether the CloudSQL instance to connect to is private or public. 

Default is Public.

Connection Name

Yes
(6.10.0)

 

Required. The CloudSQL instance to connect to in the format <PROJECT_ID>:<REGION>:<INSTANCE_NAME>. Can be found in the instance Overview page.

Port

Yes

6.9.0/1.10.5

Optional. Port that MySQL is running on.

Default is 3306.

Username

Yes

 

Optional. User identity for connecting to the specified database.

Password

Yes

 

Optional. Password to use to connect to the specified database.

Connection Arguments

Yes

 

Optional. A list of arbitrary string key/value pairs as connection arguments. These arguments will be passed to the JDBC driver as connection arguments for JDBC drivers that may need additional configurations.

Reference Name

No

 

Required. Name used to uniquely identify this sink for lineage, annotating metadata, etc.

Database

Yes
(6.10.0/1.10.5)

 

Required. MySQL database name.

Table Name

Yes

 

Required. Name of the table to export to. Table must exist prior to running the pipeline.

Transaction Isolation Level

No

 

Optional. Transaction isolation level for queries run by this sink.

Default is TRANSACTION_REPEATABLE_READ.

Connection Timeout

No

 

Optional. The timeout value (in seconds) used for socket connect operations. If connecting to the server takes longer than this value, the connection is broken. A value of 0 means that it is disabled.

Default is 10 seconds.

Data Type Mapping

MySQL Data Type

CDAP Schema Data Type

MySQL Data Type

CDAP Schema Data Type

BIT

boolean

TINYINT

int

BOOL, BOOLEAN

boolean

SMALLINT

int

MEDIUMINT

double

INT, INTEGER

int

BIGINT

long

FLOAT

float

DOUBLE

double

DECIMAL

decimal

DATE

date

DATETIME

timestamp

TIMESTAMP

timestamp

TIME

time

YEAR

date

CHAR

string

VARCHAR

string

BINARY

bytes

VARBINARY

bytes

TINYBLOB

bytes

TINYTEXT

string

BLOB

bytes

TEXT

string

MEDIUMBLOB

bytes

MEDIUMTEXT

string

LONGBLOB

bytes

LONGTEXT

string

ENUM

string

SET

string

Examples

Connecting to a public CloudSQL MySQL instance

You want to write output records to “users” table of CloudSQL MySQL database named “prod”, as “root” user with “root” password. Get the latest version of the CloudSQL socket factory jar with driver and dependencies here, and then configure plugin with:

Property

Value

Property

Value

Reference Name

sink1

Driver Name

cloudsql-mysql

Database

prod

Username

root

Password

root

CloudSQL Instance Type

Public

Connection Name

[PROJECT_ID]:[REGION]:[INSTANCE_NAME]

Table Name

users

Connecting to a private CloudSQL MySQL instance

If you want to connect to a private CloudSQL MySQL instance, create a Compute Engine VM that runs the CloudSQL Proxy docker image using the following command:

# Set the environment variables export PROJECT=[project_id] export REGION=[vm-region] export ZONE=`gcloud compute zones list --filter="name=${REGION}" --limit 1 --uri --project=${PROJECT}| sed 's/.*\///'` export SUBNET=[vpc-subnet-name] export NAME=[gce-vm-name] export MYSQL_CONN=[mysql-instance-connection-name] # Create a Compute Engine VM gcloud beta compute --project=${PROJECT_ID} instances create ${INSTANCE_NAME} --zone=${ZONE} --machine-type=g1-small --subnet=${SUBNE} --no-address --metadata=startup-script="docker run -d -p 0.0.0.0:3306:3306 gcr.io/cloudsql-docker/gce-proxy:1.16 /cloud_sql_proxy -instances=${MYSQL_CONNECTION_NAME}=tcp:0.0.0.0:3306" --maintenance-policy=MIGRATE --scopes=https://www.googleapis.com/auth/cloud-platform --image=cos-69-10895-385-0 --image-project=cos-cloud

Optionally, you can promote the internal IP address of the VM running the Proxy image to a static IP using:

# Get the VM internal IP export IP=`gcloud compute instances describe ${NAME} --zone ${ZONE} | grep "networkIP" | awk '{print $2}'` # Promote the VM internal IP to static IP gcloud compute addresses create mysql-proxy --addresses ${IP} --region ${REGION} --subnet ${SUBNET}

Get the latest version of the CloudSQL socket factory jar with driver and dependencies from here, and then configure plugin with:

Property