MariaDB Batch Source

The MariaDB Batch source plugin is available in the Hub.

Plugin version: 1.7.0

Reads from a MariaDB instance using a configurable SQL query. Outputs one record for each row returned by the query.

Use this source when you need to read from a MariaDB instance. For example, you may want to create daily snapshots of a database table by using this source and writing to a BigQuery table.

Configuration

Property

Macro Enabled?

Version Introduced

Description

Property

Macro Enabled?

Version Introduced

Description

Reference Name

No

 

Required. This will be used to uniquely identify this source for lineage, annotating metadata, etc.

Host

Yes

 

Required. Host that MariaDB is running on.

Default is localhost.

Port

Yes

 

Required. Port that MariaDB is running on.

Default is 3306.

Database

Yes

 

Required. MariaDB database name.

Import Query

Yes

 

Required. The SELECT query to use to import data from the specified table. You can specify an arbitrary number of columns to import, or import all columns using *. The Query should contain the ‘$CONDITIONS’ string. For example, ‘SELECT * FROM table WHERE $CONDITIONS’. The ‘$CONDITIONS’ string will be replaced by Split-By Field Name field limits specified by the bounding query. The ‘$CONDITIONS’ string is not required if Number of Splits to Generate is set to 1.

Bounding Query

Yes

 

Required. Bounding Query should return the min and max of the values of the ‘splitBy’ field. For example, ‘SELECT MIN(id),MAX(id) FROM table’. Not required if numSplits is set to one.

Split-By Field Name

Yes

 

Optional. Field Name which will be used to generate splits. Not required if Number of Splits to Generate is set to 1.

Number of Splits to Generate

Yes

 

Optional. Number of splits to generate.

Fetch Size

Yes

6.6.0/1.7.0

Optional. The number of rows to fetch at a time per split. Larger Fetch Size can result in faster import with the trade-off of higher memory usage.

Default is 1000.

Username

Yes

 

Optional. User identity for connecting to the specified database.

Password

Yes

 

Optional. Password to use to connect to the specified database.

Use SSL

No

 

Optional. Turns on SSL encryption. The connection will fail if SSL is not available.

Default is if available.

Keystore URL

No

 

Optional. URL to the client certificate KeyStore (if not specified, use defaults). Must be accessible at the same location on host where CDAP Master is running and all hosts on which at least one HDFS, MapReduce, or YARN daemon role is running.

Keystore Password

No

 

Optional. Password for the client certificates KeyStore.

Truststore URL

No

 

Optional. URL to the trusted root certificate KeyStore (if not specified, use defaults). Must be accessible at the same location on host where CDAP Master is running and all hosts on which at least one HDFS, MapReduce, or YARN daemon role is running.

Truststore Password

No

 

Optional. Password for the trusted root certificates KeyStore.

Driver Name

No

 

Optional. Name of the JDBC driver to use.

Default is mariadb.

Use Compression

No

 

Optional. Use zlib compression when communicating with the server. Select this option for WAN connections.

Default is No.

Use ANSI Quotes

No

 

Optional. Treats “ as an identifier quote character and not as a string quote character.

Connection Arguments

No

 

Optional. A list of arbitrary string key/value pairs as connection arguments. These arguments will be passed to the JDBC driver as connection arguments for JDBC drivers that may need additional configurations.

Auto Reconnect

No

 

Optional. Should the driver try to re-establish stale and/or dead connections.

Default is No.

Output Schema

No

 

The schema of records output by the source. This will be used in place of whatever schema comes back from the query. However, it must match the schema that comes back from the query, except it can mark fields as nullable and can contain a subset of the fields.

Data Type Mapping

MariaDB Data Type

CDAP Schema Data Type

Comments

MariaDB Data Type

CDAP Schema Data Type

Comments

TINYINT

int

 

BOOLEAN, BOOL

boolean

 

SMALLINT

int

 

MEDIUMINT

int

 

INT, INTEGER

int

 

BIGINT

long

 

DECIMAL, DEC, NUMERIC, FIXED

decimal

 

FLOAT

float

 

DOUBLE, DOUBLE PRECISION, REAL

decimal

 

BIT

boolean

 

CHAR

string

 

VARCHAR

string

 

BINARY

bytes

 

CHAR BYTE

bytes

 

VARBINARY

bytes

 

TINYBLOB

bytes

 

BLOB

bytes

 

MEDIUMBLOB

bytes

 

LONGBLOB

bytes

 

TINYTEXT

string

 

TEXT

string

 

MEDIUMTEXT

string

 

LONGTEXT

string

 

JSON

string

In MariaDB it is alias to LONGTEXT

ENUM

string

Mapping to String by default

SET

string

 

DATE

date

 

TIME

time_micros

 

DATETIME

timestamp_micros

 

TIMESTAMP

timestamp_micros

 

YEAR

date

 

Example

Suppose you want to read data from MariaDB database named “prod” that is running on “localhost” port 3306, as “root” user with “root” password, then configure plugin with:

Property

Value

Property

Value

Reference Name

src1

Driver Name

mariadb

Host

localhost

Port

3306

Database

prod

Import Query

select id, name, email, phone from users;

Number of Splits to Generate

1

Username

root

Password

root

For example, if the ‘id’ column is a primary key of type int and the other columns are non-nullable varchars, output records will have this schema:

Field Name

Type

Field Name

Type

id

int

name

string

email

string

phone

string

 

Created in 2020 by Google Inc.