SQL Server Batch Source

Reads from a SQL Server using a configurable SQL query. Outputs one record for each row returned by the query.

Use this source when you need to read from a SQL Server. For example, you might want to create daily snapshots of a database table by using this source and writing to a partitioned table on BigQuery.

For information about using this plugin in pipelines, see Microsoft SQL Server Best Practices.

Configuration

Property

Macro Enabled?

Description

Property

Macro Enabled?

Description

JDBC Driver Name

No

Required. Select the JDBC driver to use.

Host

Yes

Required. Host that SQL Server is running on.

Default is localhost.

Port

Yes

Required. Port that SQL Server is listening to. If the port number is specified in the connection string, no request to SQLbrowser is made. When the Port and Instance Name are both specified, the connection is made to the specified port. However, the Instance Name is validated and an error is thrown if it does not match the port.

Default is 1433.

Authentication Type

No

Optional. Indicates which SQL authentication method will be used for the connection. Use 'SQL Login' to connect to a SQL Server using username and password properties. Use 'Active Directory Password' to connect to an Azure SQL Database/Data Warehouse using an Azure AD principal name and password.

Default is SQL Login.

Username

Yes

Optional. User identity for connecting to the specified database.

Password

Yes

Optional. Password to use to connect to the specified database.

Connection Arguments

No

Optional. A list of arbitrary string key/value pairs as connection arguments. These arguments will be passed to the JDBC driver as connection arguments for JDBC drivers that may need additional configurations.

Reference Name

No

Required. Name used to uniquely identify this source for lineage, annotating metadata, etc.

Database

Yes

Required. SQL Server database name.

Import Query

Yes

Required. The SELECT query to use to import data from the specified table. You can specify an arbitrary number of columns to import, or import all columns using *. The Query should contain the ‘$CONDITIONS’ string. For example, ‘SELECT * FROM table WHERE $CONDITIONS’. The ‘$CONDITIONS’ string will be replaced by Split-by Field Name field limits specified by the bounding query. The ‘$CONDITIONS’ string is not required if Number of Splits to Generate is set to 1.

Bounding Query

Yes

Required. Bounding Query should return the minimum and maximum of the values of the Split-by Field Name field. For example, SELECT MIN(id),MAX(id) FROM table. Not required if Number of Splits to Generate is set to 1.

Split-By Field Name

Yes

Field Name which will be used to generate splits. Not required if Number of Splits to Generate is set to one.

Number of Splits to Generate

Yes

Number of splits to generate.

Default is 1.

Instance Name

No

Optional. SQL Server instance name to connect to. When it is not specified, a connection is made to the default instance. For the case where both the instanceName and port are specified, see the notes for port. If you specify a Virtual Network Name in the Server connection property, you cannot use instanceName connection property.

Query Timeout

No

Optional. Number of seconds to wait before a timeout has occurred on a query. The default value is -1, which means infinite timeout. Setting this to 0 also implies to wait indefinitely.

Default is -1.

Connect Timeout

No

Optional. Time in seconds to wait for a connection to the server before terminating the attempt and generating an error.

Default is 0.

Column Encryption

No

Optional. Whether to encrypt data sent between the client and server for encrypted database columns in the SQL server.

Default is Disabled.

Encrypt

No

Optional. Whether to encrypt all data sent between the client and server. This requires that the SQL server has a certificate installed.

Default is No.

Trust Server Certificate

No

Optional. Whether to trust the SQL server certificate without validating it when using SSL encryption for data sent between the client and server.

Default is No.

Workstation ID

No

Optional. Used to identify the specific workstation in various SQL Server profiling and logging tools.

Failover Partner

No

Optional. Name or network address of the SQL Server instance that acts as a failover partner.

Packet Size

No

Optional. Network packet size in bytes to use when communicating with the SQL Server.

Default is -1.

Current Language

No

Optional. Language to use for SQL sessions. The language determines datetime formats and system messages. See sys.syslanguages for the list of installed languages.

Example

You want to read data from SQL Server database named "prod" that is running on "localhost" port 1433, as "sa" user with "Test11" password. Ensure that the driver for SQL Server is installed. You can also provide driver name for some specific driver, otherwise "sqlserver42" will be used, and then configure plugin with:

Property

Value

Property

Value

Reference Name

src1

Driver Name

sqlserver42

Host

localhost

Port

1433

Database

prod

Import Query

select id, name, email, phone from users;

Number of Splits to Generate

1

Username

sa

Password

Test11

For example, if the 'id' column is a primary key of type int and the other columns are non-nullable varchars, output records will have this schema:

field name

type

field name

type

id

int

name

string

email

string

phone

string

Data Type Mapping

SQL Server Data Type

CDAP Schema Data Type

Comments

SQL Server Data Type

CDAP Schema Data Type

Comments

BIGINT

long

 

BINARY

bytes

 

BIT

boolean

 

CHAR

string

 

DATE

datetime

 

DATETIME

datetime (version 1.5.5 and later)

timestamp (version 1.5.4 and earlier)

Users can manually set output schema to map it to timestamp.

DATETIME2

datetime (version 1.5.5 and later)

timestamp (version 1.5.4 and earlier)

Users can manually set output schema to map it to timestamp.

DATETIMEOFFSET

datetime (version 1.5.5 and later)

string (version 1.5.4 and earlier)

Users can manually set output schema to map it to string.

DECIMAL

decimal

 

FLOAT

double

 

IMAGE

bytes

 

INT

int

 

MONEY

decimal

 

NCHAR

string

 

NTEXT

string

 

NUMERIC

decimal

 

NVARCHAR

string

 

NVARCHAR(MAX)

string

 

REAL

float

 

SMALLDATETIME

timestamp

 

SMALLINT

int

 

SMALLMONEY

decimal

 

TEXT

string

 

TIME

time

TIME data type has the accuracy of 100 nanoseconds which is not currently supported. Values of this type will be rounded to microsecond.

TINYINT

int

 

UDT

bytes

UDT types are mapped according to the type they are an alias of. For example, is there is an 'SSN' type that was created as'CREATE TYPE SSN FROM varchar(11);', that type would get mapped to a CDAP string. Common Language Runtime UDTs are mapped to CDAP bytes.

UNIQUEIDENTIFIER

string

 

VARBINARY

bytes

 

VARBINARY(MAX)

bytes

 

VARCHAR

string

 

VARACHAR(MAX)

string

 

XML

string

 

SQLVARIANT

string

 

GEOMETRY

bytes

 

GEOGRAPHY

bytes