Netezza Batch Source

The Netezza Batch source plugin is available in the Hub.

Plugin version: 1.7.0

Reads from Netezza using a configurable SQL query. Outputs one record for each row returned by the query. For example, you might want to create daily snapshots of a database table by using this source and writing to a Redshift table.

Configuration

Property	Macro Enabled?	Version Introduced	Description

Property	Macro Enabled?	Version Introduced	Description
Reference Name	No		Required. Name used to uniquely identify this source for lineage, annotating metadata, etc.
Driver Name	No		Required. Name of the JDBC driver to use. Default is netezza.
Host	Yes		Required. Host that Netezza is running on. Default is localhost.
Port	Yes		Required. Port that Netezza is running on. Default is 5480.
Database	Yes		Required. Netezza database name.
Import Query	Yes		Required. The SELECT query to use to import data from the specified table. You can specify an arbitrary number of columns to import, or import all columns using . The Query should contain the ‘$CONDITIONS’ string. For example, ‘SELECT FROM table WHERE $CONDITIONS’. The ‘$CONDITIONS’ string will be replaced by Split-By Field Name field limits specified by the bounding query. The ‘$CONDITIONS’ string is not required if Number of Splits to Generate is set to one.
Bounding Query	Yes		Required. Bounding Query should return the minimum and maximum of the values of the Split-By Field Name field. For example, ‘SELECT MIN(id),MAX(id) FROM table’. Not required if Number of Splits to Generate is set to 1.
Split-By Field Name	Yes		Optional. Field Name which will be used to generate splits. Not required if Number of Splits to Generate is set to 1.
Number of Splits to Generate	Yes		Optional. Number of splits to generate. Default is 1.
Fetch Size	Yes	6.6.0/1.7.0	Optional. The number of rows to fetch at a time per split. Larger Fetch Size can result in faster import with the trade-off of higher memory usage. Default is 1000.
Username	Yes		Optional. User identity for connecting to the specified database.
Password	Yes		Optional. Password to use to connect to the specified database.
Connection Arguments	Yes		Optional. A list of arbitrary string key/value pairs as connection arguments. These arguments will be passed to the JDBC driver as connection arguments for JDBC drivers that may need additional configurations.

Example

Suppose you want to read data from Netezza database named “prod” that is running on “localhost” port 5480, as “test” user with “testpwsd” password. Ensure that the driver for Netezza is installed. You can also provide driver name for some specific driver, otherwise “netezza” will be used, and then configure plugin with:

Property	Value

Property	Value
Reference Name	`src1`
Driver Name	`netezza`
Host	`localhost`
Port	`5480`
Database	`prod`
Import Query	`select id, name, email, phone from users`
Number of Splits to Generate	`1`
Username	`test`
Password	`testpwsd`

For example, if the ‘id’ column is a primary key of type int and the other columns are non-nullable varchars, output records will have this schema:

field name	type

field name	type
id	int
name	string
email	string
phone	string

Data Type Mapping

Netezza Data Type	CDAP Schema Data Type	Comment

Netezza Data Type	CDAP Schema Data Type	Comment
BOOLEAN	boolean
BYTEINT	int
CHAR	string
DATE	date
NUMERIC, DECIMAL	decimal
FLOAT(1-6)	float	Floating point number with precision of 6 or less
FLOAT(7-16)	double	Floating point number with precision of 7 to 15
REAL	float	Equivalent to FLOAT(6)
DOUBLE PRECISION	double
INTEGER	int
SMALLINT	int
BIGINT	long
NCHAR	string
NVARCHAR	string
TIME	time
TIMETZ, TIME WITH TIME ZONE	string
TIMESTAMP	timestamp
VARCHAR	string
INTERVAL	string
VARBINARY	bytes
ST_GEOMETRY	bytes