Plugin version: 1.79.0
Reads from a PostgreSQL using a configurable SQL query. Outputs one record for each row returned by the query. For example, you might want to create daily snapshots of a database table by using this source and writing to a Redshift table.
...
Property | Macro Enabled? | Version Introduced | Description |
|---|---|---|---|
Use connection | No | 6.7.0/1.8.0 | Optional. Whether to use a connection. If a connection is used, you do not need to provide the credentials. |
Connection | Yes | 6.7.0/1.8.0 | Required. Name of the connection to use. Project and service account information will be provided by the connection. You also can use the macro function ${conn(connection-name)}. |
JDBC Driver Name | No | Required. Select the JDBC driver to use. Default is postgresql. | |
Host | Yes | Required. Host that PostgreSQL is running on. Default is localhost. | |
Port | Yes | Required. Port that PostgreSQL is running on. Default is 5432. | |
Username | Yes | Optional. User identity for connecting to the specified database. | |
Password | Yes | Optional. Password to use to connect to the specified database. | |
Connection Arguments | Yes | Optional. A list of arbitrary string key/value pairs as connection arguments. These arguments will be passed to the JDBC driver as connection arguments for JDBC drivers that may need additional configurations. | |
Database | Yes | Required. PostgreSQL database name. | |
Reference Name | No | Required. Name used to uniquely identify this source for lineage, annotating metadata, etc. | |
Import Query | Yes | Required. The SELECT query to use to import data from the specified table. You can specify an arbitrary number of columns to import, or import all columns using *. The Query should contain the '$CONDITIONS' string. For example, 'SELECT * FROM table WHERE $CONDITIONS'. The '$CONDITIONS' string will be replaced by Split-By Field Name field limits specified by the bounding query. The '$CONDITIONS' string is not required if Number of Splits to Generate is set to 1. | |
Bounding Query | Yes | Optional. Bounding Query should return the min and max of the values of the Split-By Field Name field. For example, 'SELECT MIN(id),MAX(id) FROM table'. Not required if Number of Splits to Generate is set to 1. | |
Split-By Field Name | Yes | Optional. Field Name which will be used to generate splits. Not required if Number of Splits to Generate is set to 1. | |
Number of Splits to Generate | Yes | Optional. Number of splits to generate. Default is 1. | |
Fetch Size | Yes | 6.6.0/1.7.0 | Optional. The number of rows to fetch at a time per split. Larger Fetch Size can result in faster import with the trade-off of higher memory usage. Default is 1000. |
Connection Timeout | No | Optional. The timeout value used for socket connect operations. If connecting to the server takes longer than this value, the connection is broken.The timeout is specified in seconds and a value of zero means that it is disabled. Default is 100. |
...