Page Comparison

Use Connection

No

6.7.0/0.20.0

Optional. Whether to use a connection. If a connection is used, you do not need to provide the credentials.

Connection

Yes

6.7.0/0.20.0

Optional. Name of the connection to use. Project and service account information will be provided by the connection. You can also use the macro function ${conn(connection_name)}

Project ID

Yes

Optional. Google Cloud Project ID, which uniquely identifies a project. It can be found on the Dashboard in the Google Cloud Platform Console. This is the project that the BigQuery job will run in. BigQuery Job User role on this project must be granted to the specified service account to run the job. If a temporary bucket needs to be created, the bucket will also be created in this project and GCE Storage Bucket Admin role on this project must be granted to the specified service account to create buckets.

Default is auto-detect.

Dataset Project ID

Yes

Optional. Project the dataset belongs to. This is only required if the dataset is not in the same project that the BigQuery job will run in. If no value is given, it will default to the configured Project ID. BigQuery Data Viewer role on this project must be granted to the specified service account to read BigQuery data from this project.

Service Account Type

Yes

6.3.0/0.16.0

Optional. Service account key used for authorization. Select File Path or JSON. If you select File Path, enter the Service Account File Path. If you select JSON, enter the JSON Path. Default is Select File Path.

Service Account File Path

Yes

Optional. Path on the local file system of the service account key used for authorization. Can be set to 'auto-detect' when running on a Dataproc cluster. When running on other clusters, the file must be present on every node in the cluster.

Default is auto-detect.

Service Account JSON

Yes

6.3.0/0.16.0

Optional. Contents of the service account JSON file.

Reference Name

No

Required. Name used to uniquely identify this sink for lineage, annotating metadata, etc.

Dataset

Yes

Required. Dataset the tables belongs to. A dataset is contained within a specific project. Datasets are top-level containers that are used to organize and control access to tables and views. If dataset does not exist, it will be created.

Truncate Table

Yes

Optional. Whether or not to truncate the table before writing to it.

Default is False.

Temporary Bucket Name

Yes

Optional. Google Cloud Storage bucket to store temporary data in. It will be automatically created if it does not exist, but will not be automatically deleted. Temporary data will be deleted after it is loaded into BigQuery. If it is not provided, a unique bucket will be created and then deleted after the run finishes.

Syntax: gs://bucketname

GCS Upload Request Chunk Size

Yes

Optional. GCS upload request chunk size in bytes. Default is 8388608 bytes.

Split Field

Yes

Optional. The name of the field that will be used to determine which table to write to.

Allow flexible schemas in Output

Yes

Optional. When enabled, this sink will write out records with arbitrary schemas. Records may not have a well defined schema depending on the source. When disabled, table schemas must be passed in pipeline arguments.

Update Table Schema

Yes

Optional. Whether the BigQuery table schema should be modified when it does not match the schema expected by the pipeline.

• When this is set to false, any mismatches between the schema expected by the pipeline and the schema in BigQuery will result in pipeline failure.

• When this is set to true, the schema in BigQuery will be updated to match the schema expected by the pipeline, assuming the schemas are compatible.

Compatible changes fall under the following categories:

• The pipeline schema contains nullable fields that do not exist in the BigQuery schema. In this case, the new fields will be added to the BigQuery schema.

• The pipeline schema contains nullable fields that are non-nullable in the BigQuery schema. In this case, the fields will be modified to become nullable in the BigQuery schema.

• The pipeline schema does not contain fields that exist in the BigQuery schema. In this case, those fields in the BigQuery schema will be modified to become nullable.

Incompatible schema changes will result in pipeline failure.

Default is False.

Location

Yes

Optional. The location where the big query datasets will get created. This value is ignored if the dataset or temporary bucket already exist.

Default is US.

Encryption Key Name

Yes

6.5.1/0.18.1

Optional. The GCP customer managed encryption key (CMEK) used to encrypt data written to any bucket, dataset, or table created by the plugin. If the bucket, dataset, or table already exists, this is ignored. More information can be found here.