Optional. Name used to uniquely identify this sink for lineage, annotating metadata, etc.
Optional. Google Cloud Project ID, which uniquely identifies a project. It can be found on the Dashboard in the Google Cloud Platform Console. This is the project that the Dataplex job will run in. If a temporary bucket needs to be created, the service account must have permission in this project to create buckets.
Service Account Type
Optional. Select one of the following options:
Service Account File Path
Optional. Path on the local file system of the service account key used for authorization. Can be set to 'auto-detect' when running on a Dataproc cluster. When running on other clusters, the file must be present on every node in the cluster.
Default is auto-detect.
Service Account JSON
Optional. Content of the service account.
Required. ID of the location in which the Dataplex lake has been created, which can be found on the details page of the lake.
Required. ID of the Dataplex lake, which can be found on the details page of the lake.
Required. ID of the Dataplex zone, which can be found on the details page of the zone.
Required. D of the Dataplex asset. It represents a cloud resource that is being managed within a lake as a member of a zone.
Required. Type of asset selected to ingest the data in Dataplex.
Default is BigQuery Dataset.
BigQuery Dataset properties
Optional. Table to write to. A table contains individual records organized in rows. Each record is composed of columns (also called fields). Every table is defined by a schema that describes the column names, data types, and other information.
Optional. Type of write operation to perform. This can be set to Insert, Update, or Upsert.
Insert. All records will be inserted in destination table.
Update. Records that match on Table Key will be updated in the table. Records that do not match will be dropped.
Upsert. Records that match on Table Key will be updated. Records that do not match will be inserted.
Optional. List of fields that determines relation between tables during Update and Upsert operations.
Optional. Column names and sort order used to choose which input record to update/upsert when there are multiple input records with the same key. For example, if this is set to 'updated_time desc', then if there are multiple input records with the same key, the one with the largest value for 'updated_time' will be applied.
Optional. Partition filter that can be used for partition elimination during Update or Upsert operations. Only use with Update or Upsert operations for tables where Require Partition Filter is enabled. For example, if the table is partitioned and the Partition Filter is '_PARTITIONTIME > "2020-01-01" and _PARTITIONTIME < "2020-03-01"', the update operation will be performed only in the partitions meeting the criteria.
Optional. Whether or not to truncate the table before writing to it. Only use with the Insert operation.
Default is False.
Update Table Schema
Optional. Whether the BigQuery table schema should be modified when it does not match the schema expected by the pipeline.
When set to false, any mismatches between the schema expected by the pipeline and the schema in BigQuery will result in pipeline failure.
When set to true, the schema in BigQuery will be updated to match the schema expected by the pipeline, assuming the schemas are compatible.
Default is False.
Specifies the partitioning type. Can either be Integer, Time, or None. Defaults to Time. This value is ignored if the table already exists.
When set to Time, the table will be created with time partitioning.
When set to Integer, the table will be created with range partitioning.
When set to None, the table will be created without time partitioning.
Default is Time.
Range Start (inclusive)
Optional. For integer partitioning, specifies the start of the range. Only used when table doesn’t exist already, and Partitioning Type is set to Integer.
The start value is inclusive.
Range End (exclusive)
Optional. For integer partitioning, specifies the end of the range. Only used when table doesn’t exist already, and Partitioning Type is set to Integer.
The end value is exclusive.
Optional. For integer partitioning, specifies the partition interval. Only used when table doesn’t exist already, and Partitioning Type is set to Integer.
The interval value must be a positive integer.
Optional. Partitioning column for the BigQuery table. Leave blank if the BigQuery table is an ingestion-time partitioned table.
Require Partition Filter
Optional. Whether to create a table that requires a partition filter. This value is ignored if the table already exists.
When set to true, the table will be created with required partition filter.
When set to false, the table will be created without required partition filter.
Optional. List of fields that determines the sort order of the data. Fields must be of type INT, LONG, STRING, DATE, TIMESTAMP, BOOLEAN, or DECIMAL. Tables cannot be clustered on more than 4 fields. This value is only used when the BigQuery table is automatically created and ignored if the table already exists.
Compatible changes fall under the following categories:
The pipeline schema contains nullable fields that do not exist in the BigQuery schema. In this case, the new fields will be added to the BigQuery schema.
The pipeline schema contains nullable fields that are non-nullable in the BigQuery schema. In this case, the fields will be modified to become nullable in the BigQuery schema.
The pipeline schema does not contain fields that exist in the BigQuery schema. In this case, those fields in the BigQuery schema will be modified to become nullable.
Incompatible schema changes will result in pipeline failure.
Required. Schema of the data to write. If a schema is provided, it must be compatible with the table schema in BigQuery.
Storage Bucket properties
Optional. Table to write to. In case of Asset of type Storage Bucket, a table is a directory where data would be stored and read by Dataplex discover jobs.
Optional. Time format for the output directory that will be appended to the path. For example, the format ' yyyy-MM-dd-HH-mm' will result in a directory of the form '2015-01-01-20-42'. If not specified, directory will be created with default format 'yyyy-MM-dd-HH-mm'.
Optional. Format to write the records in. The format for a raw zone must be one of ‘json’, ‘avro’, ‘parquet’, ‘csv’, or ‘orc’. The format for a curated zone must be one of ‘avro’, ‘orc’, or ‘parquet’. If the format is a macro, only the pre-packaged formats can be used.
Update Dataplex Metadata
Optional. Whether to update Dataplex metadata for the newly created entities. If enabled, the pipeline will automatically copy the output schema to the destination Dataplex entities, and the automated Dataplex Discovery won’t run for them. The user is responsible for the compatibility of the changes applied to the output schema.
Default is False.
Optional. Schema of the data to write. The 'avro' and 'parquet' formats require a schema but other formats do not.