...
Property | Macro Enabled? | Version Introduced | Description |
|---|---|---|---|
Project ID | Yes | Optional. Google Cloud Project ID, which uniquely identifies a project. It can be found on the Dashboard in the Google Cloud Platform Console. Default is auto-detect. | |
Source Path | Yes | Required. Path to a source object or directory. | |
Destination Path | Yes | Required. Path to the destination. The bucket must already exist. | |
Move All Subdirectories | Yes | Optional. If the source is a directory, move all subdirectories. Default is false. | |
Overwrite Existing Files | Yes | Optional. Whether to overwrite existing files during the move. If this is set to false and an existing file would be overwritten, the pipeline will fail. This setting does not protect against race conditions. If a file is written to the destination while this plugin is running, that file may still get overwritten. Default is false. | |
Service Account Type | Yes | Optional. Select one of the following options:
| |
Service Account File Path | Yes | Optional. Path on the local file system of the service account key used for authorization. Can be set to 'auto-detect' when running on a Dataproc cluster. When running on other clusters, the file must be present on every node in the cluster. Default is auto-detect. | |
Service Account JSON | Yes | Optional. Contents of the service account JSON file. | |
Location | Yes | 6.5.1/0.18.1 | Optional. The location where the GCS buckets will get created. This value is ignored if the bucket already exists. Default is US. |
Encryption Key Name | Yes | 6.5.1/0.18.1 | Optional. The GCP customer managed encryption key (CMEK) used to encrypt data written to any bucket created by the plugin. If the bucket already exists, this is ignored. More information can be found here. |
Read Timeout | Yes | 6.109.1/0.2322.16 | Optional. The timeout in seconds to read data from an established HTTP connection (Default value is 20). For performing copy/move operation on large files in GCS buckets, a higher timeout might be needed. Setting it to 0 implies infinite timeout (no limit on the timeout). |
...