Google Cloud Storage Argument Setter Action

The Google Cloud Storage Argument Setter action plugin was introduced in CDAP 6.2.3.

Plugin version: 0.22.0

Fetch a JSON file from Google Cloud Storage to set arguments in the pipeline.

The plugin provides the ability to map JSON properties as pipeline arguments name and columns values as pipeline arguments.

This is most commonly used when the structure of a pipeline is static, and its configuration needs to be managed outside the pipeline.

The JSON file must contain arguments in a list:

{
    "arguments" : [
        { "name" : "argument name", type:"type", value" :"argument value"},
        { "name" : "argument1 name",type:"type", "value" :"argument1 value"}
    ]
}

Where type can be Schema, Int, Float, Double, Short, String, Char, Array, or Map.

Credentials

If the plugin is run on a Google Cloud Dataproc cluster, the service account key does not need to be provided and can be set to ‘auto-detect’. Credentials will be automatically read from the cluster environment.

If the plugin is not run on a Dataproc cluster, the path to a service account key must be provided. The service account key can be found on the Dashboard in the Cloud Platform Console. Make sure the account key has permission to access BigQuery and Google Cloud Storage. The service account key file needs to be available on every node in your cluster and must be readable by all users running the job.

Configuration

Property	Macro Enabled?	Description

Property	Macro Enabled?	Description
Project ID	Yes	Optional. Google Cloud Project ID, which uniquely identifies a project. It can be found on the Dashboard in the Google Cloud Platform Console. This is the project that the BigQuery job will run in. If a temporary bucket needs to be created, the service account must have permission in this project to create buckets. Default is auto-detect.
Path	Yes	Required. GCS Path to the file containing the arguments.
Provide Service Account	Yes	Optional. Provide service account as JSON or File Path. When it is set to JSON, the content of service account key needs to be copied. When it is set to File Path, the service Account file path needs to be specified. Default is File Path.
Service Account Type	Yes	Optional. Select one of the following options: File Path. File path where the service account is located. JSON. JSON content of the service account.
Service Account File Path	Yes	Optional. Path on the local file system of the service account key used for authorization. Can be set to ‘auto-detect’ when running on a Dataproc cluster. When running on other clusters, the file must be present on every node in the cluster. Default is auto-detect.
Service Account JSON	Yes	Optional. The content of the service account.

https://cdap.atlassian.net/wiki/spaces/DOCS/pages/1168343055

https://cdap.atlassian.net/wiki/spaces/DOCS/pages/382043060

Credentials

Configuration

Related Topics