GCP Plugins need a way to store the Service Account details as a blob of text
Description
Using the GCP plugins as they exist today presents a chicken-and-egg problem for pipelines designed to use the Dataproc profile. The GCP plugins take a path as an input string and require that you have the Service Account JSON file on each node of the cluster, but the nodes don't exist before the job starts therefore the pipeline fails.
Need a way to store the service account as a macro so that it can be provided at runtime or added as part of the profile definition.
Release Notes
None
Activity
Show:
Bhooshan Mogal
July 25, 2018, 6:21 PM
This should be really done via integration with secure store as well. I'm moving this to 5.2.0, because the immediate problem for Dataproc will be fixed by making these properties optional.
