Propagate labels provided in the compute profile to the dataproc job
Description
Release Notes
Users can set common metadata labels for Dataproc Cluster and Jobs using the "Common Labels" property in the Ephemeral Dataproc compute profile.
Users can set labels for the Dataproc Jobs using the "Common Labels" property in the Existing Dataproc compute profile.
Users can set a pipeline runtime argument with key "system.profile.properties.labels" and value representing labels in the following format: "key1|value1;key2|value2". This will override the common labels set in the compute profile for pipeline runs.
relates to
Activity
Show:
Arjan Bal August 10, 2023 at 4:45 AM
The new wording sounds better, thanks!
Olivia Emery August 9, 2023 at 8:53 PM
I propose this new wording for the release note. Feel free to make changes.
You can set common metadata labels for Dataproc clusters and jobs using the Common Labels property in the Ephemeral Dataproc compute profile.
You can set labels for the Dataproc jobs using the Common Labels property in the Existing Dataproc compute profile.
You can set a pipeline runtime argument with the key
system.profile.properties.labelsand a value representing the labels in the following format:key1|value1;key2|value2. This setting overrides the common labels set in the compute profile for pipeline runs.
Fixed
Pinned fields
Click on the next to a field label to start pinning.
Created June 21, 2023 at 1:27 PM
Updated August 10, 2023 at 4:45 AM
Resolved July 21, 2023 at 7:26 AM
Steps to reproduce:
Create a compute profile with some random label using runtime arg
system.profile.properties.clusterLabels = key1|value1.Run the pipeline using the compute profile created in step 1.
Verify the label
key1|value1is correctly propagated in the dataproc cluster.Verify the label
key1|value1is not present in the dataproc job.We need to propagate the cluster labels in the compute profile to dataproc job as well here.