support submitting jobs on dataproc driver node groups cluster
Description
While creating the Dataproc cluster, properties yarn:yarn.nodemanager.resource.memory.enforced=false and yarn:yarn.nodemanager.admin-env.SPARK_HOME=$SPARK_HOME are needed.
Note - The single quotes are important in the property when using gcloud CLI to create the cluster ('yarn:yarn.nodemanager.admin-env.SPARK_HOME=$SPARK_HOME') so that the shell doesn't try to resolve the $ locally before submitting.
Release Notes
Added support for Dataproc driver node groups. To use Dataproc driver node groups, when you create the Dataproc cluster, configure the following properties:
yarn:yarn.nodemanager.resource.memory.enforced=false
yarn:yarn.nodemanager.admin-env.SPARK_HOME=$SPARK_HOME
Note: The single quotes are important in the property when using gcloud CLI to create the cluster ('yarn:yarn.nodemanager.admin-env.SPARK_HOME=$SPARK_HOME') so that the shell doesn't try to resolve the $ locally before submitting.
Activity
Show:
Fixed
Pinned fields
Click on the next to a field label to start pinning.
While creating the Dataproc cluster, properties
yarn:yarn.nodemanager.resource.memory.enforced=falseandyarn:yarn.nodemanager.admin-env.SPARK_HOME=$SPARK_HOMEare needed.Note - The single quotes are important in the property when using gcloud CLI to create the cluster (
'yarn:yarn.nodemanager.admin-env.SPARK_HOME=$SPARK_HOME') so that the shell doesn't try to resolve the$locally before submitting.