Google Dataproc

Cloud Dataproc is a Google Cloud Platform (GCP) service that manages Hadoop and Spark clusters in the cloud and can be used to create large clusters quickly. The Google Dataproc provisioner simply calls the Cloud Dataproc APIs to create and delete clusters in your GCP account. The provisioner exposes several configuration settings that control what type of cluster is created.

Version compatibility

Problem: The version of your CDAP environment might not be compatible with the version of your Dataproc cluster.

Recommended: Upgrade to the latest CDAP version and use one of the supported Dataproc versions.

Earlier versions of CDAP are only compatible with unsupported versions of Dataproc. Dataproc does not provide updates and support for clusters created with these versions. Although you can continue running a cluster that was created with an unsupported version, we recommend replacing it with one created with a supported version.

CDAP version	Dataproc version

CDAP version	Dataproc version
6.10	2.1, 2.0 *
6.9	2.1, 2.0, 1.5 *
6.7, 6.8	2.0, 1.5 *
6.4-6.6	2.0 ^, 1.3 ^*
6.1-6.3	1.3^**

^* CDAP versions 6.4 and later are compatible with supported versions of Dataproc. Unless specific OS features are needed, the recommended practice is to specify the major.minor image version.

^** CDAP versions 6.1 to 6.6 are compatible with unsupported Dataproc version 1.3.

Best Practices

Configurations

Recommended: When you create a static cluster for your pipelines, use the following configurations.

Parameters

Parameters
`yarn.nodemanager.delete.debug-delay-sec`	Retains YARN logs. Recommended value: `86400` (equivalent to one day)
`yarn.nodemanager.pmem-check-enabled`	Enables YARN to check for physical memory limits and kill containers if they go beyond physical memory. Recommended value: `false`
`yarn.nodemanager.vmem-check-enabled`	Enables YARN to check for virtual memory limits and kill containers if they go beyond physical memory. Recommended value: `false`.

Account Information

Project ID

A GCP project ID must be provided. This will be the project that the Cloud Dataproc cluster is created in. The project must have the Cloud Dataproc APIs enabled.

Creator Service Account Key

The service account key provided to the provisioner must have rights to access the Cloud Dataproc APIs and the Google Compute Engine APIs. Since your account key is sensitive, we recommend that you provide your account key through the CDAP Secure Storage by adding a secure key with the Microservices. After you create the secure key, you can add it to a namespace or system compute profile. For a namespace compute profile, click the shield icon and select the secure key. For a system compute profile, type the name of the key in the Secure Account Key field to add the secure key to the compute profile.

General Settings

Region

A region is a specific geographical location where you can host resources, like the compute nodes for your Cloud Dataproc cluster.

Zone

A zone is an isolated location within a region.

Network

This is the VPC network in your GCP project that will be used when creating a Cloud Dataproc cluster.

Network Host Project ID

Google Cloud Project ID, which uniquely identifies the project where the network resides. This can be left blank if the network resides in the same project as specified in the Project ID. In the case of shared VPC, this must be set to the host Project ID where the network resides.

Subnet

Subnet to use when creating clusters. The subnet must be within the given network and it must be for the region that the zone is in. If this is left blank, a subnet will automatically be chosen based on the network and zone.

Runner Service Account

Name of the service account of the Dataproc virtual machines (VM) that are used to programs. If none is given, the default Compute Engine service account will be used.

Master Nodes Settings

Size settings control how big of a cluster to create.

Number of Masters

The number of master nodes to have in your cluster. Master nodes contain the YARN Resource Manager, HDFS NameNode, and will be the node that CDAP will connect to when executing jobs. Must be set to either 1 or 3.

Default is 1.

Master Machine Type

The type of master machine to use. Select one of the following machine types:

n1
n2
n2d
e2

Default is e2 (6.7.2)

Default is n2 (6.7.1)

Default is n1 (6.7.0 and earlier)

Master Cores

The number of virtual cores to allocate to each master node.

Default is 2.

Master Memory

The amount of memory in gigabytes to allocate to each master node.

Default is 8 GB.

Master Disk Size (GB)

The disk size in gigabytes to allocate for the master node.

Default is 1000 GB

Master Disk Type

Type of boot disk for the master node:

Standard Persistent Disk
SSD Persistent Disk

Default is Standard Persistent Disk

Worker Nodes Settings

Worker Machine Type

The type of worker machine to use. Select one of the following machine types:

n1
n2
n2d
e2

Default is e2 (6.7.2)

Default is n2 (6.7.1)

Default is n1 (6.7.0 and earlier)

Worker Cores

The number of virtual cores to allocate for each worker node.

Default is 2.

Worker Memory (GB)

The amount of memory in gigabytes to allocate to each worker node.

Default is 8 GB.

Worker Disk Size (GB)

The disk size in gigabytes to allocate for each worker node.

Default is 1000 GB

Worker Disk Type

Type of boot disk for the worker node:

Standard Persistent Disk
SSD Persistent Disk

Default is Standard Persistent Disk.

Number of Cluster Workers Settings

Enable Predefined Dataproc Autoscaling

Enable Predefined Dataproc Autoscaling was introduced in CDAP 6.6.0

Defines whether the Dataproc autoscaling with predefined Autoscaling policy by CDAP will be enabled.

When you enable predefined autoscaling:

The properties Number of primary workers, Number of secondary workers, and Autoscaling policy are not considered.
The worker machine type/configuration is the same as the chosen profile's.
Turning off the Use predefined Autoscaling toggle disables predefined autoscaling, and then runs the original behavior of the profile.

You can also use a Runtime argument to enable autoscaling for a pipeline run.

system.profile.properties.enablePredefinedAutoScaling = true.

Number of Primary Workers

Worker nodes contain a YARN NodeManager and an HDFS DataNode.

Default is 2.

Number of Secondary Workers

Secondary worker nodes contain a YARN NodeManager, but not an HDFS DataNode. This is normally set to zero unless an autoscale policy requires it to be higher.

Autoscaling Policy

Specify the Autoscaling Policy ID (name) or the resource URI.

For information about configuring and using Dataproc autoscaling to automatically and dynamically resize clusters to meet workload demands, see the Autoscaling clusters guide.

Recommended: Use autoscaling policies for increasing the cluster size, not for decreasing the size. Decreasing the cluster size with autoscaling removes nodes that hold intermediate data, which might cause your pipelines to run slowly or fail.

Cluster Metadata Settings

Metadata

Additional metadata for instances that run in your cluster. You can typically use it for tracking billing and chargebacks. For more information, see Cluster metadata.

Network Tags

Assign Network tags to apply firewall rules to the specific nodes of a cluster. Network tags must start with a lowercase letter and can contain lowercase letters, numbers, and hyphens. Tags must end with a lowercase letter or number.

Shielded VMs

Shielded VM settings were introduced in CDAP 6.5.0.

Enable Secure Boot

Defines whether the Dataproc VMs have Secure Boot enabled.

Default is False.

Enable vTPM

Defines whether the Dataproc VMs have the virtual Trusted Platform Module (vTPM) enabled.

Default is False.

Enable Integrity Monitoring

Defines whether Dataproc VMs have integrity monitoring enabled.

Default is False.

Advanced Settings

Image Version

The Dataproc image version. If none is given, one will automatically be chosen. If custom image URI is specified, this field will be ignored.

Custom Image URI

Dataproc image URI. If the URI is not specified, it will be inferred from the Image Version.

Staging Bucket

Google Cloud Storage bucket used to stage job dependencies and config files for running pipelines in Google Cloud Dataproc.

Temp Bucket

Note: Temp Bucket config was introduced in CDAP 6.9.2.

Google Cloud Storage bucket used to store ephemeral cluster and jobs data, such as Spark and MapReduce history files in Google Cloud Dataproc.

Encryption Key Name

The GCP customer managed encryption key (CMEK) name used by Cloud Dataproc.

OAuth Scopes

The OAuth 2.0 scopes that you might need to request to access Google APIs, depending on the level of access you need. Google Cloud Platform Scope is always included.

Initialization Actions

A list of scripts to be executed during initialization of the cluster. Init actions should be placed on Google Cloud Storage.

Cluster Properties

Cluster properties used to override default configuration properties for the Hadoop services. The applicable key-value pairs can found here.

Common Labels

A label is a key-value pair that helps you organize your Google Cloud Dataproc clusters and jobs. You can attach a label to each resource, and then filter the resources based on their labels. Information about labels is forwarded to the billing system, so customers can break down your billing charges by label.

Specifies labels for the Dataproc clusters and jobs being created.

Cluster Labels

Specifies labels for the Dataproc cluster being created.

Max Idle Time

Configure Dataproc to delete the cluster if it has been idle for longer than this many minutes. Clusters are normally deleted directly after a run ends, but this delete may fail in rare situations. For example, if permissions are revoked in the middle of a run, or if there is a Dataproc outage. Use this to ensure that clusters are eventually deleted even if the instance is unable to delete the cluster for any reason.

Default is 30 minutes (starting in CDAP 6.6.0)

Skip Cluster Delete

Whether to skip cluster deletion at the end of a run. Clusters will need to be deleted manually. This should only be used when debugging a failed run.

Default is False.

Enable Stack Driver Logging Integration

Enable or disable Stackdriver logging integration.

Default is True.

Enable Stack Driver Monitoring Integration

Enable or disable Stackdriver monitoring integration.

Default is True.

Enable Component Gateway

Enable Component Gateway to allow access to cluster UIs like the YARN ResourceManager and Spark HistoryServer.

Default is False.

Prefer External IP

When the system is running on Google Cloud Platform in the same network as the cluster, it will normally use the internal IP when communicating with the cluster. Set to True to always use the external IP.

Default is False.

Polling Settings

Polling settings control how often cluster status should be polled when creating and deleting clusters. You may want to change these settings if you have a lot of pipelines scheduled to run at the same time using the same GCP account.

Create Poll Delay

The number of seconds to wait after creating a cluster to begin polling to see if the cluster has been created.

Default is 60 seconds.

Create Poll Jitter

Maximum amount of random jitter in seconds to add to the create poll delay. This is used to prevent a lot of simultaneous API calls against your GCP account when you have a lot of pipelines that are scheduled to run at the exact same time.

Default is 20 seconds.

Delete Poll Delay

The number of seconds to wait after deleting a cluster to begin polling to see if the cluster has been deleted.

Default is 30 seconds.

Poll Interval

The number of seconds to wait in between polls for cluster status.

Default is 2.

Dataproc profile UI properties mapped to JSON properties

Dataproc profile UI property name	Dataproc profile JSON property name

Dataproc profile UI property name	Dataproc profile JSON property name
Profile Label	name
Profile Name	label
Description	description
Property ID	projectId
Creator Service Account Key	accountKey
Region	region
Zone	zone
Network	network
Network Host Project ID	networkHostProjectId
Subnet	subnet
Runner Service Account	serviceAccount
Number of masters	masterNumNodes
Master Machine Type	masterMachineType
Master Cores	masterCPUs
Master Memory (GB)	masterMemoryMB
Master Disk Size (GB)	masterDiskGB
Master Disk Type	masterDiskType
Number of Primary Workers	workerNumNodes
Number of Secondary Workers	secondaryWorkerNumNodes
Worker Machine Type	workerMachineType
Worker Cores	workerCPUs
Worker Memory (GB)	workerMemoryMB
Worker Disk Size (GB)	workerDiskGB
Worker Disk Type	workerDiskType
Metadata	clusterMetaData
Network Tags	networkTags
Enable Secure Boot	secureBootEnabled
Enable vTPM	vTpmEnabled
Enable Integrity Monitoring	integrityMonitoringEnabled
Image Version	imageVersion
Custom Image URI	customImageUri
GCS Bucket	gcsBucket
Encryption Key Name	encryptionKeyName
Autoscaling Policy	autoScalingPolicy
Initialization Actions	initActions
Cluster Properties	clusterProperties
Labels	clusterLabels
Max Idle Time	idleTTL
Skip Cluster Delete	skipDelete
Enable Stackdriver Logging Integration	stackdriverLoggingEnabled
Enable Stackdriver Monitoring Integration	stackdriverMonitoringEnabled
Enable Component Gateway	componentGatewayEnabled
Prefer External IP	preferExternalIP
Create Poll Delay	pollCreateDelay
Create Poll Jitter	pollCreateJitter
Delete Poll Delay	pollDeleteDelay
Poll Interval	pollInterval