...
Earlier versions of CDAP are only compatible with unsupported versions of Dataproc. Dataproc does not provide updates and support for clusters created with these versions. Although you can continue running a cluster that was created with an unsupported version, we recommend replacing it with one created with a supported version.
CDAP version | Dataproc version |
|---|---|
6.7+ | 2.0, 1.5 * |
6.4-6.6 | 2.0 *, 1.3 ** |
6.1-6.3 | 1.3** |
* CDAP versions 6.4 and later are compatible with supported versions of Dataproc. Unless specific OS features are needed, the recommended practice is to specify the major.minor image version.
...
Recommended: When you create a static cluster for your pipelines, use the following configurations.
Parameters | |
|---|---|
| Retains YARN logs. |
| Enables YARN to check for physical memory limits and kill containers if they go beyond physical memory. |
| Enables YARN to check for virtual memory limits and kill containers if they go beyond physical memory. |
Account Information
Project ID
...
Cluster properties used to override default configuration properties for the Hadoop services. The applicable key-value pairs can found here.
Common Labels
A label is a key-value pair that helps you organize your Google Cloud Dataproc clusters and jobs. You can attach a label to each resource, and then filter the resources based on their labels. Information about labels is forwarded to the billing system, so customers can break down your billing charges by label.
...
Dataproc profile UI properties mapped to JSON properties
Dataproc profile UI property name | Dataproc profile JSON property name |
|---|---|
Profile Label | name |
Profile Name | label |
Description | description |
Property ID | projectId |
Creator Service Account Key | accountKey |
Region | region |
Zone | zone |
Network | network |
Network Host Project ID | networkHostProjectId |
Subnet | subnet |
Runner Service Account | serviceAccount |
Number of masters | masterNumNodes |
Master Machine Type | masterMachineType |
Master Cores | masterCPUs |
Master Memory (GB) | masterMemoryMB |
Master Disk Size (GB) | masterDiskGB |
Master Disk Type | masterDiskType |
Number of Primary Workers | workerNumNodes |
Number of Secondary Workers | secondaryWorkerNumNodes |
Worker Machine Type | workerMachineType |
Worker Cores | workerCPUs |
Worker Memory (GB) | workerMemoryMB |
Worker Disk Size (GB) | workerDiskGB |
Worker Disk Type | workerDiskType |
Metadata | clusterMetaData |
Network Tags | networkTags |
Enable Secure Boot | secureBootEnabled |
Enable vTPM | vTpmEnabled |
Enable Integrity Monitoring | integrityMonitoringEnabled |
Image Version | imageVersion |
Custom Image URI | customImageUri |
GCS Bucket | gcsBucket |
Encryption Key Name | encryptionKeyName |
Autoscaling Policy | autoScalingPolicy |
Initialization Actions | initActions |
Cluster Properties | clusterProperties |
Labels | clusterLabels |
Max Idle Time | idleTTL |
Skip Cluster Delete | skipDelete |
Enable Stackdriver Logging Integration | stackdriverLoggingEnabled |
Enable Stackdriver Monitoring Integration | stackdriverMonitoringEnabled |
Enable Component Gateway | componentGatewayEnabled |
Prefer External IP | preferExternalIP |
Create Poll Delay | pollCreateDelay |
Create Poll Jitter | pollCreateJitter |
Delete Poll Delay | pollDeleteDelay |
Poll Interval | pollInterval |