Cloud Dataproc is a Google Cloud Platform (GCP) service that manages Hadoop and Spark clusters in the cloud and can be used to create large clusters quickly. The Google Dataproc provisioner simply calls the Cloud Dataproc APIs to create and delete clusters in your GCP account. The provisioner exposes several configuration settings that control what type of cluster is created.
Version compatibility
Problem: The version of your CDAP environment might not be compatible with the version of your Dataproc cluster.
Recommended: Upgrade to CDAP version 6.4 or later and use one of the supported Dataproc versions.
CDAP versions before 6.4 are only compatible with unsupported versions of Dataproc. Dataproc does not provide updates and support for clusters created with these versions. Although you can continue running a cluster that was created with an unsupported version, replacing the cluster with a new cluster that is created with a supported version is recommended.
CDAP version | Dataproc version |
|---|---|
6.1 to 6.3* | 1.3.x |
6.4 to 6.6 | 1.3.x and 2.0.x |
6.7 | 1.5.x and 2.0.x |
* CDAP versions 6.1 to 6.3 are compatible with Dataproc version 1.3. You don't need additional components to make them compatible. CDAP uses HDFS and Spark.
Account Information
Project ID
...