Running CDAP pipelines in spark3 enabled dataproc cluster fails with class not found exception.
Steps to reproduce,
1. Publish pipeline (database -> bq)
2. Create a dataproc profile with image configured to `preview-debian10`
3. Set the profile as default for the pipeline and run.
After the step3 the pipeline fails soon with following exception in the pipeline logs,
There is a change in Spark 3.1 about inclusion of yarn.application.classpath. For now, the workaround is to set the spark.yarn.populateHadoopClasspath to true in the engine config.