Scala compute plugin with spark2 fails to run on dataproc

re-opening since it hasn't been cherry-picked yet

Removing scala as a direct dependency doesn’t prevent scala-library from getting pulled into the twill.jar, likely because it gets traced from the Kafka dependencies anyway. Filtering it out of the libraries within DataprocJobMain does fix this issue, but need to see if it makes more sense to filter it out in the submission part rather than later in the launcher.

scala was added as a direct dependency in . Should not have added scala-library to a base class like this, it can cause all sorts of issues.

I think this is because the twill.jar we create and ship to DataprocJobMain contains some scala libraries:

scala-library-2.12 in particular is the one with scala.collection.immutable.List, which is incompatible with scala 2.11 on the cluster.

I’m not sure why scala is getting bundled into the twill jar, will see if we can exclude it