CDAP Workflow cannot fork multiple parallel Spark actions consistently


When the CDAP workflow driver encounters a fork action, it attempts to spark-submit in parallel threads. The stack trace below indicates this eventually leads to multiple parallel threads executing reflection code, which is unsafe in Scala 2.10.x. Please refer and The only way to prevent the problem is to synchronize the block that executes scala 2.10.x Reflection code.
The exception stack trace is below.

The FATAL 'tail of empty list' exception occurs only when the CDAP workflow folks to spawn multiple Spark jobs in parallel.

Release Notes

Provided a workaround for Scala bug SI-6240 ( to allow concurrent execution of Spark programs in CDAP Workflows.


Priyanka Nambiar
January 14, 2017, 12:13 AM
Terence Yim
January 12, 2017, 9:27 PM


Terence Yim


Terence Yim