CDAP Workflow cannot fork multiple parallel Spark actions consistently

Description

When the CDAP workflow driver encounters a fork action, it attempts to spark-submit in parallel threads. The stack trace below indicates this eventually leads to multiple parallel threads executing reflection code, which is unsafe in Scala 2.10.x. Please refer http://docs.scala-lang.org/overviews/reflection/thread-safety.html and https://issues.scala-lang.org/browse/SI-6240. The only way to prevent the problem is to synchronize the block that executes scala 2.10.x Reflection code.
The exception stack trace is below.

The FATAL 'tail of empty list' exception occurs only when the CDAP workflow folks to spawn multiple Spark jobs in parallel.

Release Notes

Provided a workaround for Scala bug SI-6240 (https://issues.scala-lang.org/browse/SI-6240) to allow concurrent execution of Spark programs in CDAP Workflows.

Activity

Show:
Priyanka Nambiar
January 14, 2017, 12:13 AM
Terence Yim
January 12, 2017, 9:27 PM
Fixed

Assignee

Terence Yim

Reporter

Terence Yim