Nested anonymous classes cannot be used in Dynamic Spark


It looks like nested anonymous functions are not properly shipped when evaluating dynamic spark code. To reproduce:

1. Install 'Dynamic Spark Plugin' v2.0.3 from Cask Market

2. Create a pipeline that contains just a single ScalaSparkProgram stage

3. Configure the stage with the following code:

4. Deploy and run the pipeline. There will be an exception like:

The program calculates the square of every element in an array in 2 ways. The first does not a closure within a closure and will succeed, which can be verified by looking for a message in the program log.

The exception will occur when calculating 'squaredNested', which uses a closure (map) within another closure (mapValues):

Release Notes

Fixed dynamic spark plugin to use appropriate context classloader for loading dynamic spark code


Terence Yim
April 25, 2019, 6:30 PM
Albert Shau
November 30, 2017, 9:34 PM

example pipeline

Your pinned fields
Click on the next to a field label to start pinning.


Terence Yim


Albert Shau