Nested anonymous classes cannot be used in Dynamic Spark

Description

It looks like nested anonymous functions are not properly shipped when evaluating dynamic spark code. To reproduce:

1. Install 'Dynamic Spark Plugin' v2.0.3 from Cask Market

2. Create a pipeline that contains just a single ScalaSparkProgram stage

3. Configure the stage with the following code:

4. Deploy and run the pipeline. There will be an exception like:

The program calculates the square of every element in an array in 2 ways. The first does not a closure within a closure and will succeed, which can be verified by looking for a message in the program log.

The exception will occur when calculating 'squaredNested', which uses a closure (map) within another closure (mapValues):

Release Notes

Fixed dynamic spark plugin to use appropriate context classloader for loading dynamic spark code

Activity

Show:
Terence Yim
April 25, 2019, 6:30 PM
Albert Shau
November 30, 2017, 9:34 PM

example pipeline

Fixed

Assignee

Terence Yim

Reporter

Albert Shau

Labels

Docs Impact

None

UX Impact

None

Components

Fix versions

Affects versions

Priority

Major