Arguments set by actions don't overwrite runtime arguments

Description

As part of https://cdap.atlassian.net/browse/CDAP-18643 , the program launcher will regenerate the app-spec, in order to make sure plugins specified in connections (ex: jdbc plugins in DB connections) are properly registered and usable.

This has the side effect that other macros in the pipeline will get evaluated on regeneration. This means that when the pipeline finally runs on the cluster, it doesn’t see those as macros anymore, and thus will not use any arguments set by an action within the pipeline.

Release Notes

Fixed a bug where arguments set by actions don't overwrite runtime arguments. Users need to add this runtime argument `system.skip.normal.macro.evaluation=true`.

Activity

Show:

Albert Shau April 25, 2023 at 7:24 PM

Note that in some cases, macro evaluation using just the runtime args will cause validation checks to fail. In this case, a warning will be logged that the app-spec couldn’t be regenerated, and the original app-spec will be used. Then at runtime the proper behavior will happen.