Start a spark program (realtime Hydrator pipeline works)
Stop the spark program
The CDAP program status will be stopped, but the yarn containers will still be running. I believe the cause is that DistributedProgramRuntimeService does not create a controller for spark programs discovered from the TwillRunner.
Fixes cases where Spark programs cannot be started after a master failover or restart.
Seems like it was never there, even after we introduced the first Spark integration.
During the process for fixing it, I also discovered that the permgen usage will grow linearly to the number of active Spark program runs, which is undesirable.
Including this fix to 3.5 and 3.6 since it will be critical for anyone who try to runs Spark in production.