...
User has a multi job pipeline, set of transforms run as a part of MR job doing data cleansing and a spark ML job on training data to output model. wants to add a conditional node after data cleansing, if there were less than 100 records that failed the preprocessing, he/she likes to continue running spark ML job, else he/she wants to send an email to the data engineer.
...
- Pipeline developer wants to add a conditional node after one job before running the next job.
- Pipeline developer wants to add a conditional node at the end of the pipeline and optionally send an email if there were any errors in the final stage.
- Pipeline developer wants to configure the conditional node with his custom code logic.
- Pipeline developer wants to configure the next job to run based on a condition, based on condition pipeline can run either Job-A or Job-B
...