Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Current »

Goals

Ability to specify condition for execution-path in the hydrator pipelines.

Checklist

  • User stories documented (Shankar/Sagar)
  • User stories reviewed (Nitin)
  • Design documented (Shankar/Sagar)
  • Design reviewed (Terence/Andreas)
  • Feature merged ()
  • Examples and guides ()
  • Integration tests () 
  • Documentation for feature ()
  • Blog post

Use-case

User has a multi job pipeline, set of transforms run as a part of MR job doing data cleansing and a spark ML job on training data to output model. wants to add a conditional node after data cleansing, if there were less than 100 records that failed the preprocessing, he/she likes to continue running spark ML job, else he/she wants to send an email to the data engineer.

 

User-Stories

 

  • Pipeline developer wants to add a conditional node after one job before running the next job. 
  • Pipeline developer wants to add a conditional node at the end of the pipeline and optionally send an email if there were any errors in the final stage.
  • Pipeline developer wants to configure the conditional node with his custom code logic.   
  • Pipeline developer wants to configure the next job to run based on a condition, based on condition pipeline can run either Job-A or Job-B  
  • Plugin developer wants to add a new conditional-plugin supporting a new language or expression for specifying conditional logic.
  • Pipeline developer wants to use workflow tokens and use that information to take action in the conditional node.
  • Pipeline developer wants to test that their custom condition logic is correct without having to deploy and run their pipeline.




  • No labels