Goals:

1. JIRA: CDAP-3969: CDAP should offer a temporary location to store results between jobs of a workflow.
2. JIRA: CDAP-4075: Error handling for Workflows.

Checklist

• User stories documented (Sagar)
• User stories reviewed (Nitin)
• Design documented (Sagar)
• Design reviewed (Albert/Terence/Andreas)
• Feature merged (Sagar)
• Examples and guides (Sagar)
• Integration tests (Sagar) 
• Documentation for feature (Sagar)
• Blog post

Use Cases:

1. JIRA: CDAP-3969: CDAP should offer a temporary location to store results between jobs of a workflow.
   

   

   Consider the above sample workflow from CDAP-Workflow guide. The goal is to process the raw purchase events from the purchaseEvents stream and find the purchases made by each customer and purchases made for the particular product. When workflow runs, PurchaseEventParser reads the raw events from the purchaseEvents stream and writes the purchase objects to the purchaseRecords dataset. This dataset is later used by PurchaseCounterByCustomer and PurchaseCounterByProduct MapReduce programs as input to create the datasets customerPurchases and productPurchases respectively. Note that when the workflow completes, user is only interested in the final datasets that are created by the Workflow run: customerPurchases and productPurchases. The dataset purchaseRecords created by the MapReduce program PurchaseEventParser is temporary and no longer required when the workflow run is completed.

    

2. JIRA: CDAP-4075: Error handling for Workflows.
   a) When the Workflow fails for some reason, user may want to notify appropriate parties via email, possibly with the cause of the failure and the node at which the Workflow failed.
   b) When the Workflow fails for some reason at a particular node, user may want to cleanup the datasets and files created by the previous nodes in the Workflow.
   
   

User Stories:

1. As a developer of the Workflow, I want ability to specify that the output of the particular program (MapReduce/Spark) in the Workflow is temporary, so that Workflow system can clean it up. (CDAP-3969)
2. As a developer of the MapReduce or Spark program, I should be able to run it in standalone as well as inside the Workflow with its output specified as temporary. When in run in standalone, the output should not be cleaned up. (CDAP-3969)
3. As a developer of the Workflow, I want ability to specify the functionality that will get executed when the Workflow finishes successfully. (CDAP-4075)
4. As a developer of the Workflow, I want ability to specify the functionality that will get executed when the Workflow fails at any point in time. I want access to the cause of the failure and the node at which the workflow failed. (CDAP-4075)