/
Program State Transition for 4.3
Program State Transition for 4.3
Assumptions
- State change message will be sent after action is performed successfully (so that the
action timestamp < message timestamp
) - No out of order messages (based on message timestamp)
- STARTING is safe state change -> no multiple messages, no missed messages, no out of order messages
Changes required
Program state change writer
- Store the latest message timestamp along with the run record during update
- Ignore a message if message timestamp is equal or less than what is recorded
- Update code to match the new states in the state diagram (the new changes are in red color)
- Publish the finally resolved program state message transactionally along with the store update to a new topic
Run record corrector
- Run record corrector will only process run records before the last processed message time
Scheduler
- Make scheduler listen to the changes published by the program state change writer
State Transition Diagram
Test Scenarios
Scenario | Expected Result | Actual Result |
---|---|---|
Start a Purchase History Workflow and let it complete |
STARTING -> RUNNING -> COMPLETED 2. Run Record for Mapreduce STARTING -> RUNNING -> COMPLETED | |
Start Purchase History Workflow and kill the mapreduce job when it is running in YARN |
STARTING -> RUNNING -> KILLED 2.Run Record for Mapreduce STARTING -> RUNNING -> KILLED | |
Start Purchase History Workflow and Kill the workflow before the mapreduce is started |
STARTING -> RUNNING -> KILLED 2. No runrecord mapreduce | |
Write a workflow with a custom action that throws an exception |
STARTING -> RUNNING -> KILLED 2. No run record for Custom action | |
a. Write a workflow that requests for more resources than available resources b. Kill the YARN application |
|
, multiple selections available,
Related content
Refined Program States
Refined Program States
More like this
Program Status Based Scheduling
Program Status Based Scheduling
More like this
Stateful Transform
Stateful Transform
More like this
Scheduler Improvements (WiP)
Scheduler Improvements (WiP)
More like this
Composite Trigger
Composite Trigger
More like this
Workflows
Workflows
More like this
Created in 2020 by Google Inc.