Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
Intelligent workflow scheduling: Implement limits on the number of active workflows, yarn containers, etc, to avoid resource starvation. For near term, a limit on number of active workflows will suffice.
The use case here is that a developer wants to ensure that workflows don't pile up and destroy a cluster.
Release Notes
Added a way to limit the number of concurrent schedule runs.
For that scenario, we would not want to start the program. I was a bit careless with the terminology, what I meant is that the run record has RUNNING status, which occurs once the program is in the STARTING status. It's confusing that run record and program status are different, but that's another issue.
NitinM October 25, 2015 at 4:48 PM
Assume a scenario where a program in the workflow is started and it's not running yet, but waiting for containers to available to start running. Would this scenario break the set skip rule limit and start a new run of a program if getting containers for previous run takes longer than the schedule interval.
Intelligent workflow scheduling:
Implement limits on the number of active workflows, yarn containers, etc, to avoid resource starvation. For near term, a limit on number of active workflows will suffice.
The use case here is that a developer wants to ensure that workflows don't pile up and destroy a cluster.