Flowlet callback for change instances event
Description
Release Notes
Activity
Terence Yim April 25, 2019 at 8:13 PM
Flow is removed as of 6.0
Terence Yim December 3, 2014 at 12:45 AM
Shouldn't go through notification system in this case. Notification is for something external, while changing instance count is internal to the program, and in fact, local to the flowlet.
Alvin Wang December 3, 2014 at 12:35 AM
How about instead of implementing callbacks for every possible event, we just use the notification service? (don't really know what it exactly is though) Then when flowlet instances is changed, we just push a notification to the service and any listener for that event will receive it.
Julien Guery November 24, 2014 at 11:56 PM
After further discussion, we actually need to implement the same callback logic for other programs. We need to take a step back and think about a common set of classes to achieve that. I will closed the current PR for now.
Julien Guery November 19, 2014 at 10:51 PM
has some concerns on the ``@Retry`` annotation that this PR creates, since we may create the same logic in several places in the code.
Currently there is no way for a Flowlet know about number of Flowlet instances changed, which is useful in some cases. E.g. Flowlet that poll from Kafka can use this as a signal to rebalance the topic/partition combination to poll from among themselves.
One requirement for the callback is that they need to be synchronized across all instances, i.e. Suspend Flowlet -> Change Instances -> Callback -> Resume Flowlet, with each phase being completed by all instances before the next one started. It is already like that in the Flow system, witout the "Callback" phase. Another requirement is, of course, callback needs to be transactional, as it might need to use Dataset to assist the work to be done inside the callback.