ProgramLifecycle methods should have better control over transactions

Description

Today, initialize() and destroy() run in a single transaction. That can lead to transaction timeouts. These methods should have a way to:

  • control the transaction timeout

  • start multiple transactions

  • start a "long-running" transaction that does not time out.

Release Notes

Allows program lifecycle methods to control transactions explicitly.

Activity

Show:

Andreas Neumann October 12, 2016 at 7:59 PM

Update to the Design: It would be better to use a more reusable annotation that will be able to express other aspects of transactions in the future. Instead of @NoTransaction, we will have an annotation @TransactionPolicy(TransactionControl.EXPLICIT).

In the future we can extend this, for example, to allow specifying the transaction timeout, too, or to allow program types (such as workers) that have explicit transaction control by default, to annotate @TransactionPolicy(TransactionControl.IMPLICIT).

Andreas Neumann September 21, 2016 at 12:43 AM

Addition: Nesting transactions will not be allowed and throw an exception.

Andreas Neumann September 20, 2016 at 11:36 PM
Edited

Design

Currently all life cycle methods (initialize, destroy) are executed in a (short) transaction. But:

  • some methods do not require a transaction (or it could even get in the way by causing tx timeouts)

  • some methods need a longer timeout, or they need to start multiple transactions

This can be done implementing the Transactional interface with the program context, which will allow these methods to control their own transactions and their timeout. At the same time, we need a way to declare that this method does not require an implicit transaction. That can be done using a new annotation @NoTransaction. Then, for example, an initialize() method could look like this:

This will apply to life cycle methods of flowlets, service handlers, workers, mapreduce, spark, workflow, but not to task-level life cycle methods in mapreduce or spark (they are executed in the context of the program's long tx).

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Labels

Affects versions

Components

Fix versions

Priority

Created August 30, 2016 at 7:49 PM
Updated December 16, 2016 at 7:40 PM
Resolved October 25, 2016 at 4:37 AM