We're updating the issue view to help you get more done. 

Transaction timeout should be configurable per app or program

Description

Currently, this can only be set in cdap-site.xml and is therefore site-wide. It is often desirable to control on a per-use-case basis.

In this particular case, this could also be solved by providing a specific API to a program to control its transaction timeout.

But in general, it should be possible to configure system properties on a per-app or per-program basis.

Release Notes

Allows configuration of the default transaction timeout at the level of a program, flowlet, or custom workflow action.

Activity

Show:
Andreas Neumann
September 20, 2016, 12:10 AM
Edited

Design

The idea is similar to CDAP-5770: to have the transaction timeout settable through a runtime argument:

  • system.tx.timeout - transaction timeout in seconds.

For MapReduce and Spark the timeout applies only to the client (that is, it's initialize/destroy), as all other tasks will be inside a long transaction.

For program types that have sub-components (Flow, Workflow), a prefix can be added to narrow the scope of the argument.

Flow

  • When prefixed with "flowlet.[flowletName].", the timeout will only affect the given flowlet.

Workflow Actions

  • Prefix with "action.[actionName]." to set the timeout for the named custom action only.

Terence Yim
September 20, 2016, 7:54 PM

For MR and Spark, the timeout should only affect the client lifecycle method, right?

Andreas Neumann
September 20, 2016, 8:12 PM
Edited

yes, and that is actually mentioned in the 3rd line. Oh, it says driver, not client. Will fix.

Terence Yim
September 20, 2016, 8:16 PM

Yes I saw that, just want to make it clear it's not the driver of the Spark program

Andreas Neumann
November 12, 2016, 3:10 AM
Fixed

Assignee

Andreas Neumann

Reporter

Andreas Neumann

Labels

Docs Impact

None

UX Impact

None

Components

Fix versions

Affects versions

Priority

Major
Configure