Inconsistent transactional behavior of beforeSubmit() and onFinish()
Description
Release Notes
With the new Spark API, the beforeSubmit() and onFinish() methods are executed on the client side with their own transaction.
relates to
Activity
Show:
Terence Yim April 9, 2016 at 8:05 AM
With the new spark core implementation, beforeSubmit and onFinish run with their own transaction like in MapReduce.
Bhooshan Mogal September 9, 2015 at 10:36 PM
Moving this out of 3.2, because we do not have bandwidth to fix this right now.
Fixed
Pinned fields
Click on the next to a field label to start pinning.
Details
Details
Assignee
Terence Yim
Terence YimReporter
Andreas Neumann
Andreas Neumann
Components
Fix versions
Priority
MajorCreated July 21, 2015 at 6:58 PM
Updated April 25, 2016 at 6:19 PM
Resolved April 9, 2016 at 8:05 AM
(Batch) Programs like Spark and MapReduce have a beforeSubmit() and a inFinish() method. Currently, in MapReduce, these methods run in their own transaction (not the long running transaction used by the MapReduce itself). For Spark, these methods use the same transaction as the Spark program.
This is counter-intuitive. They should behave the same in both program types.