Refactor MapReduce to use configurer style API

Description

None

Release Notes

None

Activity

Show:

Terence Yim October 21, 2014 at 7:13 AM

Terence Yim October 21, 2014 at 7:13 AM

Agree with the review of the MapReduce API. However would like to have this ticket only associates with the change to configurer style and refactoring will be a follow up ticket to keep the change set small.

Alex Baranau October 21, 2014 at 1:41 AM

beforeSubmit and onFinish run inside same transaction as MapReduce

Oh, I just found out that this was changed recently. Now they are in separate transactions.

Still, I believe there's room for improvement of the API.

Alex Baranau October 21, 2014 at 1:23 AM

I'd recommend to review the whole MR APIs to make them better. E.g. we want to address (apart from other things):

beforeSubmit and onFinish run inside same transaction as MapReduce. It is not suitable to do changes in datasets on mapreduce failure (e.g. write some metadata somewhere that idicates about failure) because any changes made to transactional resources (e.g. datasets) will not be visible.

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Components

Fix versions

Priority

Created October 7, 2014 at 9:26 PM
Updated November 20, 2014 at 9:30 PM
Resolved October 21, 2014 at 8:07 PM