We're updating the issue view to help you get more done. 

Hydrator workflow planner

Description

We need a planner that will take a logical pipeline and break it down into a physical workflow.

The contract will be to take the ETLConfig object, then turn it into one or more Pipeline objects that the individual programs will understand and know how to implement. So ETLMapReduce will take a single Pipeline in its constructor instead of the ETLBatchConfig object it used to take. ETLMapReduce only needs to know how to execute a single Pipeline.

Terminology is confusing now, should rename the Pipeline object to PipelinePhase or something like that.

Release Notes

None

Assignee

Albert Shau

Reporter

Albert Shau

Labels

None

Docs Impact

None

UX Impact

None

Components

Fix versions

Priority

Major
Configure