...
| Code Block |
|---|
{
"phases": [
{
"name": "phase1",
"stages": [
{
"name": "customersTable",
"plugin": {
"name": "Database",
"type": "batchsource", ...
}
},
{
"name": "customersFiles",
"plugin": {
"name": "TPFSParquet",
"type": "batchsink", ...
}
],
"connections": [
{ "from": "customersTable", "to": "customersFiles" }
]
},
{
"name": "phase2",
"stages": [
{
"name": "purchasesTable",
"plugin": {
"name": "Database",
"type": "batchsource"
}
},
{
"name": "purchasesFiles",
"plugin": {
"name": "TPFSParquet",
"type": "batchsink", ...
}
}
],
"connections": [
{ "from": "purchasesTable", "to": "purchasesFiles" }
]
}
]
"connections": [
{ "from": "phase1", "to": "phase2" }
]
} |
Option2: Introduce "runconditioncondition" plugin type. Connections into a run condition imply control flow rather than data flow.
| Code Block |
|---|
{
"stages": [
{
"name": "customersTable",
"plugin": {
"name": "Database",
"type": "batchsource", ...
}
},
{
"name": "customersFiles",
"plugin": {
"name": "TPFSParquet",
"type": "batchsink", ...
}
},
{
"name": "afterDump",
"plugin": {
"name": "AlwaysRun",
"type": "runconditioncondition"
}
},
{
"name": "purchasesTable",
"plugin": {
"name": "Database",
"type": "batchsource"
}
},
{
"name": "purchasesFiles",
"plugin": {
"name": "TPFSParquet",
"type": "batchsink", ...
}
},
],
"connections": [
{ "from": "customersTable", "to": "customersFiles" },
{ "from": "customersFiles", "to": "afterDump" },
{ "from": "afterDump", "to": "purchasesTable" },
{ "from": "purchasesTable", "to": "purchasesFiles" }
]
} |
...