We'll need to update all hydrator sources/sinks (both batch and realtime) to enable tracking of external datasets.
Introduced a 'referenceName' property (used for lineage, annotation metadata) in all external sources/sinks which needs to be set while using that plugin
The change is basically for hydrator plugins to define a "tracking name" configuration parameter, and use the new APIs with name and alias -
Also, it could would be good to add an integration test after the sources/sinks are updated. We need to test that lineage gets recorded for a hydrator pipeline with external source and external sink.
I am not sure whether "tracking" is the best term here. This will be used to create a dataset which will be used to uniquely refer to the source/sink for various purposes: Lineage, Metadata, Audit, perhaps even use it in programs.
Perhaps "reference name" is better?