Page Comparison

...

Given a dataset and time range, get the high level lineage both in forward and backward direction.

Code Block

GET /v3/namespaces/<namespace-id>/endpoints/<endpoint-name>/fields/lineage?start=<start-ts>&end=<end-ts>&level=<level>


Where:
namespace-id: namespace name
endpoint-name: name of the endpoint
start-ts: starting timestamp(inclusive) in seconds
end-ts: ending timestamp(exclusive) in seconds for lineage
level: how many hops to make in backward/forward direction


Sample response:
[
  ...
  list of lineage mappings
  ...
]


where each lineage mapping will be of the form:


{
  "source": {
     "namespace": "ns",
     "name": "Person" 
  },
  "Destination": {
     "namespace": "ns",
     "name": "Employee"
  },
  "fieldmap": [
     { "from": "id", "to": "id" },
     { "from": "first_name", "to": "name"},
     { "from": "last_name", "to": "name"}
  ] 
}

Given a dataset and field, find out the detailed lineage.

Code Block

GET /v3/namespaces/<namespace-id>/endpoints/<endpoint-id>/fields/<field-name>/lineage?start=<start-ts>&end=<end-ts>&direction=<backward/forward>
 
Where:
namespace-id: namespace name
endpoint-id: endpoint name
field-name: name of the field for which lineage information to be retrieved
start-ts: starting timestamp(inclusive) in seconds
end-ts: ending timestamp(exclusive) in seconds for lineage
direction: backward or forward


Sample response:
{
  [
   ....

      list of connections between fieldsnodes
   ....
]  ],
where each connection[
is as follows: {...
  "from": "id",   "to": "id",
  "operation": {
list of operations
   ...
  ],
 "name": "IDENTITY", [
   ...
 "description": "operation description"   list of connections
   ...
  ]    
}

where each Node is an object representing field. Node has id which is uniquely identifies the Node (combination of origin and name) and label which is used to display on the UI. Node can have optional sourceEndPoint and destinationEndPoint members which represents if this node is generated directly from Source EndPoint or written to the Destination EndPoint.
 
{
  "id": "origin.fieldname",
  "label": "fieldname"
  "sourceEndPoint": {
     "name": "file",
     "namespace": "ns" 
  }  
}


each Operation is represented as 
{
  "name": "IDENTITY",
  "description": "description associated with the operation". 
}
each Connection represents transformation between two nodes with operation name that caused it:
{
  "from": "Node1.id",
  "to": "Node2.id",
  "operation": "opname"
}

Store:

Based on the above example, we want following pieces of information to be stored in the "FieldLevelLineage" dataset

...

Versions Compared

Old Version 15

New Version 16

Key