...
| Id | Description | Comments |
|---|---|---|
| U1 | As a user, I should be able to search Datasets containing the specified fields | List the kinds of queries that will be supported |
| U2 | As a CDAP system, I should be able to annotate CDAP entities with system metadata automatically | System metadata for each entity is listed below |
| U3 | As a user, I should be able to access and update CDAP metadata using the CDAP CLI | |
| U4 | As a developer, I should be able to access and update CDAP metadata using the CDAP Test Framework | |
| U5 | As a user, I should be able to search CDAP entities based on metadata using the CDAP UI | |
| U6 | As a user, I should be able to view the lineage of a CDAP dataset/stream in a specified time window using the CDAP UI |
New Metadata Search:
Metadata Storage and Search to support:
...
- Schema: {EmpName: String, EmpContact: {EmpTel: Integer, EmpAddr: String}} (Note: This is a nested schema)
Use case: User should be able to search with- FieldName scoped to schema (complete or partial): Schema: EmpName or Schema: EmpContact or Schema: EmpTel or Schema: EmpAddr or Schema: Emp*
- FieldName (complete or partial): EmpName or EmpContact or EmpTel or EmpAddr or Emp*
Storage:
Main Table: This table stores stores the metadata for the entity. It will be used when an user wants to get metadata of an entity. This table is not for searching.
| Key: Entity with key | Value: Value of Metadata |
|---|---|
| <Entity-Id><CodeName> | Alpha Tango Charlie |
| <Entity-Id><Tags> | {Tag1, Tag22} |
| <Entity-Id><Schema-Id> | {EmpName: String, EmpContact: {EmpTel: Integer, EmpAddr: String}} |
Index Table: This table will be used for searching and it will use IndexedTable.
| Key: Entity with index | Value: Index |
|---|---|
| <Entity-Id><CodeName: Alpha Tango Charlie> | CodeName: Alpha Tango Charlie |
| <Entity-Id><Codename: Alpha> | Codename: Alpha |
| <Entity-Id><Codename: Tango> | Codename: Tango |
| <Entity-Id><Codename: Charlie> | Codename: Charlie |
| <Entity-Id><Alpha Tango Charlie> | Alpha Tango Charlie |
| <Entity-Id><Alpha> | Alpha |
| <Entity-Id><Tango> | Tango |
| <Entity-Id><Charlie> | Charlie |
| <Entity-Id><Tags: Tag1> | Tags: Tag1 |
| <Entity-Id><Tags: Tag22> | Tags: Tag22 |
| <Entity-Id><Tag1> | Tag1 |
| <Entity-Id><Tag22> | Tag22 |
| <Entity-Id><EmpName> | EmpName |
| <Entity-Id><EmpContact> | EmpContact |
| <Entity-Id><EmpTel> | EmpTel |
| <Entity-Id><EmpAddr> | EmpAddr |
Search Result:
Metadata search will return Entities with the following details depending upon the type of the Entity.
...
Type
...
Programs which uses this View.
...
System Metadata
Kinds of system metadata:
...