What is Cask Tracker ?

...

• Metadata
  • Tags, Properties, Schema for CDAP Datasets and Programs
  • System and User 
• Data Quality
  • Metadata that include Feed-level and Field-level quality metrics of datasets
• Data Usage Statistics 
  
  • Usage statistics of dataset and programs. 

Checklist

•  User stories documented (Nitin)
•  User stories reviewed (Nitin)
•  Design documented (Shankar)
•  Design reviewed (Terence/Andreas/Albert)
•  Feature merged (Shankar)
•  UI updated (Ajai/Edwin)
•  Documentation for feature (Shankar)

...

Following are the use-cases for the Cask Tracker.

Case #1

• Rishab is a data scientist/engineer at a company that implements a Data Lake. He is analyzing the effectiveness of the recommendation engine on the company's e-commerce site. For this investigation, he wants to analyze a dataset that includes click log for the last year. He is looking for clean click log data that is up-to-date. He wants to use part of the data to build model and rest to score the model and validate the predictions.  
• Before he can conduct an analysis, Rishab needs to confirm the dataset is available in the Data Lake.
• To do so, he wishes to find all entities that include “click log”.
• He arrives at the Finder home screen (from nav, search results, other entry points?).
  • Enters “click log” in the Search Box and clicks Search.
  • He arrives at the Results Page. 
    • Results returned
    • By default, they are sorted by creation time
    • Each Result includes:
      • Snippet of the metadata that matches his query in context.
        • Important to help him evaluate the relevance of the results.
      • Date Created
        • To know how recent/new it is.
• For this analysis, Rishab is most concerned with the recency, the accuracy, and the integrity of the data.
• He clicks the result and arrives at the Entity Detail Page where he can view all of the metadata associated with an entity. 
• Rishab wished to verify the validity of the sources of this dataset. To do so, he clicks the Lineage Tab to trace the creation of this dataset to its source.
• Finder displays the lineage for this dataset as a diagram. The selected dataset displays in the center; to the left is the entity that precedes it and to the right is the one it precedes.
• Rishab discovers that it has been created from two separate sources.
• He then clicks one of the sources which takes him to the Entity Page of that dataset.
• He clicks on a program to see what has been done to the dataset.
• Rishab clicks the Audit Logs Tab to see how active this dataset has been - when was it last updated, who is using it, writing to it, reading from it.
• Rishab clicks the appropriate action to make this dataset a new source for his existing Click Log processing pipeline.
• This takes him to the Hydrator Studio where he can edit the Master Click Log pipeline.

Case #2 (Incomplete)

• D-Rock, an IT genius, has been tasked with finding out all the datasets on the cluster that have the field SSN.
• He needs to find every dataset with an SSN field and mask the field by generating new feed containing a masked SSN. 
• In order to achieve this task, he uses Cask Finder to achieve this task
  • Enters “SSN” in the Search Box and clicks Search.
  • He then filters by Schema – meaning he would only like to see results which includes datasets where the SSN field is matched in Schema.
• He then selects the dataset that has this field and ...

Design

Release 1.0 Deliverables

...