Tracker Audit Metrics

This is not for 3.4 release.

The goal of this page is to document the design of the Tracker Audit Metrics.

Use-Cases

1. As a user of Tracker, I would like to see total number of audit messages by type/subtype in the past T timeframe.

   1. "Show me the total number of reads in the system in the past 1 hour."

2. As a user of Tracker, I would like to see the top N datasets/streams by audit message type/subtype activity in the past T timeframe.

   1. "Show me the 5 datasets with the most writes in the past 24 hours."

   2. "Show me the 5 streams with the most metadata_changes in the past 7 days."

3. As a user of Tracker, I would like to see the top N namespaces with the most type/subtype activity in the past T timeframe.

   1. "Show me the 5 namespaces with the most reads in the past 1 hour."

4. As a user of Tracker, I would like to see the top N programs reading/writing to a specific dataset in the past T timeframe.

   1. "Show me the top 5 programs writing to dataset1 in the past 1 hour."

Initial High Level Plan

• As messages come from the Kafka broker and are written to the AuditLog Table, when a message matches one of the metrics criteria, update metrics in a separate OLAP Cube (but the same dataset) as required.

• In the service layer, expose a new endpoint that allows users to query the data in the metrics table and returns the results in JSON.

Storing Metrics in AuditLog Dataset

• Add an additional Cube table to the AuditLog custom dataset to hold metrics.

• The properties of the cube will be as follows

  • Resolutions: 1h, 6h, 24h, 1w, 1m, 3m, 6m, 1y

  • Aggs: 

    • namespace (default ns1 ns2)

    • namespace,entity_type,entity_name (default,stream,stream1 default,dataset,dataset1)

    • namespace,entity_type,entity_name,program (default,stream,stream1,program1 default,dataset,dataset1,program2)

  • Measurements: 

    • access

    • read

    • write

    • unknown

    • create

    • update

    • truncate

    • delete

    • metadata_change

    • count

• Queries to OLAP cube

  • "Show me the total number of reads in the system in the past 1 hour."

    • {"aggregation":"agg2","resolution":3600,"startTs":now,"endTs":now-1h,"measurements":{"access_reads":"SUM"},"limit":1}

       

  • "Show me the 5 datasets with the most writes in the past 24 hours."

    • {"aggregation":"agg2","resolution":86400,"startTs":now,"endTs":now-24h,"measurements":{"access_writes":"SUM"},
"dimensionValues" : { "entity_type" : "dataset" },
"groupByDimensions":["namespace","entity_type","entity_name"],
"limit" : 10000
}

    • Results then sorted and top 5 returned

  • "Show me the 5 streams with the most metadata_changes in the past 7 days."

    • {"aggregation":"agg2","resolution":604800,"startTs":now,"endTs":now-7d,"measurements":{"metadata_changes":"SUM"},
"dimensionValues" : { "entity_type" : "stream" },
"groupByDimensions":["namespace","entity_type","entity_name"],
"limit" : 10000
}

    • Results then sorted and top 5 returned

  • "Show me the 5 namespaces with the most reads in the past 1 hour."

    • {"aggregation":"agg2","resolution":86400,"startTs":now,"endTs":now-1h,"measurements":{"access_reads":"SUM"},
"groupByDimensions":["namespace"],
"limit" : 10000
}

    • Results then sorted and top 5 returned

Endpoints