| Table of Contents |
|---|
...
In DefaultStore, each time when a program run stops, the start time and the stop time of the program run will be recorded in the row key in the index table with row key <namespace-id>:<inverted-start-time>:<stop-time>:<program-run-id>
Read path:
Given a query for active runs from a given namespace <query-namespace> within the time range [start, end), the scanning will be started from the start row key: <query-namespace>:0 since we don't know what is the earliest start time of the active runs within the given time range [start, end), and stop at the stop row key <query-namespace>:<end>. A possible optimization is discussed in the next section for getting a larger start row key with the earliest start time of the runs in a given time range. After getting the runs with program_start_time < end, a row key filter will be applied on these runs to get runs with program_stop_time > begin. Then corresponding run records will be read from the AppMetaStore with these run id's.
...
The read path of this approach will be almost the same as in Approach 2, except the range of reading around the query time range will depend on the period of such metrics is emitted. For instance, if the metric is emitted every 30 minutes, and the query time range is 7:30am - 9pm, then scanning range will be 7:00am - 9am.
Performance Experiment with Approach 1 optimization VS. no optimization
Experiments were run to evaluate the performance of indexing run records compared to the existing brute force method of scanning the whole run record table:
...