...
Various cleanup of data needs to be performed. And the clean up strategy varies based on the underlying storage - LevelDB vs HBase. The matrix below describes the combinations and types of cleanup we need to do and how it is going to be performed.
| HBase | LevelDB | ||
|---|---|---|---|
| Cleanup Strategy | Coprocessors attached to tables (one for messageTable and another for payloadTable). No cleanup is required for MetadataTable. | Periodically scheduled thread that cleans up data in MessageTable and PayloadTable. Frequency of schedule is configurable via cdap-site.xml. | |
| TTL Expiration (MessageTable and Payload Table) | Get TableId (namespace, topic) from rowKey and get the TopicMetadata from the MetadataTable (cached). From the cell rowkey, get the timestamp of write and use the TTL and determine if that cell needs to be skipped. | Scan the tables, topic wise and remove rows that have exceeded TTL based on TopicMetadata info. | |
| Older Generation (MessageTable and PayloadTable) | Check the generation id of the row and compare it with the one we get from MetadataTable. If it is the current generation, then do nothing. If it is an older generation (gen < abs(currentgen) || gen == -1*currentgen), then skip the cell. | Same logic as in HBase. While pruningMessages, scan the should start with generation '1' of that topic. | |
| Invalid Transactions (MessageTable only) | This requires (periodically refreshed) tx.snapshot in the Coprocessor. If the cell belongs to a TX_COL column, then get the tx id from it. If the transaction id is present in the invalid list (from the tx.snapshot), then invert the sign (-1 * tx_id) and put back the value. This way, the data is still visible to non-tx consumption and will be eventually cleared by TTL or when the topic is deleted. | Not necessary in LevelDB since we don't support pruning invalid transactions in SDK! | |
| Latest min tx timestamp (for manual invalid TX pruning, required only for MessageTable) | The last used tx.snapshot info can be directly used to prune the invalid transaction list! So we need to log that info. Also need to write a TMS table debugger tool that can print this info as well. |
Updates: Please refer to
for the new levelDB cleanup logic.Jira Legacy server System JIRA serverId 39880e21-34be-3f5d-91a2-e11c549f905d key CDAP-17939