Goals

• Improving performance of single Tephra Transaction Server
• Make Tephra Transaction server scale horizontally
• Make Tephra Transaction server Highly Available (HA) with Isolation
• Improve operational aspects of Transaction Server
• Improve performance of Workflow scheduling to schedule 1000s of jobs / second

Areas of Focus

• Transaction Invalid List Management
• Tephra Single Server Performance Improvements
• Isolation and Scalability of Tephra Transaction Server
• Improving scheduling performance of Workflow system

High Level Requirements

Transaction Invalid List Management

• System should automatically handle pruning of the transaction invalid list
• Reduce operational complexity for running manual steps to prune invalid transaction list
• Applied during major and striped compaction
• Metrics around the current invalid list size
• Tool to inspect and report progress on pruning

Tephra Performance Improvement

• Single Tephra Server should be able to support up-to ~ 10K transactions/second
• Support read-only and hierarchical conflict detection

Scale and Isolation

• Run multiple instances of Tephra Transaction Server in active-active in single DC or multiple DC
• Isolation at namespace level 

Technical Breakdown

P&S-001 : Invalid list pruning with major compaction

Currently, the invalid list keeps growing over time, if it's not pruned periodically using the manual process ( which is very tricky, time consuming and hard to operationalize ) the performance of transaction server gets affected. If we remove the manual process and make the list pruning automatic, it would reduce the operational complexity and also help improve the performance of the transactions. This will be implemented as hook into major compaction, meaning that the invalid list pruning would be triggered during major compaction. 

P&S-002 : Tool(s) and User Interface to inspect and report progress on pruning

A tool and user interface that can show the progress of pruning when running, impact of pruning on invalid list, show any regions that are behind preventing pruning on invalid transaction list.

P&S-003 : Performance improvements of Single Transaction server

This will be on focused on improving the performance of single transaction server. As part of this we will be improving on the locking granularity during conflict detection, support read-only transactions, improve group commit efficiency, hierarchical conflict detection, transmit only latest snapshot of invalid list and more. 

P&S-004 : Transaction server per namespace for resource isolation

Instead of using a single transaction server across all namespace, we would like to be able to have multiple instance of transaction servers supporting isolation for namespaces. So, there could be one instance of transaction server that would be supporting a namespace and that responsibility could be rotate among different instances of transaction server.  

P&S-005 : Support for running multiple instance of Transaction server in Active-Active mode

Running multiple instance of transaction server in active-active mode to shed load or for disaster recovery. This will also tie in with many of the stories for replication. 

P&S-006 : Invalid List pruning with stripped compaction

This will provide the capability to prune the invalid list during the stripped compaction or minor compaction depending on flexibility to do so. 

Open Questions

Action Items