HBase Coprocessors in CDAP

CDAP uses the following HBase Coprocessors as of release 3.6.0

Transaction Coprocessor

Source: TransactionProcessor.java and CDAP's extension DefaultTransactionProcessor.java

Transaction Coprocessor is used to ensure that data reads and writes confirm to transactional guarantees. It also cleans up data from invalid transactions on flush and compactions.

It runs on HBase region servers and intercepts the following operations -

  • preGetOp
  • preDelete
  • preScannerOpen
  • preFlushScannerOpen
  • preCompactScannerOpen

Increment Coprocessor

Source: IncrementHandler.java

Increment coprocessor is used for readless increments. It converts an increment into a delta increment without reading the previous value. It also sums up delta increments on reads or a flush or compaction.

It runs on HBase region servers and intercepts the following operations -

  • preGetOp
  • prePut
  • preDelete
  • preScannerOpen
  • postScannerOpen
  • preFlush
  • preCompact

Queue Coprocessors

CDAP flow-queue has two coprocessors on HBase region servers.

  1. Source: HBaseQueueRegionObserver.java 
    • Responsibles for eviction of consumed data
    • Overrides the preFlush and preCompact methods
      • Returns an InternalScanner that will skip over cells that are already processed based on the queue logic
  2. Source: DequeueScanObserver.java
    • Responsibles for scan time filtering based on transaction information provided by the client
    • Overrides the preScannerOpen method
      • Returns a RegionScanner that will skip over cells that are not yet committed based on the transaction information

Created in 2020 by Google Inc.