Cannot run upgrade tool on the same cluster more than once


Running upgrade tool for more than once results in following error:

Steps to reproduce:
1.) Create 3.5 cluster and install 4.1 rpms
2.) Run upgrade tool twice

Release Notes



Albert Shau
February 23, 2017, 11:32 PM

somehow, that table has the exact same coprocessor twice:

We remove HBaseQueueRegionObserver, then add HBaseQueueRegionObserver, and it fails because there's still one HBaseQueueRegionObserver in the list. Not sure how 2 of the same coprocessor got there to begin with...

Albert Shau
February 24, 2017, 2:31 AM

this is a tricky bug. It only affects tables that have more than 1 coprocessor.

The root cause is the logic that converts an HTableDescriptor into a TableDescriptor for the HBase SPI. Internally, when you call addCoprocessor() on an HTableDescriptor, it converts the coprocessor information into a property with key 'coprocessor$[num]', where 'num' is the max coprocessor number. So if there is a table with 2 existing coprocessors, when we add the new updated coprocessors, it ends up adding 'coprocessor$3' and 'coprocessor$4' to the coprocessor properties. When we delete the old coprocessors from the descriptor, 'coprocessor$1' and 'coprocessor$2' get removed. Then, when we convert it to a TableDescriptor, we copy all properties into the TableDescriptor, which means 'coprocessor$3' and 'coprocessor$4' properties get copied. In addition, the TableDescriptor contains the CoprocessorDescriptor for the 2 new coprocessor. When we convert that TableDescriptor back into an HTableDescriptor, we copy all properties and we add all coprocessors. This results in 'coprocessor$1' and 'coprocessor$2' from the addCoprocessor() call, as well as 'coprocessor$3' and 'coprocessor$4' from the properties.

To fix, we'll just make sure not to copy coprocessor properties into the TableDescriptor.

Albert Shau
February 24, 2017, 2:39 AM


Albert Shau


Chengfeng Mao



Docs Impact


UX Impact



Fix versions

Affects versions