This was actually a schedule hbase table whose region was stuck in transition for the past ~25 hours. Scheduler upgrade could not happen, which blocked run records from being persisted to hbase. This dependency will be removed as a part of CDAP-12563.
Albert ShauSeptember 7, 2017 at 6:12 PM
These log messages are misleading and should have been updated when the corrector was switched to just send a message on TMS.
The TMS message should indicate that the state change was initiated by the corrector, and whatever is in charge of performing the actual state change should log these messages when the state is actually changed.
Ali AnwarSeptember 7, 2017 at 6:19 AM
There might be an issue with the TMS service or the subscriber (more likely the latter). To investigate further - see if any run record updates are working. For instance, launch a simple program and see if its run record lifecycle is fine. Also, it'd be good to know what the current run record state is.
Duplicate
Pinned fields
Click on the next to a field label to start pinning.
When CDAP detects an incorrect RunRecord the logs indicate that the records are corrected but the status is never updated in HBase.
In this environment the app was killed while CDAP was stopped.