The cluster was previously running on CDAP 3.5.
It was upgraded to 4.0.0-SNAPSHOT. Upon running the upgrade tool, I see the following errors.
It indicates that DatasetService is unable to load dataset types, due to the data being in a different format than requested (due to Id --> EntityId changes).
This is due to the changes in https://github.com/caskdata/cdap/pull/6735/files#diff-97ff04cf135c1921e196b23f2b6fc93f
After the UpgradeTool ran, I started CDAP.
I ran the "list dataset types" command from the CLI and see this in the master logs:
We either need an upgrade step for this, or change the DatasetTypeManager to expect the old format.
Edit: Leaning towards the latter, to handle things in the code as much as possible.
It seems that only the mapping from dataset type to data module id in the DatasetTypeMDS is affected (all other data structures in the dataset MDS do not the Id classes). Here is the difference: