If we have an impersonated namespace with a non-default keytab url, that is, keytab not under the value of security.keytab.path, then creation of the local datasets for workflow will fail with the following message:
This is happening since currently we are creating the local dataset with the principal from the ProgramOptions. When the principal is not null, cdap will try to look for the keytab file in the default path, which result in this error.
Creation on local datasets works fine if the namespace's keytab url is in the default path.
It should never read the namespace key tab URI and it should not be readable at all even if the container attempts to do so.
why it can read the key tab if it is in default location? Is it on HDFS? The permission should be readable by the "cdap" user only and the container should be launched with the impersonated user.
The container is launched by the impersonated user but it is not this user who is actually reading the keytab file. The container will just provide the principal for the dataset creation, using the addInstance(..., ..., principal) method in DatasetFramework. This call will get to the DatasetInstanceService to create the dataset, which is done by the "cdap" user I think. Since the principal is provided, the DefaultUGIProvider will try to look for the keytab file in the default keytab path.
On CDAP 4.2.0, seems like local datasets (even on a non-impersonated namespace) have the same failure affecting them.