kafka.server properties not deprecated properly
Description
Release Notes
Activity
Derek WoodSeptember 9, 2016 at 12:49 AM
So likely what I observed on the CM cluster was the following:
1. kafka.log.dir is not exposed as a configuration in the CSD. Instead, the CSD automatically adds it to cdap-site.xml with the default location (the only guaranteed writable directory). To modify kafka.log.dir, the user must use the safety valve, which overrides the automatically generated default (end result is only a single kafka.log.dir with the safety valve value).
2. For 3.5.0, the automatic default was updated to use the new property name kafka.server.log.dirs
3. Now, if you try to specify kafka.log.dir using the safety valve as usual, you end up with a cdap-site.xml with both properties: kafka.server.log.dirs (via automatic default) and kafka.log.dir (via safety valve). CDAP code will use the new property.
4. This means that when upgrading from 3.4 to 3.5, the user would need to know to update their safety valve settings to preserve any customized kafka.log.dir.
Derek WoodSeptember 8, 2016 at 11:49 PM
hmmm, I was seeing issues on a CM cluster, but mostly submitted this bug after Poorna looked at the code and said there was a bug I guess it's possible the issues I saw were unrelated... does it actually obey 'kafka.log.dir', if it's the only related property set?
Sagar KapareSeptember 8, 2016 at 7:37 PM
@Derek Wood I tried reproducing the issue on 3.5 cluster. I set the following property in the cdap-site.xml
<property>
<name>kafka.log.dir</name>
<value>/data/cdap/kafka-logs-another</value>
</property>When I start the kafka server in the kafka logs, I see that the setting for the log directory is getting picked up correctly
16/09/08 19:17:41 INFO utils.VerifiableProperties: Property log.dirs is overridden to /data/cdap/kafka-logs-anotherWhen kafka server is started in 3.4 we used to set the kafka property "log.dir", however in 3.5 we started setting the kafka property "log.dirs". However per kafka documentation both properties should work with "log.dirs" taking precedence.
Can you please provide more information about the environment used?
John JacksonAugust 26, 2016 at 3:29 AM
Docs in release/3.5 updated in PR https://github.com/caskdata/cdap/pull/6590
in 3.5.0, new
kafka.server.*properties deprecated older properties such askafka.log.dir. The problem is, the old properties are now just ignored. For proper deprecation, the old properties should still work.kafka.log.dir -> kafka.server.log.dirs: either should work