Backend is logging excessively
Description
Release Notes
Activity
Sagar Kapare September 11, 2015 at 5:39 PM
In order to pass the cdap configurations (cdap-default.xml and cdap-site.xml) to the mapper and reducer tasks, they are serialized into the hConf under property name "hconf.cconf".
While deserializing, we first create the new CConfiguration object which includes the cdap-default.xml and cdap-site.xml. Then to this object we add the value of "hconf.cconf" property as a new resource. This results in the CConfiguration object to contain duplicate configurations.
As a result we see lot of warning messages when the MapReduce program is run as mentioned in the JIRA.
In this PR, while deserializing, we create an empty CConfiguration object which will have no resource added to it and then to this object we add the value of "hconf.cconf" property as a new resource.
Tested on standalone and I no longer see those warning messages for app.template.plugin.dir and stream.instance.file.prefix
Running integration tests.
Sagar Kapare September 11, 2015 at 4:26 PM
https://github.com/caskdata/cdap/pull/3892
Tested on the standalone. Pending testing on the cluster.
Sreevatsan Raman July 31, 2015 at 8:09 PM
The problem occurs because we are overriding the final parameters - app.template.plugin.dir, and stream.instance.file.prefix. These files are only present in cdap-default.xml in the standalone this setting is not present in any other file.
CConfiguration class loads cConf and hConf. In MapReduceRuntime service we merge cConf into HConf that is causing this the final parameter to be present in both these configurations. The right fix is to avoid merging the cConf into hConf in MapReduceRuntimeService.
It is too risky to change is in the last minute of the release. Marking as a blocker for next.
Details
Details
Assignee
Reporter
Labels
Components
Fix versions
Priority
Blocker
When you run a big MR job, you come across logs like these. Probably not required.
2015-07-31 02:58:01,469 - WARN [LocalJobRunner Map Task Executor #1:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@1d4c15a9:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,470 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@a93980:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,470 - WARN [LocalJobRunner Map Task Executor #1:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@1d4c15a9:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,470 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@a93980:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,470 - WARN [LocalJobRunner Map Task Executor #5:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@65fe02d6:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,472 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@76c63ea:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,473 - WARN [LocalJobRunner Map Task Executor #5:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@65fe02d6:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,473 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@76c63ea:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,474 - WARN [LocalJobRunner Map Task Executor #4:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@c357a0b:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,474 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@adf6e70:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,474 - WARN [LocalJobRunner Map Task Executor #4:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@c357a0b:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,474 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@adf6e70:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,481 - WARN [LocalJobRunner Map Task Executor #3:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@6f39932b:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,482 - WARN [LocalJobRunner Map Task Executor #3:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@6f39932b:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,484 - WARN [LocalJobRunner Map Task Executor #0:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@2f488d8d:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,485 - WARN [LocalJobRunner Map Task Executor #0:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@2f488d8d:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,849 - WARN [LocalJobRunner Map Task Executor #3:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@7e641879:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,850 - WARN [LocalJobRunner Map Task Executor #3:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@7e641879:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,851 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@658cd29b:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,851 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@658cd29b:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,859 - WARN [LocalJobRunner Map Task Executor #0:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@778ae4b9:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,860 - WARN [LocalJobRunner Map Task Executor #0:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@778ae4b9:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,868 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@68e92aa4:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,869 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@68e92aa4:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,876 - WARN [LocalJobRunner Map Task Executor #4:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@6480f05a:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,876 - WARN [LocalJobRunner Map Task Executor #4:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@6480f05a:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,879 - WARN [LocalJobRunner Map Task Executor #5:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@341d9839:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,880 - WARN [LocalJobRunner Map Task Executor #5:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@341d9839:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,883 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@2368e74d:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,884 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@2368e74d:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,938 - WARN [LocalJobRunner Map Task Executor #1:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@1be5cd74:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,939 - WARN [LocalJobRunner Map Task Executor #1:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@1be5cd74:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,992 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@4034ea76:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,993 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@4034ea76:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,094 - WARN [LocalJobRunner Map Task Executor #0:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@44834e3e:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,094 - WARN [LocalJobRunner Map Task Executor #5:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@885eb96:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,095 - WARN [LocalJobRunner Map Task Executor #5:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@885eb96:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,095 - WARN [LocalJobRunner Map Task Executor #0:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@44834e3e:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,318 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@4e11b449:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,318 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@50ce22ce:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,319 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@4e11b449:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,319 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@50ce22ce:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,344 - WARN [LocalJobRunner Map Task Executor #1:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@7c67a834:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,344 - WARN [LocalJobRunner Map Task Executor #1:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@7c67a834:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,505 - WARN [LocalJobRunner Map Task Executor #4:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@2016693d:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,506 - WARN [LocalJobRunner Map Task Executor #4:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@2016693d:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,512 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@6a80b67b:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,513 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@6a80b67b:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,514 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@157baaff:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,515 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@157baaff:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,517 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@8736e3c:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,518 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@8736e3c:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,527 - WARN [LocalJobRunner Map Task Executor #3:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@14afc9cd:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,528 - WARN [LocalJobRunner Map Task Executor #3:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@14afc9cd:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,662 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@135a7682:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,662 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@135a7682:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,663 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@1d598ff6:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,665 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@1d598ff6:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,669 - WARN [LocalJobRunner Map Task Executor #5:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@5e0310f6:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,713 - WARN [LocalJobRunner Map Task Executor #5:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@5e0310f6:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,729 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@64e742d2:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,730 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@64e742d2:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,734 - WARN [LocalJobRunner Map Task Executor #1:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@620abcc0:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,735 - WARN [LocalJobRunner Map Task Executor #1:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@620abcc0:an attempt to override final parameter: stream.instance.file.prefix; Ignoring.