Backend is logging excessively

Description

When you run a big MR job, you come across logs like these. Probably not required.

2015-07-31 02:58:01,469 - WARN [LocalJobRunner Map Task Executor #1:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@1d4c15a9:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,470 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@a93980:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,470 - WARN [LocalJobRunner Map Task Executor #1:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@1d4c15a9:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,470 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@a93980:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,470 - WARN [LocalJobRunner Map Task Executor #5:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@65fe02d6:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,472 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@76c63ea:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,473 - WARN [LocalJobRunner Map Task Executor #5:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@65fe02d6:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,473 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@76c63ea:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,474 - WARN [LocalJobRunner Map Task Executor #4:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@c357a0b:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,474 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@adf6e70:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,474 - WARN [LocalJobRunner Map Task Executor #4:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@c357a0b:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,474 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@adf6e70:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,481 - WARN [LocalJobRunner Map Task Executor #3:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@6f39932b:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,482 - WARN [LocalJobRunner Map Task Executor #3:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@6f39932b:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,484 - WARN [LocalJobRunner Map Task Executor #0:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@2f488d8d:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,485 - WARN [LocalJobRunner Map Task Executor #0:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@2f488d8d:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,849 - WARN [LocalJobRunner Map Task Executor #3:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@7e641879:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,850 - WARN [LocalJobRunner Map Task Executor #3:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@7e641879:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,851 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@658cd29b:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,851 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@658cd29b:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,859 - WARN [LocalJobRunner Map Task Executor #0:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@778ae4b9:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,860 - WARN [LocalJobRunner Map Task Executor #0:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@778ae4b9:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,868 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@68e92aa4:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,869 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@68e92aa4:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,876 - WARN [LocalJobRunner Map Task Executor #4:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@6480f05a:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,876 - WARN [LocalJobRunner Map Task Executor #4:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@6480f05a:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,879 - WARN [LocalJobRunner Map Task Executor #5:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@341d9839:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,880 - WARN [LocalJobRunner Map Task Executor #5:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@341d9839:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,883 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@2368e74d:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,884 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@2368e74d:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,938 - WARN [LocalJobRunner Map Task Executor #1:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@1be5cd74:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,939 - WARN [LocalJobRunner Map Task Executor #1:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@1be5cd74:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:01,992 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@4034ea76:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:01,993 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@4034ea76:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,094 - WARN [LocalJobRunner Map Task Executor #0:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@44834e3e:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,094 - WARN [LocalJobRunner Map Task Executor #5:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@885eb96:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,095 - WARN [LocalJobRunner Map Task Executor #5:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@885eb96:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,095 - WARN [LocalJobRunner Map Task Executor #0:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@44834e3e:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,318 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@4e11b449:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,318 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@50ce22ce:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,319 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@4e11b449:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,319 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@50ce22ce:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,344 - WARN [LocalJobRunner Map Task Executor #1:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@7c67a834:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,344 - WARN [LocalJobRunner Map Task Executor #1:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@7c67a834:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,505 - WARN [LocalJobRunner Map Task Executor #4:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@2016693d:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,506 - WARN [LocalJobRunner Map Task Executor #4:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@2016693d:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,512 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@6a80b67b:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,513 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@6a80b67b:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,514 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@157baaff:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,515 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@157baaff:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,517 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@8736e3c:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,518 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@8736e3c:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,527 - WARN [LocalJobRunner Map Task Executor #3:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@14afc9cd:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,528 - WARN [LocalJobRunner Map Task Executor #3:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@14afc9cd:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,662 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@135a7682:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,662 - WARN [LocalJobRunner Map Task Executor #2:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@135a7682:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,663 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@1d598ff6:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,665 - WARN [LocalJobRunner Map Task Executor #7:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@1d598ff6:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,669 - WARN [LocalJobRunner Map Task Executor #5:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@5e0310f6:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,713 - WARN [LocalJobRunner Map Task Executor #5:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@5e0310f6:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,729 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@64e742d2:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,730 - WARN [LocalJobRunner Map Task Executor #6:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@64e742d2:an attempt to override final parameter: stream.instance.file.prefix; Ignoring. 2015-07-31 02:58:02,734 - WARN [LocalJobRunner Map Task Executor #1:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@620abcc0:an attempt to override final parameter: app.template.plugin.dir; Ignoring. 2015-07-31 02:58:02,735 - WARN [LocalJobRunner Map Task Executor #1:c.c.c.c.c.Configuration@1748] - java.io.ByteArrayInputStream@620abcc0:an attempt to override final parameter: stream.instance.file.prefix; Ignoring.

Release Notes

None

Activity

Show:

Sagar Kapare September 11, 2015 at 5:39 PM

In order to pass the cdap configurations (cdap-default.xml and cdap-site.xml) to the mapper and reducer tasks, they are serialized into the hConf under property name "hconf.cconf".

While deserializing, we first create the new CConfiguration object which includes the cdap-default.xml and cdap-site.xml. Then to this object we add the value of "hconf.cconf" property as a new resource. This results in the CConfiguration object to contain duplicate configurations.

As a result we see lot of warning messages when the MapReduce program is run as mentioned in the JIRA.

In this PR, while deserializing, we create an empty CConfiguration object which will have no resource added to it and then to this object we add the value of "hconf.cconf" property as a new resource.

Tested on standalone and I no longer see those warning messages for app.template.plugin.dir and stream.instance.file.prefix

Running integration tests.

Sagar Kapare September 11, 2015 at 4:26 PM

https://github.com/caskdata/cdap/pull/3892
Tested on the standalone. Pending testing on the cluster.

Sreevatsan Raman July 31, 2015 at 8:09 PM

The problem occurs because we are overriding the final parameters - app.template.plugin.dir, and stream.instance.file.prefix. These files are only present in cdap-default.xml in the standalone this setting is not present in any other file.

CConfiguration class loads cConf and hConf. In MapReduceRuntime service we merge cConf into HConf that is causing this the final parameter to be present in both these configurations. The right fix is to avoid merging the cConf into hConf in MapReduceRuntimeService.

It is too risky to change is in the last minute of the release. Marking as a blocker for next.

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Components

Fix versions

Priority

Created July 31, 2015 at 10:04 AM
Updated December 9, 2020 at 8:58 PM
Resolved September 12, 2015 at 12:44 AM