Use Hadoop configuration files bundled with Explore Service

Description

YARN containers run through Cloudera SCM Agent do not have complete mapred-site.xml in the classpath. Hence Explore container will not be able to launch mapreduce tasks for Hive queries. This can be overcome by serializing Hadoop config files on Master and transferring it to Explore container during its launch. Even though Hadoop config files are bundled with Explore Service, they were not being used as /etc/hadoop/conf appeared first in classpath.

Release Notes

None

Linked issues

relates to

CDAP-1178

Test explore with CDH 5.3.0

CDAP-3754

explore container fails on MapR5

CDAP-3820

Remove distro specific code used in Explore container start-up

Activity

Show:

Poorna Chandra September 21, 2015 at 6:45 PM

PR https://github.com/caskdata/cdap/pull/1116

Poorna Chandra January 23, 2015 at 9:51 PM

Stacktrace of error seen when mapreduce job cannot be started by Explore container -

2015-01-17 02:32:53,210 WARN  [main]: security.UserGroupInformation (UserGroupInformation.java:doAs(1645)) - PriviledgedActionException as:yarn
 (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: hdfs://dwcm122-1000.dev.continuuity.net:8020/yarn/nm/usercache/cdap/ap
pcache/application_1421447091990_0042/container_1421447091990_0042_01_000008/tmp/twill.launcher-1421458743961-0/lib/jsp-api-2.1-6.1.14.jar
2015-01-17 02:32:53,211 WARN  [main]: security.UserGroupInformation (UserGroupInformation.java:doAs(1645)) - PriviledgedActionException as:yarn
 (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: hdfs://dwcm122-1000.dev.continuuity.net:8020/yarn/nm/usercache/cdap/ap
pcache/application_1421447091990_0042/container_1421447091990_0042_01_000008/tmp/twill.launcher-1421458743961-0/lib/jsp-api-2.1-6.1.14.jar
2015-01-17 02:32:53,218 ERROR [main]: mr.ExecDriver (SessionState.java:printError(565)) - Job Submission failed with exception 'java.io.FileNotFoundException(File does not exist: hdfs://dwcm122-1000.dev.continuuity.net:8020/yarn/nm/usercache/cdap/appcache/application_1421447091990_0042/container_1421447091990_0042_01_000008/tmp/twill.launcher-1421458743961-0/lib/jsp-api-2.1-6.1.14.jar)'
java.io.FileNotFoundException: File does not exist: hdfs://dwcm122-1000.dev.continuuity.net:8020/yarn/nm/usercache/cdap/appcache/application_1421447091990_0042/container_1421447091990_0042_01_000008/tmp/twill.launcher-1421458743961-0/lib/jsp-api-2.1-6.1.14.jar
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1093)
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085)
        at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
        at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
        at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93)
        at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
        at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:267)
        at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:388)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:481)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
        at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:564)
        at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:559)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:559)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:550)
        at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:420)
        at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:740)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Fixed

Pinned fields

Click on the next to a field label to start pinning.

Details

Assignee

Poorna Chandra

Reporter

Poorna Chandra

Priority

Major

Created January 23, 2015 at 8:53 AM

Updated September 23, 2015 at 1:35 AM

Resolved January 30, 2015 at 3:38 AM

Configure