Mapreduce metrics aren't recorded on a secure hadoop cluster

https://github.com/caskdata/cdap/pull/1838

Fixing will require quite some effort: changing the way we run MR jobs. At same time we have to fix it. Keeping in 3.0 - but not sure how to plan for it. It could be at least week of work based on above.

moving out to 3.0 release

Moved to 2.7. We want to put this into the "known issues". There seem to be no quick fix that we can squeeze into 2.6.

The summary is:

• we launch mapreduce job from a twill container - mr job runner - as that one executes user code, we want it to be executed in isolated env, as that user

• that twill container watches for MR job to complete and reports its stats

• to report stats mr job runner polls MR AM for mr job status

• in secure cluster to talk to MR AM you need to have token

• only kerberos authenticated clients can aqcuire tokens, which the mr job runner is not: all twill containers are not

One other option to fix is to have service as a part of cdap-master that reports stats for all mr jobs.

The MapreduceProgramRunner get Token from the master to communicate with RM, but when the MapreduceProgramRunner launches the mapreduce job, it does not have the token for the AM (MR job) to communicate and get the metrics counters. but it can talk with RM and we guess that's how it knows the MR job's status.

since MapreduceProgramRunner is the one that start's AM, and it cannot communicate with AM (as it does not have the token for AM) it cannot get the token for AM.

Possible fix would be for Master to generate token for the Mapreduce job identified by a name and send it to MapreduceProgramRunner, so it can communicate with AM using that. but need to check the security implications of that.