We're updating the issue view to help you get more done. 

User exception in pipeline logs should be in app scope

Description

User Exception that is thrown, as an example - "User class threw exception" should be in app scope. It has useful information -

"2020-06-11 19:18:56,835 - ERROR [Driver:o.a.s.d.y.ApplicationMaster@91] - User class threw exception: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory gs://demo-outputs/test/2020-06-11-19-14 already exists"

Release Notes

Includes all ERROR level logs logged under the application logging context

Activity

Show:
Sreevatsan Raman
June 15, 2020, 5:01 PM

One more use case for capturing the system logs in app context is failure due to memory issues

Example: "java.lang.IllegalArgumentException: System memory 268435456 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
"

Terence Yim
June 15, 2020, 5:26 PM

This error message is from Spark as well?

Terence Yim
June 15, 2020, 7:55 PM

We can have all ERROR logs from app execution to be tagged as program log.

Sreevatsan Raman
June 15, 2020, 9:10 PM

This error is from Spark.

Fixed

Assignee

Terence Yim

Reporter

Sreevatsan Raman

Labels

None

Docs Impact

None

UX Impact

None

Fix versions

Priority

Major
Configure