User Exception that is thrown, as an example - "User class threw exception" should be in app scope. It has useful information -
"2020-06-11 19:18:56,835 - ERROR [Driver:o.a.s.d.y.ApplicationMaster@91] - User class threw exception: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory gs://demo-outputs/test/2020-06-11-19-14 already exists"
Includes all ERROR level logs logged under the application logging context
One more use case for capturing the system logs in app context is failure due to memory issues
Example: "java.lang.IllegalArgumentException: System memory 268435456 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
This error message is from Spark as well?
We can have all ERROR logs from app execution to be tagged as program log.
This error is from Spark.