Page Comparison

...

• To configure the log level before an application starts, you can add the logger name as the key and log level as the value in the preferences, using the CDAP UI, CDAP CLI, or other command line tools. The logger name should be prefixed with system.log.level.

  For example, if you want to set the log level of a class named MyService with package my.sample.package to DEBUG, you would use system.log.level.my.sample.package.MyService as the key and DEBUG as the value. This can be applied to any package or classes. If the logger name is omitted, it will change the log level of ROOT.

• To configure the log level of a program dynamically, such as a service or worker which is currently running, see the Logging Microservices.

CDAP System Services Logs

...

• Logs are collected from individual programs running in YARN containers.

• YARN writes the log messages emitted by containers to files inside the container.

• In addition, CDAP programs publish these messages to Kafka.

• CDAP System Services run (depending on the service) either on cluster edge nodes or in YARN containers. Where they run determines the file that configures that service's logging.

• The Log Saver Service (log.saver) is configured to read log messages for the logs.user-v2 Kafka topic (set by the property log.kafka.topic). The number of log saver instances can be scaled to process the Kafka partitions in parallel, if needed.

  Log saver, by default, runs only the CDAP Log Pipeline: it reads the messages from Kafka, groups them by program or application, buffers and sorts them in memory, and finally persists them to files in HDFS. Each of these files corresponds to one program or application, depending on how the grouping is configured. (This is set by the property log.publish.partition.key, described below.)

  Note: These files are configured to rotate based on time and size; they can be changed using the properties log.pipeline.cdap.file.max.size.bytes and log.pipeline.cdap.file.max.lifetime.ms in the cdap-site.xml file as described in “Log Pipeline Configuration”.

• For security, the files written out to persistent storage in HDFS have permissions set such that they are accessible only by the cdap user.

• In addition, custom log pipelines can be configured by adding an XML file in a prescribed location. Each pipeline buffers log messages in memory and sorts them based on their timestamp.

• In addition to persisting logs to files, the log saver also emits metrics about the number of log messages emitted by each program. These metrics can be retrieved by querying the CDAP metrics system.

  These tables list the metrics from the section Available System Metrics of the Metrics Microservices. See that section for further information.

  Application Logging MetricDescriptionsystem.app.log.{error, info, warn}Number of error, info, or warn log messages logged by an application or applicationsSystem Services Logging MetricDescriptionsystem.services.log.{error, info, warn}Number of error, info, or warn log messages logged by a system service or system services

...