| Table of Contents | ||||
|---|---|---|---|---|
|
CDAP collects logs and metrics for all of its internal services and user applications. Being able to view these details can be very helpful in debugging CDAP applications as well as analyzing their performance. CDAP gives access to its logs, metrics, and other monitoring information through Microservices, the CDAP UI, as well as a Java Client.
...
You can also use a custom "logback" file with your application, as described in the Developer Manual section Application Logback.
Changing Program Log Levels
...
These properties control the writing of logs to Kafka:
Parameter Name | Default Value | Description |
|---|---|---|
|
| Kafka topic name used to publish logs |
|
| Number of CDAP Kafka service partitions to publish the logs to |
|
| Publish logs from an application or a program to the same partition. Valid values are "application" or "program". If set to "application", logs from all the programs of an application go to the same partition. If set to "program", logs from the same program go to the same partition. Changes to this property requires restarting of all CDAP applications. |
Notes:
If an external Kafka service is used (instead of the CDAP Kafka service), the number of partitions used for
log.publish.num.partitionsmust match the number set in the external service for the topic being used to publish logs (log.kafka.topic).By default,
log.publish.partition.keyis set toprogram, which means that all logs for the same program go to the same partition. Set this toapplicationif you want all logs from an application to go to the same instance of the Log Saver Service.
...
These properties control the Log Saver Service:
Parameter Name | Default Value | Description |
|---|---|---|
|
| Maximum number of log saver instances to run in YARN |
|
| Number of log saver instances to run in YARN |
|
| Memory in megabytes for each log saver instance to run in YARN. |
|
| Number of virtual cores for each log saver instance in YARN |
Log saver instances should be from a minimum of one to a maximum of ten. The maximum is set by the number of Kafka partitions (log.publish.num.partitions), which by default is 10.
...
These properties control the CDAP log pipeline:
Parameter Name | Default Value | Description |
|---|---|---|
|
| Permissions used by the system log pipeline when creating directories |
|
| Batch size to clean up log metadata table |
|
| Maximum time span in milliseconds of a log file created by the system log pipeline |
|
| Maximum size in bytes of a log file created by the system log pipeline |
|
| Permissions used by the system log pipeline when creating files |
|
| Time in days a log file is retained |
These properties control both the CDAP log pipeline and custom log pipelines:
Parameter Name | Default Value | Description |
|---|---|---|
|
| The time between log processing pipeline checkpoints in milliseconds |
|
| A local directory on the CDAP Master that is scanned for log processing pipeline configurations. Each pipeline is defined by a file in the logback XML format, with ".xml" as the file name extension. |
|
| The time a log event stays in the log processing pipeline buffer before writing out to log appenders in milliseconds. A longer delay will result in better time ordering of log events before presenting to log appenders but will consume more memory. |
|
| The buffer size in bytes, per topic partition, for fetching log events from Kafka |
|
| Comma-separated list of local directories on the CDAP Master scanned for additional library JAR files to be included for log processing |
The log.process.pipeline.* properties can be over-ridden and specified at the custom pipeline level by providing a value in a pipeline's "logback" file for any of these properties.
...