Logging Microservices

Use the CDAP Logging Microservices to download the logs of applications and the system, with the option of formatting and filtering the logs that are downloaded, and setting the log levels of programs at runtime when under Distributed CDAP mode.

Additional details on logging can be found in Logging and Monitoring.

All methods or endpoints described in this API have a base URL (typically http://<host>:11015 or https://<host>:10443) that precedes the resource identifier, as described in the Microservices Conventions. These methods return a status code, as listed in the Microservices Status Codes.

Downloading Application Logs

Logs emitted by a MapReduce, service, Spark, worker, or workflow program running in CDAP can be downloaded with the Logging Microservices. To do that, send an HTTP GET request:

Example

Downloading Application Logs for a Run

To download logs for a program run, send an HTTP GET request:

Example

Downloading System Logs

Logs emitted by a system service running in CDAP can be downloaded with the Logging HTTP API. To do that, send an HTTP GET request:

where:

Note that the start and stop times are not optional.

Example

Formatting and Filtering

Formatting

The output is, by default, formatted as HTML-embeddable text; that is, characters that have a special meaning in HTML will be escaped. A line of a log may look like this:

The text can also be formatted as JSON. To do so, suffix your URL with the parameter &format=json.

Filtering

A filter string can be supplied as an optional parameter. It will filter the returned log entries to those that match the supplied string.

Changing System Service Log Levels

Log levels can be set for any system service at runtime. Once changed, they can be reset back to what they were by using the system service reset endpoint shown below.

Note: Dynamically-changing log levels for system services is only supported under Distributed CDAP.

Setting System Service Log Levels

To set the log levels of a system service at runtime under Distributed CDAP, submit an HTTP PUT request:

with a JSON map in the request body consisting of pairs of logger names and log levels, such as:

Note that changing the log levels of the appfabric service will also change the log levels of the dataset.service as they run on the same node.

For example:

will update the log levels of the system service metrics.processor with the log level of io.cdap.cdap changed to ERROR and the log level of ROOT changed to TRACE.

Resetting System Service Log Levels

Resetting the log levels will change the log levels back to what they were when the system service was started.

To reset the log levels of a system service, at runtime under Distributed CDAP mode, submit an HTTP PUT request:

with an optional JSON array in the request body consisting of the logger names to be reset, such as:

If the body is not provided or is empty, it will reset the log levels of all logger names.

Note that resetting the log levels of the appfabric service will also rest the log levels of the dataset.service as they run on the same node.

For example:

will reset the log level of the system service metrics.processor with log levels of io.cdap.cdap and ROOT reset back to what they were when the metrics.processor service was started.

Changing Program Log Levels

Log levels can be set for a particular run of a program. Once changed, they can be reset back to what they started with by using the program reset endpoint shown below.

Setting Program Log Levels

To set the log levels of a program run at runtime under Distributed CDAP, submit an HTTP PUT request:

To set the log levels of a program run in a particular version of an application:

and, in all cases, with a JSON map in the request body consisting of pairs of logger names and log levels, such as:

For example:

will update the log levels of the service UploadService in the SportResults application with the log level of io.cdap.cdap changed to ERROR and log level of ROOT changed to TRACE.

Updating the Log Level for a Running Program

To update the log level for a running program according to the request body. Currently supported program types are Service and Worker. The request body is expected to contain a map of log levels, where key is loggername and value is one of (FATAL, ERROR, WARN, INFO, DEBUG, TRACE).

Path Parameters

JSON Parameters

Resetting Program Log Levels

Resetting the log levels will change the log levels back to what they were when the program was started.

To reset the log levels of a program run at runtime under Distributed CDAP mode, submit an HTTP PUT request:

To reset the log levels of a program run in a particular version of an application:

and, in all cases, with an optional JSON array in the request body consisting of the logger names to be reset, such as:

If the body is not provided or is empty, it will reset the log levels of all logger names.

For example:

will reset the log level of the service UploadService in the SportResults application with the log levels of io.cdap.cdap and ROOT reset back to what they were when the flow was started.

Resetting System Log Levels for a Running Program

Reset the log level for a running program according to the request body. Currently supported program types are Service and Worker. The request body can either be empty, which will reset all loggers for the program, or contain a list of logger names, which will reset for these particular logger names for the program.

Path Parameters

JSON Parameters