Viewing and downloading logs in the Pipeline Studio

For each data pipeline run, CDAP generates logs to help you monitor data pipeline runs and debug issues. You can view logs and filter the log levels as soon as the pipeline run begins.

To view a log file, from the Preview page or Pipeline details page, click Logs:

Open

To monitor the progress of a pipeline, click Logs while the pipeline is running.

Viewing different log levels

When viewing the pipeline log messages, you can filter the logs by different levels:

• Error

• Warning

• Informational

• Debug

• Trace

To set the log level, click the drop-down menu next to Level and select the types of messages you want to appear in the log:

Open

Basic and advanced logs

For each pipeline run, you can view basic or advanced logs. Basic logs include messages about the stages of the pipeline. Advanced logs include all of the basic messages as well as information about the cluster health and the execution engine.

To view advanced logs at any time during a pipeline preview or run, click View Advanced Logs:

To return to viewing basic logs, click Hide Advanced Logs.

Viewing the latest log messages

Starting in 6.1.4, when you view a log, the Pipeline Detail page displays the latest log messages. If you have a pipeline run with a large number of messages, you can click Scroll to Latest Logs to quickly jump to the latest messages.

Downloading logs

After a pipeline run completes, you can download a log file that includes all the advanced log messages and stack trace messages to help you debug any issues. To download the log, click Download All.

You can also view the raw log as plain text in a new browser tab. Click the drop-down menu next to Download All and select View Raw Logs.