Debugging CDAP system services

The intent of this wiki page is to document debugging tools/endpoints for CDAP system services.

Thread stack trace endpoint

A Java thread dump is a snapshot of what every thread in the JVM is doing at a particular point in time. Each thread in the JVM is listed with it's name and id, it's current state and the Java call stack showing what monitor it has locked or is waiting on. This is especially useful if CDAP system services seems to hang when running under load, as an analysis of the dump will show where the threads are stuck, either by deadlock or other thread contention.

CDAP exposes below REST endpoint to get thread stack trace of CDAP system services.

GET v3/system/services/{service-name}/stacks?depth=10
where,
service-name is name of CDAP system service 
depth is the maximum number of entries in the stack trace to be dumped. Integer.MAX_VALUE could be used to request the entire stack to be dumped.

Below are the REST endpoints to get thread stack trace for each CDAP system service.

System service	REST Endpoint

System service	REST Endpoint
App Fabric	system/services/appfabric/stacks?depth=2147483647
Dataset Executor	system/services/dataset.executor/stacks?depth=2147483647
Explore Service	system/services/explore.service/stacks?depth=2147483647
Log Saver	system/services/log.saver/stacks?depth=2147483647
Messaging Service	system/services/messaging.service/stacks?depth=2147483647
Metadata Service	system/services/metadata.service/stacks?depth=2147483647
Metrics	system/services/metrics/stacks?depth=2147483647
Metrics Processor	system/services/metrics.processor/stacks?depth=2147483647