As applications process data, CDAP collects metrics about the application’s behavior and performance. Some of these metrics are the same for every application, such as how many events are processed and how many data operations are performed. These metrics are called system or CDAP metrics.

Other metrics are user-defined or "custom" and differ from application to application. To add user-defined metrics to your application, read this section in conjunction with the details on available system metrics in the Metrics Microservices.

You embed user-defined metrics in the methods defining the components of your application. The metrics system currently supports two kinds of metrics: count and gauge:

• count: Increments (or decrements) the metric named metricName by delta:

  count(metricName, delta)

• gauge: Sets the metric named metricName to a specific value:

  gauge(metricName, value)

They will then emit their metrics and you can retrieve them (along with system metrics) via the CDAP’s CDAP Microservices v3. The names given to the metrics (such as names.longnames and names.bytes as in the example below) should be composed only of alphanumeric characters.

Under high load, metrics system may overload HBase. To mitigate the effect, Metrics system also provides an option to enable/disable metrics using app.program.metrics.enabled parameter which can have values as true or false. By default, program metrics are enabled, so the default value for this parameter is true. This option can be configured at 3 different levels:

• System wide In cdap-site.xml (system wide) to control whether app containers emit metrics

• Per Run Using the config system.metrics.enabled as program argument to override the system-wide setting per program run.

• Program Preference Using the system.metrics.enabled permanently for a program by setting a preference.

• Scoping Additionally for program types that have sub-components (such as MapReduce and Spark programs), a prefix can be added to system.metrics.enabled to limit the scope of the arguments. 

Flow Control

A flow control mechanism in CDAP prevents you from submitting too many requests, which can cause stuck or failed pipeline runs. This mechanism applies to start (Launch) requests for the following pipelines and jobs in all editions of CDAP, versions 6.6 and later:

• Batch pipelines

• Real-time pipelines

• Replication jobs

Thresholds

To control the number of outstanding start requests in CDAP, flow control relies on two thresholds:

• Launch threshold. Limits the number of launch requests being made. If the number of launch requests received by Cloud Data Fusion that are not yet in Running state is greater than the launch threshold (number of launch requests that aren't running yet > launch threshold), the request is rejected. The number includes batch pipelines, real-time pipelines, and Replication jobs.

• Runs threshold. Limits the number of outstanding launch requests in the system. If the number of outstanding launch requests received by Cloud Data Fusion that have not yet finished is greater than a specified runs threshold (number of outstanding requests that aren't finished yet > runs threshold), the request is rejected.

Rejected requests

If either threshold is met, every new start request gets rejected. For example, a rejected batch pipeline has a Rejected status in the logs, and a dialog in the UI describes why the launch was rejected.

Metrics

The number of launching and running pipelines along with the number of rejected pipelines in the system can be accessed through the metrics service as follows.

Launching metric

Reports the number of launching requests in the system. That is, it reports the number of launch requests that are in Provisioning or Starting states.

Metrics name: flowcontrol.launching.count

POST /v3/metrics/query?metric=system.flowcontrol.launching.count

Running metric

Reports the number of running pipelines and jobs in the system. The run threshold is based on the sum of the Launching and Running metrics values.

Metrics name: flowcontrol.running.count

POST /v3/metrics/query?metric=system.flowcontrol.running.count

Rejected metric

Reports the number of rejected runs due to the flow control mechanism.

Metrics name: program.rejected.runs

POST /v3/metrics/query?metric=system.program.rejected.runs