Lifecycle Microservices
- 1 Application Lifecycle
- 1.1 Create an Application
- 1.2 Update an Application (DEPRECATED)
- 1.3 Deploy an Artifact and Application
- 1.4 List Applications
- 1.5 Details of an Application
- 1.6 Upgrade an Application
- 1.7 Upgrade a List of Applications
- 1.8 List Versions of an Application
- 1.9 Delete an Application
- 1.10 Delete All Applications
- 1.11 Export All Application Details
- 1.12 Delete a Streaming Application State (6.9.1+)
- 2 Program Lifecycle
- 2.1 Details of a Program
- 2.2 MapReduce Jobs Associated with a Namespace (Deprecated)
- 2.3 Spark Jobs Associated with a Namespace
- 2.4 Workflows Associated with a Namespace
- 2.5 Services Associated with a Namespace
- 2.6 Workers Associated with a Namespace
- 2.7 Spark program status for an application
- 2.8 Start a Program
- 2.9 Start Multiple Programs
- 2.10 Stop a Program
- 2.11 Stop a Program Run
- 2.12 Stop Multiple Programs
- 2.13 Status of a Program
- 2.14 Status of Multiple Programs
- 3 Schedule Lifecycle
- 4 Container Information
- 5 Scaling
- 5.1 Scaling Services
- 5.2 Scaling Workers
- 6 Run Records
Use the CDAP Lifecycle Microservices to deploy or delete applications and manage the lifecycle of MapReduce (DEPRECATED) and Spark programs, custom services, workers, and workflows.
For more information about CDAP components, see CDAP Components.
All methods or endpoints described in this API have a base URL (typically http://<host>:11015
or https://<host>:10443
) that precedes the resource identifier, as described in the Microservices Conventions. These methods return a status code, as listed in the Microservices Status Codes.
Application Lifecycle
Create an Application
To create an application, submit an HTTP PUT request:
PUT /v3/namespaces/<namespace-id>/apps/<app-id>
(DEPRECATED) To create an application with a non-default version, submit an HTTP POST request with the version specified:
POST /v3/namespaces/<namespace-id>/apps/<app-id>/versions/<version-id>/create
Parameter | Description |
---|---|
| Namespace ID |
| Name of the application |
| (DEPRECATED) Version of the application, typically following semantic versioning; The |
The request body is a JSON object specifying the artifact to use to create the application, and an optional application configuration. For example:
PUT /v3/namespaces/default/apps/purchaseWordCount
{
"artifact": {
"name": "WordCount",
"version": "6.8.0",
"scope": "USER"
},
"config": {
“datasetName”: “purchases”
},
"principal":"user/example.net@EXAMPLEKDC.NET",
"app.deploy.update.schedules":"true"
}
will create an application named purchaseWordCount
from the example WordCount
artifact. The application will receive the specified config
, which will configure the application to create a dataset named purchases
instead of using the default dataset name.
Optionally, you can specify a Kerberos principal with which the application should be deployed. If a Kerberos principal is specified, then all the datasets created by the application will be created with the application's Kerberos principal.
Optionally, you can set or reset the flag app.deploy.update.schedules
. If true, redeploying an application will modify any schedules that currently exist for the application; if false, redeploying an application does not create any new schedules and existing schedules are neither deleted nor updated.
Update an Application (DEPRECATED)
To update an application, submit an HTTP POST request:
Parameter | Description |
---|---|
| Namespace ID |
| Name of the application |
The request body is a JSON object specifying the updated artifact version and the updated application config. For example, a request body of:
will update the purchaseWordCount
application to use version 6.3.0 of the WordCount
artifact, and update the name of the dataset to logs
. If no artifact is given, the current artifact will be used.
Only changes to artifact version are supported; changes to the artifact name are not allowed. If no config
is given, the current config
will be used. If the config
key is present, the current config
will be overwritten by the config
specified in the request. As the principal of an application cannot be updated, during an update the principal should either be the same or absent.
Deploy an Artifact and Application
To deploy an application from your local file system into the namespace namespace-id
, submit an HTTP POST request:
with the name of the JAR file as a header:
and Kerberos principal with which the application is to be deployed (if required):
and enable or disable updating schedules of the existing workflows using the header:
This will add the JAR file as an artifact and then create an application from that artifact. The archive name must be in the form <artifact-name>-<artifact-version>.jar
. An optional header can supply a configuration object as a serialized JSON string:
The application's content is the body of the request:
Invoke the same command to update an application to a newer version. However, be sure to stop all of its Spark and MapReduce programs before updating the application.
For an application that has a configuration class such as:
We can deploy it with this call:
Note: The X-App-Config
header contains the JSON serialization string of the MyAppConfig
object.
List Applications
To list all of the applications in the namespace namespace-id
, issue an HTTP GET request:
Parameter | Version Introduced | Description |
---|---|---|
|
| Namespace ID. |
|
| Optional filter to list all applications that use the specified artifact name. Valid values are |
|
| Optional filter. This is the version of the artifact given in the |
| 6.6.0 | Optional filter. The number of pipelines to be returned in the response. If |
| 6.6.0 | Optional filter. If |
| 6.6.0 | Optional filter. Specifies the sorting order. The sorting is by Application Name and then Application Version. Values can be |
| 6.8.0 | Optional filter. Filters the application name based on |
| 6.8.0 | Optional filter. Values can be
|
| 6.8.0 | Optional Boolean. Values can be Default is |
| 6.8.0 | Optional filter. Values can be |
Note: When upgrading the instance from versions < 6.8 to versions >= 6.8, it’s important to follow the “upgrade applications” process in order to have the UI properly render all existing applications.
If pageSize
is not specified, this returns an array of JSON Objects that lists each application with its name, description, and artifact. The list can optionally be filtered by one or more artifact names. It can also be filtered by artifact version. For example:
will return all applications that use either the cdap-data-pipeline
,cdap-data-streams
,or delta-app
artifacts.
The following is an example response when pageSize
is not specified:
If pageSize
is specified, the result is a JSON object that returns the applications as a JSON array under the applications
key. The page token identifier for the next page of results is specified under nextPageToken
key. The absence of nextPageToken
in the response denotes that it was the last page in the results.
Details of an Application
For detailed information on an application in a namespace namespace-id
, use:
Parameter | Description |
---|---|
| Namespace ID. |
| Name of the application. |
Note: To get the creation time of an application and other types of metadata, see Metadata Microservices.
The information will be returned in the body of the response. It includes the name and description of the application; the artifact and datasets that it uses, all of its programs; and the Kerberos principal, if that was provided during the deployment. For example:
HTTP Responses
Status Codes | Description |
---|---|
| The event successfully called the method, and the body contains the results |
Upgrade an Application
To upgrade an application in a namespace to use the latest version of application artifacts and plugin artifacts, run the following POST request:
Parameter | Description |
---|---|
| Namespace ID. |
| Name of the application. |
| Optional scope filter. If not specified, artifacts in the |
| Optional filter to allow SNAPSHOT version of artifacts for upgrade. Set to TRUE to allow SNAPSHOT version of artifacts for upgrade. Set to FALSE to ignore SNAPSHOT version of artifacts for upgrade. Default is false. |
The response will contain a list of application details containing name, application version, namespace, and entity. For example:
Upgrade a List of Applications
To upgrade a list of existing applications in a namespace to use the latest version of application artifacts and plugin artifacts, run the following POST request:
Parameter | Description |
---|---|
| Namespace ID. |
| Optional filter to allow artifacts of scope USER and SYSTEM for upgrade. Leave blank to allow artifacts of scope USER and SYSTEM for upgrade. Optional scope filter. If not specified, artifacts in the |
| Optional filter to allow SNAPSHOT version of artifacts for upgrade. Set to TRUE to allow SNAPSHOT version of artifacts for upgrade. Set to FALSE to ignore SNAPSHOT version of artifacts for upgrade. Default is false. |
The request body is a JSON object specifying the updated artifact version and the updated application config.
For example, the following request body will upgrade the listed applications in the default namespace to use the latest version of application artifacts and plugin artifacts.
List Versions of an Application
To list all the versions of an application, submit an HTTP GET:
Parameter | Description |
---|---|
| Namespace ID |
| Name of the application being called |
The response will be a JSON array containing details about the application. The details returned depend on the application.
For example, depending on the versions deployed:
could return in a JSON array a list of the versions of the application:
Delete an Application
To delete an application, together with all of its MapReduce or Spark programs, schedules, custom services, and workflows, submit an HTTP DELETE:
(DEPRECATED) To delete a specific version of an application, submit an HTTP DELETE that includes the version:
Parameter | Description |
---|---|
| Namespace ID |
| Name of the application to be deleted |
| (DEPRECATED) Version of the application to be deleted |
This does not delete the datasets associated with the application because they belong to the namespace, not the application. Also, this does not delete the artifact used to create the application.
Delete All Applications
To delete all the applications in a namespace, use:
Export All Application Details
If you’re running Windows, you can export all application details for all namespaces as a ZIP archive file, with the following request:
If you’re running Linux or Mac, you can use the curl command to get the output and write it to file using the command:
If you’re running Windows and have powershell, you can use this command:
These commands create a folder with the name of the zip file and write the contents to a file called outfile.zip in the directory you ran the command from. output.zip contains the JSON files for all of the applications in all namespaces in the CDAP instance.
Delete a Streaming Application State (6.9.1+)
To delete a streaming application state, submit an HTTP DELETE:
DELETE namespaces/<namespace-name>/apps/<app-name>/state
Parameter | Description |
| Namespace name. |
| Name of the application with the state to be deleted. |
You might use this endpoint after you upgrade a CDAP instance or stop a streaming pipeline to delete the state for the last processed record.
Note: This endpoint is supported for Kafka Consumer Streaming and Google Cloud Pub/Sub Streaming sources.
Program Lifecycle
Details of a Program
After an application is deployed, you can retrieve the details of its MapReduce and Spark programs, custom services, schedules, workers, and workflows by submitting an HTTP GET request:
To retrieve information about the schedules of the program's workflows, use:
Parameter | Description |
---|---|
| Namespace ID |
| Name of the application being called |
| One of |
| Name of the MapReduce, custom service, Spark, worker, or workflow being called |
| Name of the workflow being called, when retrieving schedules |
The response will be a JSON array containing details about the program. The details returned depend on the program type.
For example:
will return in a JSON array information about the UploadService of the application SportResults. The results will be similar to this (pretty-printed and portions deleted to fit):
MapReduce Jobs Associated with a Namespace (Deprecated)
To get a list of MapReduce jobs associated with a namespace, use:
Parameter | Description |
---|---|
| Namespace ID |
The response will be a JSON array containing details about the MapReduce program:
Parameter | Description |
---|---|
| One of |
| Name of the application being called |
| Name of the MapReduce, custom service, Spark, worker, or workflow being called |
| Description of the program |
Spark Jobs Associated with a Namespace
To get a list of Spark jobs associated with a namespace, use:
Parameter | Description |
---|---|
| Namespace ID |
The response will be a JSON array containing details about the Spark program:
Parameter | Description |
---|---|
| One of |
| Name of the application being called |
| Name of the MapReduce, custom service, Spark, worker, or workflow being called |
| Description of the program |
Workflows Associated with a Namespace
To get a list of workflows associated with a namespace, use:
Parameter | Description |
---|---|
| Namespace ID |
The response will be a JSON array containing details about the workflows:
Parameter | Description |
---|---|
| One of |
| Name of the application being called |
| Name of the MapReduce, custom service, Spark, worker, or workflow being called |
| Description of the program |
Services Associated with a Namespace
To get a list of services associated with a namespace, use:
Parameter | Description |
---|---|
| Namespace ID |
The response will be a JSON array containing details about the services:
Parameter | Description |
---|---|
| One of |
| Name of the application being called |
| Name of the MapReduce, custom service, Spark, worker, or workflow being called |
| Description of the program |
Workers Associated with a Namespace
To get a list of workers associated with a namespace, use:
The response will be a JSON array containing details about the workers:
Parameter | Description |
---|---|
| One of |
| Name of the application being called |
| Name of the MapReduce, custom service, Spark, worker, or workflow being called |
| Description of the program |
Spark program status for an application
To check if a Spark program is available for an application, use:
GET /v3/namespaces/<namespace-id>/apps/{app-name}/spark/{program-name}/available
Parameter | Description |
---|---|
| Namespace ID |
| Name of the application. |
| Name of the program. |
Start a Program
After an application is deployed, you can start its MapReduce and Spark programs, custom services, workers, or workflows by submitting an HTTP POST request:
You can start a program of a particular version of the application by submitting an HTTP POST request that includes the version:
When starting a program, you can optionally specify runtime arguments as a JSON map in the request body. CDAP will use these runtime arguments only for this single invocation of the program.
Parameter | Description |
---|---|
| Namespace ID |
| Name of the application being called |
| Version of the application being called |
| One of |
| Name of the MapReduce, custom service, Spark, worker, or workflow being called |
Service, Spark, and Worker programs do not allow concurrent program runs. Programs of these types cannot be started unless the program is in the STOPPED state. MapReduce and Workflow programs support concurrent runs. If you start one of these programs, a new run will be started even if other runs of the program have not finished yet.
For example:
will start the UploadService of the SportResults application with two runtime arguments.
Start Multiple Programs
You can start multiple programs from different applications and program types by submitting an HTTP POST request:
with a JSON array in the request body consisting of multiple JSON objects with these parameters:
Parameter | Description |
---|---|
| Name of the application being called |
| One of |
| Name of the MapReduce, custom service, Spark, worker, or workflow being started |
|