Lifecycle Microservices

Use the CDAP Lifecycle Microservices to deploy or delete applications and manage the lifecycle of MapReduce (DEPRECATED) and Spark programs, custom services, workers, and workflows.

For more information about CDAP components, see CDAP Components.

All methods or endpoints described in this API have a base URL (typically http://<host>:11015 or https://<host>:10443) that precedes the resource identifier, as described in the Microservices Conventions. These methods return a status code, as listed in the Microservices Status Codes.

Application Lifecycle

Create an Application

To create an application, submit an HTTP PUT request:

(DEPRECATED) To create an application with a non-default version, submit an HTTP POST request with the version specified:

The request body is a JSON object specifying the artifact to use to create the application, and an optional application configuration. For example:

will create an application named purchaseWordCount from the example WordCount artifact. The application will receive the specified config, which will configure the application to create a dataset named purchases instead of using the default dataset name.

Optionally, you can specify a Kerberos principal with which the application should be deployed. If a Kerberos principal is specified, then all the datasets created by the application will be created with the application's Kerberos principal.

Optionally, you can set or reset the flag app.deploy.update.schedules. If true, redeploying an application will modify any schedules that currently exist for the application; if false, redeploying an application does not create any new schedules and existing schedules are neither deleted nor updated.

Update an Application (DEPRECATED)

To update an application, submit an HTTP POST request:

The request body is a JSON object specifying the updated artifact version and the updated application config. For example, a request body of:

will update the purchaseWordCount application to use version 6.3.0 of the WordCount artifact, and update the name of the dataset to logs. If no artifact is given, the current artifact will be used.

Only changes to artifact version are supported; changes to the artifact name are not allowed. If no config is given, the current config will be used. If the config key is present, the current config will be overwritten by the config specified in the request. As the principal of an application cannot be updated, during an update the principal should either be the same or absent.

Deploy an Artifact and Application

To deploy an application from your local file system into the namespace namespace-id, submit an HTTP POST request:

with the name of the JAR file as a header:

and Kerberos principal with which the application is to be deployed (if required):

and enable or disable updating schedules of the existing workflows using the header:

This will add the JAR file as an artifact and then create an application from that artifact. The archive name must be in the form <artifact-name>-<artifact-version>.jar. An optional header can supply a configuration object as a serialized JSON string:

The application's content is the body of the request:

Invoke the same command to update an application to a newer version. However, be sure to stop all of its Spark and MapReduce programs before updating the application.

For an application that has a configuration class such as:

We can deploy it with this call:

List Applications

To list all of the applications in the namespace namespace-id, issue an HTTP GET request:

If pageSize is not specified, this returns an array of JSON Objects that lists each application with its name, description, and artifact. The list can optionally be filtered by one or more artifact names. It can also be filtered by artifact version. For example:

will return all applications that use either the cdap-data-pipeline,cdap-data-streams,or delta-app artifacts.

The following is an example response when pageSize is not specified:

If pageSize is specified, the result is a JSON object that returns the applications as a JSON array under the applications key. The page token identifier for the next page of results is specified under nextPageToken key. The absence of nextPageToken in the response denotes that it was the last page in the results.

Details of an Application

For detailed information on an application in a namespace namespace-id, use:

The information will be returned in the body of the response. It includes the name and description of the application; the artifact and datasets that it uses, all of its programs; and the Kerberos principal, if that was provided during the deployment. For example:

HTTP Responses

Upgrade an Application

To upgrade an application in a namespace to use the latest version of application artifacts and plugin artifacts, run the following POST request:

The response will contain a list of application details containing name, application version, namespace, and entity. For example:

Upgrade a List of Applications

To upgrade a list of existing applications in a namespace to use the latest version of application artifacts and plugin artifacts, run the following POST request:

The request body is a JSON object specifying the updated artifact version and the updated application config.

For example, the following request body will upgrade the listed applications in the default namespace to use the latest version of application artifacts and plugin artifacts.

List Versions of an Application

To list all the versions of an application, submit an HTTP GET:

The response will be a JSON array containing details about the application. The details returned depend on the application.

For example, depending on the versions deployed:

could return in a JSON array a list of the versions of the application:

Delete an Application

To delete an application, together with all of its MapReduce or Spark programs, schedules, custom services, and workflows, submit an HTTP DELETE:

(DEPRECATED) To delete a specific version of an application, submit an HTTP DELETE that includes the version:

This does not delete the datasets associated with the application because they belong to the namespace, not the application. Also, this does not delete the artifact used to create the application.

Delete All Applications

To delete all the applications in a namespace, use:

Export All Application Details

If you’re running Windows, you can export all application details for all namespaces as a ZIP archive file, with the following request:

If you’re running Linux or Mac, you can use the curl command to get the output and write it to file using the command:

If you’re running Windows and have powershell, you can use this command:

These commands create a folder with the name of the zip file and write the contents to a file called outfile.zip in the directory you ran the command from. output.zip contains the JSON files for all of the applications in all namespaces in the CDAP instance.

Delete a Streaming Application State (6.9.1+)

To delete a streaming application state, submit an HTTP DELETE:

DELETE namespaces/<namespace-name>/apps/<app-name>/state

You might use this endpoint after you upgrade a CDAP instance or stop a streaming pipeline to delete the state for the last processed record.

Note: This endpoint is supported for Kafka Consumer Streaming and Google Cloud Pub/Sub Streaming sources.

Program Lifecycle

Details of a Program

After an application is deployed, you can retrieve the details of its MapReduce and Spark programs, custom services, schedules, workers, and workflows by submitting an HTTP GET request:

To retrieve information about the schedules of the program's workflows, use:

The response will be a JSON array containing details about the program. The details returned depend on the program type.

For example:

will return in a JSON array information about the UploadService of the application SportResults. The results will be similar to this (pretty-printed and portions deleted to fit):

MapReduce Jobs Associated with a Namespace (Deprecated)

To get a list of MapReduce jobs associated with a namespace, use:

The response will be a JSON array containing details about the MapReduce program:

Spark Jobs Associated with a Namespace

To get a list of Spark jobs associated with a namespace, use:

The response will be a JSON array containing details about the Spark program:

Workflows Associated with a Namespace

To get a list of workflows associated with a namespace, use:

The response will be a JSON array containing details about the workflows:

Services Associated with a Namespace

To get a list of services associated with a namespace, use:

The response will be a JSON array containing details about the services:

Workers Associated with a Namespace

To get a list of workers associated with a namespace, use:

The response will be a JSON array containing details about the workers:

Spark program status for an application

To check if a Spark program is available for an application, use:

GET /v3/namespaces/<namespace-id>/apps/{app-name}/spark/{program-name}/available

Start a Program

After an application is deployed, you can start its MapReduce and Spark programs, custom services, workers, or workflows by submitting an HTTP POST request:

You can start a program of a particular version of the application by submitting an HTTP POST request that includes the version:

When starting a program, you can optionally specify runtime arguments as a JSON map in the request body. CDAP will use these runtime arguments only for this single invocation of the program.

Service, Spark, and Worker programs do not allow concurrent program runs. Programs of these types cannot be started unless the program is in the STOPPED state. MapReduce and Workflow programs support concurrent runs. If you start one of these programs, a new run will be started even if other runs of the program have not finished yet.

For example:

will start the UploadService of the SportResults application with two runtime arguments.

Start Multiple Programs

You can start multiple programs from different applications and program types by submitting an HTTP POST request:

with a JSON array in the request body consisting of multiple JSON objects with these parameters:

The response will be a JSON array containing a JSON object for each object in the input. Each JSON object will contain these parameters:

For example:

will attempt to start the three programs listed in the request body. It will return a response such as:

In this particular example, the service and Spark programs in the App1 application were successfully started, and there was an error starting the last program because the App2 application does not exist.

Stop a Program

You can stop the MapReduce and Spark programs, custom services, workers, and workflows of an application by submitting an HTTP POST request:

You can stop the programs of a particular application version by submitting an HTTP POST request that includes the version:

A program that is in the STOPPED state cannot be stopped. If there are multiple runs of the program in the RUNNING state, this call will stop one of the runs, but not all of the runs.

For example:

will stop the UploadService service in the SportResults application.

Stop a Program Run

You can stop a specific run of a program by submitting an HTTP POST request:

For example:

will stop a specific run of the PurchaseHistoryBuilder MapReduce program in the PurchaseHistory application.

Stop Multiple Programs

You can stop multiple programs from different applications and program types by submitting an HTTP POST request:

with a JSON array in the request body consisting of multiple JSON objects with these parameters:

The response will be a JSON array containing a JSON object corresponding to each object in the input. Each JSON object will contain these parameters:

For example:

will attempt to stop the three programs listed in the request body. It will return a response such as:

In this particular example, the service and MapReduce programs in the App1 application were successfully stopped, and there was an error starting the last program because the App2 application does not exist.

Status of a Program

To retrieve the status of a program, submit an HTTP GET request:

The response will be a JSON array with status of the program. For example, retrieving the status of the UploadService of the program SportResults:

will return (pretty-printed) a response such as:

Status of Multiple Programs

You can retrieve the status of multiple programs from different applications and program types by submitting an HTTP POST request:

with a JSON array in the request body consisting of multiple JSON objects with these parameters:

The response will be the same JSON array as submitted with additional parameters for each of the underlying JSON objects:

The status and error fields are mutually exclusive meaning if there is an error, then there will never be a status and vice versa.

For example:

will retrieve the status of two programs. It will return a response such as:

Schedule Lifecycle

Schedules can only be created for workflows.

Add a Schedule

To add a schedule for a program to an application, submit an HTTP PUT request:

To add the schedule to an application with a non-default version, submit an HTTP PUT request with the version specified:

The request body is a JSON object specifying the details of the schedule to be created:

where a trigger is one of the trigger types. It can be a time trigger:

or a partition trigger:

or a program status trigger:

or an AND trigger, where "triggers" is a non-empty list of any type of triggers:

or an OR trigger, where "triggers" is a non-empty list of any type of triggers:

and a constraint can be one of:

HTTP Responses

Update a Schedule

To update a schedule, submit an HTTP POST request:

To update a schedule of an application with a non-default version, submit an HTTP POST request with the version specified:

The request body is a JSON object specifying the details of the schedule to be updated, and follows the same form as documented in Add a Schedule.

Only changes to the schedule configurations are supported; changes to the schedule name, or to the program associated with it are not allowed. If any properties are provided, they will overwrite all existing properties with what is provided. You must include all properties, even ones you are are not altering.

HTTP Responses

Retrieving a Schedule

To retrieve a schedule in an application, submit an HTTP GET request:

The response will contain the schedule in the same form described in this topic in “Add a Schedule”.

List Schedules

To list all of the schedules for an application, use an HTTP GET request:

As schedules are created for a workflow, you can also list schedules for a workflow of an application. You can use the Details of a Deployed Application to obtain the workflows of an application.

Optionally, you can filter the schedules by trigger type and schedule status using the query parameters trigger-type and schedule-status. For more information, see Schedules.

To list all of the schedules of a workflow of an application, use an HTTP GET request:

The response will contain a list of schedules in the same form as described in “Add a Schedule”.

Next Scheduled Run Time

To list the next time that the workflow will be be scheduled by a time trigger, use the parameter nextruntime:

Example: Retrieving The Next Runtime

Next Scheduled Run Time in Batch

To list the next time that all workflows in a namespace will be be scheduled by a time trigger, use the parameter nextruntime:

The request body must be a JSON array of objects with the following parameters:

The response will be an array of JSON Objects, each of which will contain the three input parameters as well as two of three possible extra fields: "schedules" or “error” if an error occurs.

Example

Previous Run Time of All Schedules

To list the previous scheduled run time for all programs that are passed into the data, use the parameter previousruntime:

The request body must be a JSON array of objects with the following parameters:

The response will be an array of JSON Objects, each of which will contain the three input parameters as well as two of three possible extra fields: "schedules" or “error” if an error occurs.

Example

Previous Run Time of a Schedule

To list the previous runtime when the scheduled program ran, use the parameter previousruntime:

Example

Delete a Schedule

To delete a schedule, submit an HTTP DELETE:

To delete a schedule of an application with a non-default version, submit an HTTP DELETE request with the version specified:

HTTP Responses

Schedule: Disable and and Enable

For a schedule, you can disable and enable it using the Microservices.

Disable: To disable a schedule means that the program associated with that schedule will not trigger again until the schedule is enabled.

Enable: To enable a schedule means that the trigger is reset, and the program associated will run again at the next scheduled time.

As a schedule is initially deployed in a disabled state, a call to this API is needed to enable it.

To disable or enable a schedule, use:

Example: Disabling a Schedule

Container Information

To find out the address of an program's container host and the container’s debug port, you can query CDAP for a service’s live info via an HTTP GET method:

Example:

The response is formatted in JSON; an example of this is shown in CDAP Testing and Debugging.

Scaling

You can retrieve the instance count executing different components from various applications and different program types using an HTTP POST method: