Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
POST /v3/namespaces/<namespace-id>/apps/<app-id>/versions/<version-id>/create

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application

version-id

(DEPRECATED) Version of the application, typically following semantic versioning;

The version-id defaults to -SNAPSHOT when you don’t specify a version-id.

The request body is a JSON object specifying the artifact to use to create the application, and an optional application configuration. For example:

...

Code Block
POST /v3/namespaces/<namespace-id>/apps/<app-id>/update

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application

The request body is a JSON object specifying the updated artifact version and the updated application config. For example, a request body of:

...

Code Block
GET /v3/namespaces/<namespace-id>/apps[?
[artifactName=<artifact-names>]
[&artifactVersion=<artifact-version>]
[&pageSize=<page-size>]
[&pageToken=<page-token>]
[&orderBy=<order-by>]
[&nameFilter=<name-filter>]
[&nameFilterType=<name-filter-type>]
[&sortCreationTime=<sort-creation-time>]
[&latestOnly=<latest-only>]]

Parameter

Version Introduced

Description

namespace-id

Namespace ID.

artifactName

Optional filter to list all applications that use the specified artifact name. Valid values are cdap-data-pipeline, cdap-data-streams, and delta-app.

artifactVersion

Optional filter. This is the version of the artifact given in the artifactName parameter. It will be different depending on the artifact given. To get the version list, use Artifact Microservices.

pageSize

6.6.0

Optional filter. The number of pipelines to be returned in the response. If pageSize is not specified, all pipelines will be returned. Basically, pageSize enables pagination of the results. Using this parameter changes the result format. For details, see the examples that follow.

pageToken

6.6.0

Optional filter. If pageSize is specified, use nextPageToken as the value from the response. If pageToken is not specified, returns the first page.

orderBy

6.6.0

Optional filter. Specifies the sorting order. The sorting is by Application Name and then Application Version.

Values can be ASCor DESC. If unspecified, the value will be ASC. ASC stands for ascending order whereas DESC stands for descending order.

nameFilter

6.8.0

Optional filter. Filters the application name based on nameFilterType. If nameFilterType is not set, it will use default CONTAINS type.

nameFilterType

6.8.0

Optional filter. Values can be EQUALS, CONTAINS or EQUALS_IGNORE_CASE. Specifies the way to filter application names based on nameFilter.

EQUALS: Application name must be equal to the nameFilter, case sensitive.

CONTAINS: Application name must contain the nameFilter, case insensitive.

EQUALS_IGNORE_CASE: Application name must be equal to the nameFilter, case insensitive.

sortCreationTime

6.8.0

Optional Boolean. Values can be true or false. If set to true, the sorting order will be based on the creation time of the applications, instead of by Application Name and then Application Version.

Default is false.

latestOnly

6.8.0

Optional filter. Values can be trueor false. If set to false, it returns all versions of the apps. If not specified, it will use a default true value, that only returns the latest version of the applications.

Note: When upgrading the instance from versions < 6.8 to versions >= 6.8, it’s important to follow the “upgrade applications” process in order to have the UI properly render all existing applications.

...

Code Block
GET /v3/namespaces/<namespace-id>/apps/<app-id>

Parameter

Description

namespace-id

Namespace ID.

app-id

Name of the application.

Note: To get the creation time of an application and other types of metadata, see Metadata Microservices.

...

Code Block
{
  "name": "POS_Sales_per_Region",
  "appVersion": "-SNAPSHOT",
  "description": "Data Pipeline Application",
  "change": {
    "author": "joe",
    "creationTimeMillis": 1668540944833,
    "latest": true
  },
  "configuration": "{\"resources\":{\"memoryMB\":2048.0,\"virtualCores\":1.0},\"driverResources\":{\"memoryMB\":2048.0,\"virtualCores\":1.0},\"connections\":[{\"from\":\"GCS - POS Sales\",\"to\":\"Wrangler\"},{\"from\":\"Wrangler\",\"to\":\"GCS2\"}],\"comments\":[],\"postActions\":[],\"properties\":{},\"processTimingEnabled\":true,\"stageLoggingEnabled\":false,\"stages\":[{\"name\":\"GCS - POS Sales\",\"plugin\":{\"name\":\"GCSFile\",\"type\":\"batchsource\",\"label\":\"GCS - POS Sales\",\"artifact\":{\"name\":\"google-cloud\",\"version\":\"0.15.3\",\"scope\":\"SYSTEM\"},\"properties\":{\"project\":\"auto-detect\",\"format\":\"text\",\"skipHeader\":\"false\",\"serviceFilePath\":\"auto-detect\",\"filenameOnly\":\"false\",\"recursive\":\"false\",\"encrypted\":\"false\",\"schema\":\"{\\\"type\\\":\\\"record\\\",\\\"name\\\":\\\"etlSchemaBody\\\",\\\"fields\\\":[{\\\"name\\\":\\\"offset\\\",\\\"type\\\":\\\"long\\\"},{\\\"name\\\":\\\"body\\\",\\\"type\\\":\\\"string\\\"}]}\",\"referenceName\":\"pos-sales\",\"path\":\"gs://flat-files-1/POS-r01.txt\"}},\"outputSchema\":\"{\\\"type\\\":\\\"record\\\",\\\"name\\\":\\\"etlSchemaBody\\\",\\\"fields\\\":[{\\\"name\\\":\\\"offset\\\",\\\"type\\\":\\\"long\\\"},{\\\"name\\\":\\\"body\\\",\\\"type\\\":\\\"string\\\"}]}\",\"id\":\"GCS---POS-Sales\"},{\"name\":\"Wrangler\",\"plugin\":{\"name\":\"Wrangler\",\"type\":\"transform\",\"label\":\"Wrangler\",\"artifact\":{\"name\":\"wrangler-transform\",\"version\":\"4.2.3\",\"scope\":\"SYSTEM\"},\"properties\":{\"field\":\"*\",\"precondition\":\"false\",\"threshold\":\"1\",\"schema\":\"{\\\"type\\\":\\\"record\\\",\\\"name\\\":\\\"etlSchemaBody\\\",\\\"fields\\\":[{\\\"name\\\":\\\"Store_Nbr\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Item_Nbr\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"WM_Week\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Daily\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Whse_Nbr\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Whse_Name\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"POS_Sales\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"POS_Qty\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"POS_Cost\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Net_Ship_Qty\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Sales_Type\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Sales_Description\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Max_Shelf_Qty\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Store_Specific_Retail\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Store_Specific_Cost\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Current_HO_Retail\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]}]}\",\"workspaceId\":\"af95f757-a2d8-4efb-90b0-fad0ff2a543b\",\"directives\":\"parse-as-csv :body \\u0027,\\u0027 true\\ndrop body\"}},\"outputSchema\":\"{\\\"type\\\":\\\"record\\\",\\\"name\\\":\\\"etlSchemaBody\\\",\\\"fields\\\":[{\\\"name\\\":\\\"Store_Nbr\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Item_Nbr\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"WM_Week\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Daily\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Whse_Nbr\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Whse_Name\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"POS_Sales\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"POS_Qty\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"POS_Cost\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Net_Ship_Qty\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Sales_Type\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Sales_Description\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Max_Shelf_Qty\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Store_Specific_Retail\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Store_Specific_Cost\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Current_HO_Retail\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]}]}\",\"inputSchema\":[{\"name\":\"GCS - POS Sales\",\"schema\":\"{\\\"type\\\":\\\"record\\\",\\\"name\\\":\\\"etlSchemaBody\\\",\\\"fields\\\":[{\\\"name\\\":\\\"offset\\\",\\\"type\\\":\\\"long\\\"},{\\\"name\\\":\\\"body\\\",\\\"type\\\":\\\"string\\\"}]}\"}],\"id\":\"Wrangler\"},{\"name\":\"GCS2\",\"plugin\":{\"name\":\"GCS\",\"type\":\"batchsink\",\"label\":\"GCS2\",\"artifact\":{\"name\":\"google-cloud\",\"version\":\"0.15.3\",\"scope\":\"SYSTEM\"},\"properties\":{\"project\":\"auto-detect\",\"suffix\":\"yyyy-MM-dd-HH-mm\",\"format\":\"csv\",\"serviceFilePath\":\"auto-detect\",\"location\":\"us\",\"referenceName\":\"pos-sales-per-region\",\"path\":\"gs://flat-files-1\"}},\"inputSchema\":[{\"name\":\"Wrangler\",\"schema\":\"{\\\"type\\\":\\\"record\\\",\\\"name\\\":\\\"etlSchemaBody\\\",\\\"fields\\\":[{\\\"name\\\":\\\"Store_Nbr\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Item_Nbr\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"WM_Week\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Daily\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Whse_Nbr\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Whse_Name\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"POS_Sales\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"POS_Qty\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"POS_Cost\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Net_Ship_Qty\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Sales_Type\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Sales_Description\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Max_Shelf_Qty\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Store_Specific_Retail\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Store_Specific_Cost\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]},{\\\"name\\\":\\\"Current_HO_Retail\\\",\\\"type\\\":[\\\"string\\\",\\\"null\\\"]}]}\"}],\"id\":\"GCS2\"}],\"schedule\":\"0 * * * *\",\"engine\":\"spark\",\"numOfRecordsPreview\":100.0,\"description\":\"Data Pipeline Application\",\"maxConcurrentRuns\":1.0}",
  "datasets": [],
  "programs": [
    {
      "type": "Spark",
      "app": "POS_Sales_per_Region",
      "name": "phase-1",
      "description": "Sources 'GCS - POS Sales' to sinks 'GCS2'."
    },
    {
      "type": "Workflow",
      "app": "POS_Sales_per_Region",
      "name": "DataPipelineWorkflow",
      "description": "Data Pipeline Workflow"
    }
  ],
  "plugins": [
    {
      "id": "GCS - POS Sales",
      "name": "GCSFile",
      "type": "batchsource"
    },
    {
      "id": "GCS2:csv",
      "name": "csv",
      "type": "validatingOutputFormat"
    },
    {
      "id": "GCS - POS Sales:text",
      "name": "text",
      "type": "validatingInputFormat"
    },
    {
      "id": "GCS2",
      "name": "GCS",
      "type": "batchsink"
    },
    {
      "id": "Wrangler",
      "name": "Wrangler",
      "type": "transform"
    }
  ],
  "artifact": {
    "name": "cdap-data-pipeline",
    "version": "6.8.0",
    "scope": "SYSTEM"
  }
}

HTTP Responses

Status Codes

Description

200 OK

The event successfully called the method, and the body contains the results

Upgrade an Application

Notes:

  • For all realtime pipelines, except pipelines created in CDAP 6.8.0 with a Kafka streaming source, upgrading realtime pipelines to use the latest version of application artifacts is not supported.

  • Back up all applications before performing the upgrade.

  • To get the name of the application you want to upgrade, use the GET request listed in “List Applications”.

  • If upgrading the instance from versions < 6.8 to versions >= 6.8, it’s important to set latestOnly=false for the above GET request.

...

Code Block
POST /v3/namespaces/<namespace-id>/apps/<app-id>/upgrade

Parameter

Description

namespace-id

Namespace ID.

app-id

Name of the application.

artifactScope

Optional scope filter. If not specified, artifacts in the USER and SYSTEM scopes are upgraded. Otherwise, only artifacts in the specified scope are upgraded.

allowSnapshot

Optional filter to allow SNAPSHOT version of artifacts for upgrade. Set to TRUE to allow SNAPSHOT version of artifacts for upgrade. Set to FALSE to ignore SNAPSHOT version of artifacts for upgrade.

Default is false.

The response will contain a list of application details containing name, application version, namespace, and entity. For example:

...

Code Block
POST /v3/namespaces/<namespace-id>/upgrade

Parameter

Description

namespace-id

Namespace ID.

artifactScope

Optional filter to allow artifacts of scope USER and SYSTEM for upgrade. Leave blank to allow artifacts of scope USER and SYSTEM for upgrade.

Optional scope filter. If not specified, artifacts in the USER and SYSTEM scopes are upgraded. Otherwise, only artifacts in the specified scope are upgraded.

allowSnapshot

Optional filter to allow SNAPSHOT version of artifacts for upgrade. Set to TRUE to allow SNAPSHOT version of artifacts for upgrade. Set to FALSE to ignore SNAPSHOT version of artifacts for upgrade.

Default is false.

The request body is a JSON object specifying the updated artifact version and the updated application config.

...

Code Block
GET /v3/namespaces/<namespace-id>/apps/<app-id>/versions

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application being called

The response will be a JSON array containing details about the application. The details returned depend on the application.

...

Code Block
DELETE /v3/namespaces/<namespace-id>/apps/<app-id>/versions/<version-id>

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application to be deleted

version-id

(DEPRECATED) Version of the application to be deleted

Note: The app-id in this URL is the name of the application as configured by the application specification, and not necessarily the same as the name of the JAR file that was used to deploy the application.

...

DELETE namespaces/<namespace-name>/apps/<app-name>/state

Parameter

Description

namespace-name

Namespace name.

app-name

Name of the application with the state to be deleted.

You might use this endpoint after you upgrade a CDAP instance or stop a streaming pipeline to delete the state for the last processed record.

...

Code Block
GET /v3/namespaces/<namespace-id>/apps/<app-id>/workflows/<workflow-id>/schedules

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application being called

program-type

One of mapreduceservicessparkworkers, or workflows

program-id

Name of the MapReducecustom serviceSparkworker, or workflow being called

workflow-id

Name of the workflow being called, when retrieving schedules

The response will be a JSON array containing details about the program. The details returned depend on the program type.

...

Code Block
GET /v3/namespaces/<namespace-id>/mapreduce

Parameter

Description

namespace-id

Namespace ID

The response will be a JSON array containing details about the MapReduce program:

Parameter

Description

program-type

One of mapreduceservicessparkworkers, or workflows

app-id

Name of the application being called

program-id

Name of the MapReducecustom serviceSparkworker, or workflow being called

description

Description of the program

Spark Jobs Associated with a Namespace

...

Code Block
GET /v3/namespaces/<namespace-id>/spark

Parameter

Description

namespace-id

Namespace ID

The response will be a JSON array containing details about the Spark program:

Parameter

Description

program-type

One of mapreduceservicessparkworkers, or workflows

app-id

Name of the application being called

program-id

Name of the MapReducecustom serviceSparkworker, or workflow being called

description

Description of the program

Workflows Associated with a Namespace

...

Code Block
GET /v3/namespaces/<namespace-id>/workflows

Parameter

Description

namespace-id

Namespace ID

The response will be a JSON array containing details about the workflows:

Parameter

Description

program-type

One of mapreduceservicessparkworkers, or workflows

app-id

Name of the application being called

program-id

Name of the MapReducecustom serviceSparkworker, or workflow being called

description

Description of the program

Services Associated with a Namespace

...

Code Block
GET /v3/namespaces/<namespace-id>/services

Parameter

Description

namespace-id

Namespace ID

The response will be a JSON array containing details about the services:

Parameter

Description

program-type

One of mapreduceservicessparkworkers, or workflows

app-id

Name of the application being called

program-id

Name of the MapReducecustom serviceSparkworker, or workflow being called

description

Description of the program

Workers Associated with a Namespace

...

The response will be a JSON array containing details about the workers:

Parameter

Description

program-type

One of mapreduceservicessparkworkers, or workflows

app-id

Name of the application being called

program-id

Name of the MapReducecustom serviceSparkworker, or workflow being called

description

Description of the program

Spark program status for an application

...

GET /v3/namespaces/<namespace-id>/apps/{app-name}/spark/{program-name}/available

Parameter

Description

namespace-id

Namespace ID

app-name

Name of the application.

program-name

Name of the program.

Start a Program

After an application is deployed, you can start its MapReduce and Spark programs, custom services, workers, or workflows by submitting an HTTP POST request:

...

When starting a program, you can optionally specify runtime arguments as a JSON map in the request body. CDAP will use these runtime arguments only for this single invocation of the program.

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application being called

version-id

Version of the application being called

program-type

One of mapreduceservicessparkworkers, or workflows

program-id

Name of the MapReducecustom serviceSparkworker, or workflow being called

ServiceSpark, and Worker programs do not allow concurrent program runs. Programs of these types cannot be started unless the program is in the STOPPED state. MapReduce and Workflow programs support concurrent runs. If you start one of these programs, a new run will be started even if other runs of the program have not finished yet.

...

with a JSON array in the request body consisting of multiple JSON objects with these parameters:

Parameter

Description

appId

Name of the application being called

programType

One of mapreduceservicesparkworker, or workflow

programId

Name of the MapReducecustom serviceSparkworker, or workflow being started

runtimeargs

Optional JSON object containing a string to string mapping of runtime arguments to start the program with

The response will be a JSON array containing a JSON object for each object in the input. Each JSON object will contain these parameters:

Parameter

Description

appId

Name of the application being called

programType

One of mapreduceservicesparkworker, or workflow

programId

Name of the MapReducecustom serviceSparkworker, or workflow being started

statusCode

The status code from starting an individual JSON object

error

If an error, a description of why the program could not be started (for example, the specified program was not found)

runId

A UUID that uniquely identifies a run with CDAP

For example:

Code Block
POST /v3/namespaces/default/start
  [
    {"appId": "App1", "programType": "Service", "programId": "Service1"},
    {"appId": "App1", "programType": "Spark", "programId": "Spark2"},
    {"appId": "App2", "programType": "Spark", "programId": "Spark1", "runtimeargs": { "arg1":"val1" }}
  ]'

...

Code Block
POST /v3/namespaces/<namespace-id>/apps/<app-id>/versions/<version-id>/<program-type>/<program-id>/stop

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application being called

version-id

Version of the application being called

program-type

One of mapreduceservicessparkworkers, or workflows

program-id

Name of the MapReducecustom serviceSparkworker, or workflow being stopped

A program that is in the STOPPED state cannot be stopped. If there are multiple runs of the program in the RUNNING state, this call will stop one of the runs, but not all of the runs.

...

Code Block
POST /v3/namespaces/<namespace-id>/apps/<app-id>/<program-type>/<program-id>/runs/<run-id>/stop

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application being called

program-type

One of mapreduceservicessparkworkers, or workflows

program-id

Name of the MapReducecustom serviceSparkworker, or workflow being called

run-id

Run id of the run being called

For example:

Code Block
POST /v3/namespaces/default/apps/PurchaseHistory/mapreduce/PurchaseHistoryBuilder/runs/631bc459-a9dd-4218-9ea0-d46fb1991f82/stop

...

with a JSON array in the request body consisting of multiple JSON objects with these parameters:

Parameter

Description

appId

Name of the application being called

programType

One of mapreduceservicesparkworker, or workflow

programId

Name of the MapReducecustom serviceSparkworker, or workflow being stopped

The response will be a JSON array containing a JSON object corresponding to each object in the input. Each JSON object will contain these parameters:

Parameter

Description

appId

Name of the application being called

programType

One of mapreduceservicesparkworker, or workflow

programId

Name of the MapReducecustom serviceSparkworker, or workflow being stopped

statusCode

The status code from stopping an individual JSON object

error

If an error, a description of why the program could not be stopped (for example, the specified program was not found)

For example:

Code Block
POST /v3/namespaces/default/stop
  [
    {"appId": "App1", "programType": "Service", "programId": "Service1"},
    {"appId": "App1", "programType": "Mapreduce", "programId": "MapReduce2"},
    {"appId": "App2", "programType": "Spark", "programId": "Spark2"}
  ]'

...

Code Block
GET /v3/namespaces/<namespace-id>/apps/<app-id>/<program-type>/<program-id>/status

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application being called

program-type

One of mapreduceschedulesservicessparkworkers, or workflows

program-id

Name of the MapReduceschedulecustom serviceSparkworker, or workflow being called

The response will be a JSON array with status of the program. For example, retrieving the status of the UploadService of the program SportResults:

...

with a JSON array in the request body consisting of multiple JSON objects with these parameters:

Parameter

Description

appId

Name of the application being called

programType

One of mapreducescheduleservicesparkworker, or workflow

programId

Name of the MapReduceschedulecustom serviceSparkworker, or workflow being called

The response will be the same JSON array as submitted with additional parameters for each of the underlying JSON objects:

Parameter

Description

status

Maps to the status of an individual JSON object's queried program if the query is valid and the program was found

statusCode

The status code from retrieving the status of an individual JSON object

error

If an error, a description of why the status was not retrieved (for example, the specified program was not found)

The status and error fields are mutually exclusive meaning if there is an error, then there will never be a status and vice versa.

...

Code Block
PUT /v3/namespaces/<namespace-id>/apps/<app-id>/versions/<version-id>/schedules/<schedule-id>

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application

schedule-id

Name of the schedule; it is unique to the application and, if specified, the application version

version-id

Version of the application, typically following semantic versioning

The request body is a JSON object specifying the details of the schedule to be created:

...

Note: For any schedule, the program must be for a workflow and the programType must be set to WORKLFLOW.

HTTP Responses

Status Codes

Description

409 Conflict

Schedule with the same name already exists

Update a Schedule

To update a schedule, submit an HTTP POST request:

...

Code Block
POST /v3/namespaces/<namespace-id>/apps/<app-id>/versions/<version-id>/schedules/<schedule-id>/update

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application

schedule-id

Name of the schedule; it is unique to the application and, if specified, the application version.

version-id

Version of the application, typically following semantic versioning.

The request body is a JSON object specifying the details of the schedule to be updated, and follows the same form as documented in Add a Schedule.

Only changes to the schedule configurations are supported; changes to the schedule name, or to the program associated with it are not allowed. If any properties are provided, they will overwrite all existing properties with what is provided. You must include all properties, even ones you are are not altering.

HTTP Responses

Status Codes

Description

400 Bad Request

If the new schedule type does not match the existing schedule type or there are other client errors

Retrieving a Schedule

To retrieve a schedule in an application, submit an HTTP GET request:

Code Block
GET /v3/namespaces/<namespace-id>/apps/<app-id>/schedules/<schedule-name>

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application

schedule-name

Name of the schedule

The response will contain the schedule in the same form described in this topic in “Add a Schedule”.

...

The response will contain a list of schedules in the same form as described in “Add a Schedule”.

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application

workflow-id

Name of the workflow

Next Scheduled Run Time

To list the next time that the workflow will be be scheduled by a time trigger, use the parameter nextruntime:

Code Block
GET /v3/namespaces/<namespace-id>/apps/<app-id>/workflows/<workflow-id>/nextruntime

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application

workflow-id

Name of the workflow

Example: Retrieving The Next Runtime

HTTP Method

GET /v3/namespaces/default/apps/PurchaseHistory/workflows/PurchaseHistoryWorkflow/nextruntime

HTTP Response

[{"id":"DEFAULT.WORKFLOW:developer:PurchaseHistory:PurchaseHistoryWorkflow:0:DailySchedule","time":1415102400000}]

Description

Retrieves the next runtime of the workflow PurchaseHistoryWorkflow of the application PurchaseHistory

Next Scheduled Run Time in Batch

...

Code Block
POST /v3/namespaces/<namespace-id>/nextruntime

Parameter

Description

namespace-id

Namespace ID

The request body must be a JSON array of objects with the following parameters:

Parameter

Description

appId

Name of the application being called

programType

Currently, only the Workflow type is supported.

programId

Name of the program being called

The response will be an array of JSON Objects, each of which will contain the three input parameters as well as two of three possible extra fields: "schedules" or “error” if an error occurs.

Parameter

Description

schedules

The next scheduled runtimes for the program defined by the individual JSON object's parameters

statusCode

The status code from retrieving the program runs

error

If an error, a description of why the status was not retrieved (for example, the specified program was not found, or the requested JSON object was missing a parameter)

Example

HTTP Method

POST /v3/namespaces/default/nextruntime

HTTP Body

[{"appId": "App1", "programType": "Service", "programId": "Service1"}, {"appId": "App1", "programType": "Workflow", "programId": "testWorkflow"}, {"appId": "App2", "programType": "Workflow", "programId": "DataPipelineWorkflow"}]

HTTP Response

[{"appId": "App1", "programType": "Service", "programId": "Service1", "statusCode": 200, "schedules": [...]}, {"appId": "App1", "programType": "Workflow", "programId": "testWorkflow", "statusCode": 404, "error": "Program 'testWorkflow' is not found"}, {"appId": "App2", "programType": "Workflow", "programId": "DataPipelineWorkflow", "statusCode": 200, "schedules": [...]]

Description

Attempt to retrieve the next scheduled run of the service Service1 in the application App1, the workflow testWorkflow in the application App1 and the workflow DataPipelineWorkflow in the application App2, all in the namespace default

Previous Run Time of All Schedules

...

Code Block
POST /v3/namespaces/<namespace-id>/previousruntime

Parameter

Description

namespace-id

Namespace ID

The request body must be a JSON array of objects with the following parameters:

Parameter

Description

appId

Name of the application being called

programType

Currently, only the Workflow type is supported.

programId

Name of the program being called

The response will be an array of JSON Objects, each of which will contain the three input parameters as well as two of three possible extra fields: "schedules" or “error” if an error occurs.

Parameter

Description

schedules

The previous scheduled runtimes for the program defined by the individual JSON object's parameters

statusCode

The status code from retrieving the program runs

error

If an error, a description of why the status was not retrieved (for example, the specified program was not found, or the requested JSON object was missing a parameter)

Example

HTTP Method

POST /v3/namespaces/default/previousruntime

HTTP Body

[{"appId": "App1", "programType": "Service", "programId": "Service1"}, {"appId": "App1", "programType": "Workflow", "programId": "testWorkflow"}, {"appId": "App2", "programType": "Workflow", "programId": "DataPipelineWorkflow"}]

HTTP Response

[{"appId": "App1", "programType": "Service", "programId": "Service1", "statusCode": 200, "schedules": [...]}, {"appId": "App1", "programType": "Workflow", "programId": "testWorkflow", "statusCode": 404, "error": "Program 'testWorkflow' is not found"}, {"appId": "App2", "programType": "Workflow", "programId": "DataPipelineWorkflow", "statusCode": 200, "schedules": [...]]

Description

Attempt to retrieve the previous scheduled run of the service Service1 in the application App1, the workflow testWorkflow in the application App1 and the workflow DataPipelineWorkflow in the application App2, all in the namespace default

Previous Run Time of a Schedule

...

Code Block
GET /v3/namespaces/<namespace-id>/apps/<app-id>/workflows/<workflow-id>/previousruntime

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application to be deleted

workflow-id

Name of the Workflow

Example

HTTP Method

GET /v3/namespaces/default/apps/PurchaseHistory/workflows/PurchaseHistoryWorkflow/previousruntime

HTTP Response

[{"id":"DEFAULT.WORKFLOW:developer:PurchaseHistory:PurchaseHistoryWorkflow:0:DailySchedule","time":1415102400000}]

Description

Retrieves the previous runtime of the workflow PurchaseHistoryWorkflow of the application PurchaseHistory

Delete a Schedule

To delete a schedule, submit an HTTP DELETE:

...

Code Block
DELETE /v3/namespaces/<namespace-id>/apps/<app-id>/versions/<version-id>/schedules/<schedule-id>

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application to be deleted

schedule-id

Name of the schedule; it is unique to the application and, if specified, the application version

version-id

Version of the application to be deleted

HTTP Responses

Status Codes

Description

404 Bad Request

If the schedule given by schedule-id was not found

Schedule: Disable and and Enable

...

Note: You can also use suspend and resume instead of disable and enable.

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application

schedule-id

Name of the schedule

Example: Disabling a Schedule

HTTP Method

POST /v3/namespaces/default/apps/PurchaseHistory/schedules/DailySchedule/disable

HTTP Response

OK if successfully set as disabled

Description

Disables the schedule DailySchedule of the application PurchaseHistory

Container Information

To find out the address of an program's container host and the container’s debug port, you can query CDAP for a service’s live info via an HTTP GET method:

Code Block
GET /v3/namespaces/<namespace-id>/apps/<app-id>/<program-type>/<program-id>/live-info

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application being called

program-type

One of services or workers

program-id

Name of the program (service or worker)

Example:

Code Block
GET /v3/namespaces/default/apps/WordCount/flows/WordCounter/live-info

...

Code Block
POST /v3/namespaces/<namespace-id>/instances

Parameter

Description

namespace-id

Namespace ID

with a JSON array in the request body consisting of multiple JSON objects with these parameters:

Parameter

Description

appId

Name of the application being called

programType

One of service or worker

programId

Name of the program (service or worker) being called

The response will be the same JSON array as submitted with additional parameters for each of the underlying JSON objects:

Parameter

Description

requested

Number of instances the user requested for the program defined by the individual JSON object's parameters

provisioned

Number of instances that are actually running for the program defined by the individual JSON object's parameters.

statusCode

The status code from retrieving the instance count of an individual JSON object

error

If an error, a description of why the status was not retrieved (for example, the specified program was not found, or the requested JSON object was missing a parameter)

Note: The requested and provisioned fields are mutually exclusive of the error field.

Example

HTTP Method

POST /v3/namespaces/default/instances

HTTP Body

[{"appId":"MyApp1","programType":"Worker","programId":"MyWorker1",}, {"appId":"MyApp3","programType":"Service","programId":"MySvc1,}]

HTTP Response

[{"appId":"MyApp1","programType":"Worker","programId":"MyWorker1", "provisioned":2,"requested":2,"statusCode":200}, {"appId":"MyApp3","programType":"Service","programId":"MySvc1,}]

Description

Attempt to retrieve the instances of programType Worker in the application MyApp1, and the service handler MyHandler1 in the user service MySvc1 in the application MyApp3, all in the namespace default

Scaling Services

You can query or change the number of instances of a service by using the instances parameter with HTTP GET or PUT methods:

...

Code Block
{ "instances" : <quantity> }

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application

service-id

Name of the service

quantity

Number of instances to be used

Note: You can scale system services using the Monitor HTTP RESTful API Scaling System Services.

...

Code Block
{ "instances" : <quantity> }

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application

worker-id

Name of the worker

quantity

Number of instances to be used

Example

Retrieve the number of instances of the worker DataWorker in the application DemoApp in the namespace default:

...

Code Block
GET /v3/namespaces/<namespace-id>/apps/<app-id>/<program-type>/<program-id>/runs

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application

program-type

One of mapreduceservicesspark, or workflows

program-id

Name of the MapReducecustom serviceSpark, or workflow being called

You can filter the runs by the status of a program, the start and end times, and can limit the number of returned records:

Query Parameter

Description

status

PENDING, STARTING, RUNNING, STOPPING, COMPLETED, FAILED, KILLED, REJECTED

start

start timestamp

end

end timestamp

limit

maximum number of returned records

The response will be a JSON array containing a JSON object for each object in the input. Each JSON object will contain these parameters:

Parameter

Description

run-id

A UUID that uniquely identifies a run within CDAP, with the start and end times in seconds since the start of the Epoch (midnight 1/1/1970). Use that runid in subsequent calls to obtain additional information.

starting

The timestamp at which the program was requested to start by the user.

start

The timestamp at which the program actually started.

suspend

The timestamp at which this run was suspended (if it was suspended).

resume

The timestamp at which this run was resumed (if it was resumed after being suspended).

stopping

The timestamp at which the request to stop this run was made.

terminate

The timestamp after which the run will be forcefully killed if it does not stop gracefully.

status

The status of the run in question.

properties

A map of the properties of the run. Has subfields.

properties.runtimeArgs

The runtime arguments provided to the run serialized as a JSON string.

cluster

provides information about the cluster on which the run was executed. Has subfields.

cluster.status

The current status of the cluster.

cluster.numNodes

The number of nodes in the cluster.

profile

The compute profile used for the run.

profile.profileName

The name of the compute profile.

profile.namespace

The namespace of the compute profile.

profile.entity

The profile’s entity type.

Example: Retrieving Run Records

HTTP Method

GET /v3/namespaces/default/apps/SportResults/mapreduce/ScoreCounter/runs

HTTP Response

{"runid":"...","start":1382567598,"status":"RUNNING"},

{"runid":"...","start":1382567447,"end":1382567492,"status":"STOPPED"},

{"runid":"...","start":1382567383,"end":1382567397,"status":"STOPPED"}

Description

Retrieve the run records of the MapReduce ScoreCounter of the application SportResults.

Retrieving Specific Run Information

...

Code Block
GET /v3/namespaces/<namespace-id>/apps/<app-id>/<program-type>/<program-id>/runs/<run-id>

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application

program-type

One of mapreduceservicesspark, or workflows

program-id

Name of the MapReducecustom serviceSpark, or workflow being called

run-id

Run id of the run

The response will be a JSON array containing a JSON object for each object in the input. Each JSON object will contain these parameters:

Parameter

Description

run-id

A UUID that uniquely identifies a run within CDAP, with the start and end times in seconds since the start of the Epoch (midnight 1/1/1970). Use that runid in subsequent calls to obtain additional information.

starting

The timestamp at which the program was requested to start by the user.

start

The timestamp at which the program actually started.

suspend

The timestamp at which this run was suspended (if it was suspended).

resume

The timestamp at which this run was resumed (if it was resumed after being suspended).

stopping

The timestamp at which the request to stop this run was made.

terminate

The timestamp after which the run will be forcefully killed if it does not stop gracefully.

status

The status of the run in question.

properties

A map of the properties of the run. Has subfields.

properties.runtimeArgs

The runtime arguments provided to the run serialized as a JSON string.

cluster

provides information about the cluster on which the run was executed. Has subfields.

cluster.status

The current status of the cluster.

cluster.numNodes

The number of nodes in the cluster.

profile

The compute profile used for the run.

profile.profileName

The name of the compute profile.

profile.namespace

The namespace of the compute profile.

profile.entity

The profile’s entity type.

Example: Retrieving a Particular Run Record

HTTP Method

GET /v3/namespaces/default/apps/SportResults/mapreduce/ScoreCounter/runs/b78d0091-da42-11e4-878c-2217c18f435d

HTTP Response

{"runid":"...","start":1382567598,"status":"RUNNING"}

Description

Retrieve the run record of the MapReduce ScoreCounter of the application SportResults run b78d0091-da42-11e4-878c-2217c18f435d

For services, you can retrieve:

...

Example: Retrieving The Most Recent Run

HTTP Method

GET /v3/namespaces/default/apps/PurchaseHistory/services/CatalogLookup/runs?status=completed&limit=1

HTTP Response

[{"runid":"cad83d45-ecfb-4bf8-8cdb-4928a5601b0e","start":1415051892,"end":1415057103,"status":"STOPPED"}]

Description

Retrieve the most recent successful completed run of the service CatalogLookup of the application PurchaseHistory

Retrieving Run Records in Batch

...

Code Block
POST /v3/namespaces/<namespace-id>/runs

Parameter

Description

namespace-id

Namespace ID

The request body must be a JSON array of objects with the following parameters:

Parameter

Description

run-id

A UUID that uniquely identifies a run within CDAP, with the start and end times in seconds since the start of the Epoch (midnight 1/1/1970). Use that runid in subsequent calls to obtain additional information.

starting

The timestamp at which the program was requested to start by the user.

start

The timestamp at which the program actually started.

suspend

The timestamp at which this run was suspended (if it was suspended).

resume

The timestamp at which this run was resumed (if it was resumed after being suspended).

stopping

The timestamp at which the request to stop this run was made.

terminate

The timestamp after which the run will be forcefully killed if it does not stop gracefully.

status

The status of the run in question.

properties

A map of the properties of the run. Has subfields.

properties.runtimeArgs

The runtime arguments provided to the run serialized as a JSON string.

cluster

provides information about the cluster on which the run was executed. Has subfields.

cluster.status

The current status of the cluster.

cluster.numNodes

The number of nodes in the cluster.

profile

The compute profile used for the run.

profile.profileName

The name of the compute profile.

profile.namespace

The namespace of the compute profile.

profile.entity

The profile’s entity type.

The response will be an array of JSON Objects, each of which will contain the three input parameters as well as two of three possible extra fields: runs, which is a list of the latest run record for that program, statusCode, which maps to the status code for retrieving the runs for that program, and error if there was an error retrieving runs for that program. The "statusCode" property will always be included, but runs and error are mutually exclusive.

Parameter

Description

runs

The latest run records for the program defined by the individual JSON object's parameters

statusCode

The status code from retrieving the program runs

error

If an error, a description of why the status was not retrieved (for example, the specified program was not found, or the requested JSON object was missing a parameter)

Example

HTTP Method

POST /v3/namespaces/default/runs

HTTP Body

[{"appId": "App1", "programType": "Service", "programId": "Service1"}, {"appId": "App1", "programType": "Workflow", "programId": "testWorkflow"}, {"appId": "App2", "programType": "Workflow", "programId": "DataPipelineWorkflow"}]

HTTP Response

[{"appId": "App1", "programType": "Service", "programId": "Service1", "statusCode": 200, "runs": [...]}, {"appId": "App1", "programType": "Workflow", "programId": "testWorkflow", "statusCode": 404, "error": "Program 'testWorkflow' is not found"}, {"appId": "App2", "programType": "Workflow", "programId": "DataPipelineWorkflow", "statusCode": 200, "runs": [...]]

Description

Attempt to retrieve the latest run records of the service Service1 in the application App1, the workflow testWorkflow in the application App1 and the workflow DataPipelineWorkflow in the application App2, all in the namespace default

Retrieving Run Counts in Batch

...

Code Block
POST /v3/namespaces/<namespace-id>/runcount

Parameter

Description

namespace-id

Namespace ID

The request body must be a JSON array of objects with the following parameters:

Parameter

Description

appId

Name of the application being called

programType

One of mapreducesparkworkflowservice, or worker

programId

Name of the program (mapreducesparkworkflowservice, or worker) being called

The response will be an array of Json Objects, each of which will contain the three input parameters as well as two of three possible extra fields -- "runCount", which is count for the program run, "statusCode", which maps to the status code for retrieving the run count for that program, and "error" if there was an error retrieving runs for that program. The "statusCode" property will always be included, but "runCount" and "error" are mutually exclusive.

Parameter

Description

runCount

The number of program runs for the program defined by the individual JSON object's parameters over the entire lifetime

statusCode

The status code from retrieving the program run count

error

If an error, a description of why the status was not retrieved (for example, the specified program was not found, or the requested JSON object was missing a parameter)

Example

HTTP Method

POST /v3/namespaces/default/runcount

HTTP Body

[{"appId": "App1", "programType": "Service", "programId": "Service1"}, {"appId": "App1", "programType": "Workflow", "programId": "testWorkflow"}, {"appId": "App2", "programType": "Workflow", "programId": "DataPipelineWorkflow"}]

HTTP Response

[{"appId": "App1", "programType": "Service", "programId": "Service1", "statusCode": 200, "runCount": 20}, {"appId": "App1", "programType": "Workflow", "programId": "testWorkflow", "statusCode": 404, "error": "Program 'testWorkflow' is not found"}, {"appId": "App2", "programType": "Workflow", "programId": "DataPipelineWorkflow", "statusCode": 200, "runCount": 300}]

Description

Attempt to retrieve the run count of the service Service1 in the application App1, the workflow testWorkflow in the application App1 and the workflow DataPipelineWorkflow in the application App2, all in the namespace default

Retrieving Specific Run Count

...

Code Block
GET /v3/namespaces/<namespace-id>/apps/{app-id}/{program-type}/{program-id}/runcount

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application

program-type

One of mapreducesparkworkflowservice, or worker

program-id

Name of the program (mapreducesparkworkflowservice, or worker) being called

Example

HTTP Method

GET /v3/namespaces/default/apps/myApp/workflows/DataPipelineWorkflow/runcount

HTTP Response

[2]

Description

Retrieve the run count of the workflow DataPipelineWorkflow of the application myApp

Workflow Runs: Suspend and Resume

...

Code Block
POST /v3/namespaces/<namespace-id>/apps/<app-id>/workflows/<workflow-id>/runs/<run-id>/suspend
POST /v3/namespaces/<namespace-id>/apps/<app-id>/workflows/<workflow-id>/runs/<run-id>/resume

Parameter

Description

namespace-id

Namespace ID

app-id

Name of the application

workflow-id

Name of the workflow

run-id

UUID of the workflow run

Example: Suspending a Workflow

HTTP Method

POST /v3/namespaces/default/apps/PurchaseHistory/workflows/PurchaseHistoryWorkflow/runs/0ce13912-e980-11e4-a7d7-8cae4cfd0e64/suspend

HTTP Response

Program run suspended. if successfully set as suspended

Description

Suspends the run 0ce13912-e980-11e4-a7d7-8cae4cfd0e64 of the workflow PurchaseHistoryWorkflow of the application PurchaseHistory