Pipeline Microservices

Use the CDAP Pipeline Microservices to manage data pipelines and connections.

All methods or endpoints described in this API have a base URL (typically http://<host>:11015 or https://<host>:10443) that precedes the resource identifier, as described in the Microservices Conventions. These methods return a status code, as listed in the Microservices Status Codes.

Create or Update a Draft Pipeline

To create or update a pipeline in Draft mode in a namespace, submit an HTTP PUT request:

PUT /v3/namespaces/system/apps/pipeline/services/studio/methods/v1/contexts/<context>/drafts/<draft>

Parameter	Description

Parameter	Description
`context`	The namespace for the Draft pipelines.
`draft`	The Draft ID of the pipeline. The Draft ID can be any alphanumeric string. It must be unique in the namespace.

The request body is a JSON object specifying the name, description, and artifact for the pipeline. There are two supported artifacts for connections: cdap-data-pipeline and cdap-data-streams.

For example:

{
    "name": "Draft1",
    "description": "This is an example draft",
    "artifact": {
      "name": "cdap-data-pipeline",
      "version": "6.3.0-SNAPSHOT",
      "scope": "SYSTEM"
    },
    "config": {<pipeline-config>}
  }

Delete a Draft Pipeline

To delete a pipeline in Draft mode in a namespace, submit an HTTP DELETE request:

DELETE /v3/namespaces/system/apps/pipeline/services/studio/methods/v1/contexts/<context>/drafts/<draft>

Parameter	Description

Parameter	Description
`context`	The namespace for the Draft pipeline.
`draft`	The Draft ID of the pipeline.

Details of a Draft Pipeline

To list the details of a pipeline in Draft mode in a namespace, submit an HTTP GET request:

Parameter	Description

Parameter	Description
`context`	The namespace for the Draft pipelines.
`draft`	The Draft ID of the pipeline.

The information will be returned in the body of the response. It includes the Draft ID, name, description, and revision number of each pipeline in Draft mode; the artifact that the pipelines uses; and details of the pipeline configuration (pipeline json). For example:

List Draft Pipelines

To list all of the pipeline in Draft mode in a namespace, submit an HTTP GET request:

Path Parameter	Description

Path Parameter	Description
`context`	The namespace for the Draft pipelines.

Query Parameter	Description

Query Parameter	Description
`sortBy`	Name of the fields to sort by. These field names must match the Draft pipeline json.
`filter`	Takes in any text and does prefix matching on the draft name field.
`includeConfig`	Includes the pipeline json for each draft in the response. Default value is false.

The information will be returned in the body of the response. It includes the Draft ID, create time, update time, name, description, and revision number of each pipeline in Draft mode; and the artifact that each pipelines uses. For example:

Create a Connection

To create a connection in a namespace, submit an HTTP PUT request:

Parameter	Description

Parameter	Description
`context`	The namespace for the connection.
`draft`	The Connection ID of the connection. The Connection ID can be any alphanumeric string. It must be unique in the namespace.

The request body is a JSON object specifying the name, description, and artifact for the connection. These are the supported artifacts for connections: google-cloud, database-plugin, mysql-plugin, oracle-plugin, postgresql-plugin, sqlserver-plugin.

For example:

List Connection

To list all of the connections in a namespace, submit an HTTP GET request:

Path Parameter	Description

Path Parameter	Description
`context`	The namespace for the connections.

The information will be returned in the body of the response. It includes the Connection ID, create time, update time, name, description, and plugin category, plugin type, and plugin properties for each in connection; and the artifact that each connection uses. For example:

Details of a Connection

To list the details of a connection in a namespace, submit an HTTP GET request:

Parameter	Description

Parameter	Description
`context`	The namespace for the connection.
`connection-id`	The Connection ID of the connection.

The information will be returned in the body of the response. It includes the Connection ID, create time, update time, name, description, and plugin category, plugin type, and plugin properties for each in connection; and the artifact that each connection uses. For example:

Delete a Connection

To delete a connection in a namespace, submit an HTTP DELETE request:

Parameter	Description

Parameter	Description
`context`	The namespace for the Connection.
`connection-id`	The Connection ID of the pipeline.

Test a Connection

To test a connection in a namespace, submit an HTTP POST request:

Parameter	Description

Parameter	Description
`context`	The namespace for the Connection.

The request body is a JSON object specifying the name, description, and artifact for the connection. These are the supported artifacts for connections: google-cloud, database-plugin, mysql-plugin, oracle-plugin, postgresql-plugin, sqlserver-plugin.

For example:

Browse a Connection

To browse a connection in a namespace, submit an HTTP POST request:

Parameter	Description

Parameter	Description
`context`	The namespace for the Connection.
`connection-id`	The Connection ID of the pipeline.

The request body is a JSON object specifying the name, description, and artifact for the connection. These are the supported artifacts for connections: google-cloud, database-plugin, mysql-plugin, oracle-plugin, postgresql-plugin, sqlserver-plugin.

For example:

If you are browsing a database that doesn’t have a schema, such as MySQL, enter the database name for the path. For example if the MySQL database name is mydb, enter:

For information about setting the path for each type of connection, see the Connection Reference.

Get Sample Results for a Connection

To get sample results for a connection in a namespace, submit an HTTP POST request:

Parameter	Description

Parameter	Description
`context`	The namespace for the Connection.
`connection-id`	The Connection ID of the pipeline.

The request body is a JSON object specifying the name, description, and artifact for the connection. These are the supported artifacts for connections: google-cloud, database-plugin, mysql-plugin, oracle-plugin, postgresql-plugin, sqlserver-plugin.

For example:

For information about setting the path for each type of connection, see the Connection Reference.

Get JSON Specification for a Connection

To get the JSON specification for a connection in a namespace, submit an HTTP POST request:

Parameter	Description

Parameter	Description
`context`	The namespace for the Connection.
`connection-id`	The Connection ID of the pipeline.

The request body is a JSON object specifying the name, description, and artifact for the connection. These are the supported artifacts for connections: google-cloud, database-plugin, mysql-plugin, oracle-plugin, postgresql-plugin, sqlserver-plugin.

For example:

For information about setting the path for each type of connection, see the Connection Reference.

Call GetSchema from REST API

This topic describes the process for getting the output schema for plugins using the REST API.

The GetSchema button sends a POST request to this endpoint:/v3/namespaces/system/apps/pipeline/services/studio/methods/v1/contexts/{namespace}/validations/stage

This endpoint expects a payload with the stage to validate and the inputSchemas to this stage. The stage object can be directly copied from the pipeline json and the inputSchemas can be left empty since it is a source. Example payload:

The response will be a fully populated stage object, one of the fields is "outputSchema" which will contain the schema of the input you are attempting to use. Example response: