Hub API

The Hub APIs are simply a contract about the directory structure of the Hub. All APIs are relative to a base path. For example, the base path for the public CDAP-hosted Hub is hub.cdap.io.

The directory structure must be:

<base>/v2/packages.json
<base>/v2/packages/<package-name>/<version>/icon.png
<base>/v2/packages/<package-name>/<version>/spec.json
<base>/v2/packages/<package-name>/<version>/spec.json.asc
<base>/v2/packages/<package-name>/<version>/<resource1>
<base>/v2/packages/<package-name>/<version>/<resource1>.asc
<base>/v2/packages/<package-name>/<version>/<resource2>
<base>/v2/packages/<package-name>/<version>/<resource2>.asc
...

All methods or endpoints described in this API have a base URL (typically http://<host>:11015 or https://<host>:10443) that precedes the resource identifier, as described in the Microservices Conventions. These methods return a status code, as listed in the Microservices Status Codes.

Get Hub Catalog

To retrieve a list of available packages, submit an HTTP GET request:

GET /v2/packages.json

This will return a JSON array that lists each package and its metadata:

[
  {
    "name": "access-log",
    "version": "1.0.0",
    "description": "Sample access logs in Combined Log Format (CLF)",
    "label": "Access Log Sample",
    "author": "Cask",
    "org": "Cask Data, Inc.",
    "cdapVersion": "[6.2.0,6.3.0)",
    "created": 1473901763,
    "beta": false,
    "categories": [ "datapack" ]
  },
  {
    "name": "bulk-data-transfer",
    "version": "1.0.0",
    "description": "Moving data from structured data source such as a traditional relational
                  database into Hadoop is very common in building Data Lakes. This
                  data application allows you to set-up periodic full data dumps from
                  RDBMS into Hadoop cluster. Data on Hadoop is stored as DB table
                  snapshot. Supports other relational databases.",
    "label": "Bulk Data Transfer",
    "author": "Cask",
    "org": "Cask Data, Inc.",
    "cdapVersion": "[6.2.0,6.3.0)",
    "created": 1473901763,
    "beta": false,
    "license": "license.txt",
    "categories": [ "usecase" ]
  },
  ...
]

Get a Package Specification

To retrieve a package specification, submit an HTTP GET request:

This will return a JSON object that contains metadata about the package, and a list of actions required to install the package:

Action Specification

There are several supported actions, each with its own specification. If an action fails for any reason, actions completed before it are not rolled back. However, each action is idempotent, which means the installation can simply be retried once the underlying cause of the failure has been fixed.

Each action contains a label, type, and arguments:

The label is a short description that will be displayed to users during the install process. Some arguments will reference package resources.

These actions are available:

informational
create_driver_artifact
create_plugin_artifact
one_step_deploy_plugin
deploy_app
one_step_deploy_app
create_pipeline
create_pipeline_draft

Descriptions of each action type and their supported arguments follow.

informational

Displays information for the user. Does not perform any actions against the CDAP Microservices. This can be used, for example, to tell the user to download a JAR from a 3rd-party website.

Argument	Description	Required?	Default

Argument	Description	Required?	Default
`steps`	JSON array of strings listing steps the user should take	Yes

Example action:

create_driver_artifact

Creates a CDAP artifact containing a third-party JDBC Driver.

Argument	Description	Required?	Default

Argument	Description	Required?	Default
`name`	Artifact name	Yes
`jar`	JAR resource containing the artifact	Yes
`scope`	Artifact scope	No	`user`
`version`	Artifact version	No	Version contained in the JAR manifest
`config`	Package resource containing artifact parents, plugins, and properties	No

Example action:

where mysql-connector-java.json is a package resource with content such as:

create_plugin_artifact

Creates a CDAP artifact that contains plugins that extend another artifact. For example, it may contain plugins for CDAP pipelines. Prompts the user to upload the artifact JAR.

Argument	Description	Required?	Default

Argument	Description	Required?	Default
`name`	Artifact name	Yes
`scope`	Artifact scope	No	`user`
`version`	Artifact version	No	Version contained in the JAR manifest
`config`	Package resource containing artifact parents, plugins, and properties	No

Example action:

where solrsearch-plugins.json is a package resource with content such as:

one_step_deploy_plugin

Deploys a CDAP artifact that contains plugins that extend another artifact. For example, it may contain plugins for CDAP pipelines. It deploys the artifact without prompting users.

Argument	Description	Required?	Default

Argument	Description	Required?	Default
`name`	Artifact name	Yes
`jar`	JAR resource containing the artifact	Yes
`scope`	Artifact scope	No	`user`
`version`	Artifact version	No	Version contained in the JAR manifest
`config`	Package resource containing artifact parents, plugins, and properties	No

Example action:

where solrsearch-plugins.json is a package resource with content such as:

deploy_app

Deploys a CDAP application by prompting the user to upload the application JAR. Does not take any arguments.

Example action:

one_step_deploy_app

Deploys a CDAP application.

Argument	Description	Required?	Default

Argument	Description	Required?	Default
`jar`	JAR resource containing the application	Yes

Example action:

create_pipeline

Creates a CDAP pipeline.

Argument	Description	Required?	Default

Argument	Description	Required?
`artifact`	JSON Object containing the pipeline's artifact scope, name, and version	Yes
`name`	Pipeline name	Yes
`config`	Package resource containing the pipeline config	Yes

Example action:

where pipeline.json is a package resource containing the pipeline config.

create_pipeline_draft

Creates a CDAP pipeline draft. Similar to create_pipeline, except that the pipeline will not be published. Instead, a draft will be created that the user can then modify.

Argument	Description	Required?	Default

Argument	Description	Required?
`artifact`	JSON Object containing the pipeline's artifact scope, name, and version	Yes
`name`	Pipeline name	Yes
`config`	Package resource containing the pipeline config	Yes

Example action:

where pipeline.json is a package resource containing the pipeline config.

Get Package Specification Signature

To retrieve the signature for a package specification, submit an HTTP GET request:

The signature is a PGP signature that can be used to validate a package resource. The package publisher signs the package specification with their private key. The signature can then be used in conjunction with the publisher's public key to validate that the specification was signed by the publisher.

Get Package Resource

To retrieve a package resource, submit an HTTP GET request:

The resource can contain arbitrary data. They can be artifact JARs, configuration files, sample data, or anything else that a package action may require.

Get Package Resource Signature

To retrieve the signature for a package resource, submit an HTTP GET request:

The signature is a PGP signature that can be used to validate a package resource. The package publisher signs the package resource with their private key. The signature can then be used in conjunction with the publisher's public key to validate that a package resource was signed by the publisher.

Get Package Icon

To retrieve the icon for a package, submit an HTTP GET request: