Plugin Management

This section covers how to manage the deployment of plugins on your system:

Plugin Deployment: deploying as either a system or user artifact
Deployment Verification: verifying that an artifact was deployed successfully
Deploying Third-party JARs: making JDBC drivers and other artifacts available to applications
Managing Multiple Version: different versions can co-exist and be available at the same time
Deleting Plugins: removing deployed artifacts from CDAP

If you are creating your own plugins, see the section on developing plugins for information on writing plugins, including packaging plugins in a JAR and their presentation in a UI such as the Pipeline Studio.

If you are installing a third-party JAR (such as a JDBC driver) to make it accessible to other plugins or applications, see “Deploying Third-Party JARs” below.

Available Plugins

Plugins available with CDAP are listed beginning on a separate reference page. User-installed plugins are not listed there, but if they are installed correctly, the reference documentation for the plugin will be available through the Pipeline Studio.

Deploying Plugins

To make plugins available to another artifact (and thus available to any application created from one of the artifacts), the plugins must first be packaged in a JAR file. After that, the JAR file must be deployed either as a system artifact or a user artifact.

A system artifact is available to users across any namespace. A user artifact is available only to users in the namespace to which it is deployed. By design, deploying as a user artifact just requires access to the Artifact Microservices, while deploying as a system artifact requires access to the filesystem of the CDAP Master. This then requires administrator access and permission.

Deploying as a System Artifact

To deploy the artifact as a system artifact, both the JAR file and a matching configuration file must be placed in the appropriate directory.

CDAP Sandbox: $CDAP_INSTALL_DIR/artifacts
Distributed CDAP: The plugin JARs should be placed in the local file system and the path can be provided to CDAP by setting the property app.artifact.dir in cdap-site.xml. Multiple directories can be defined by separating them with a semicolon. The default path is /opt/cdap/master/artifacts.

For each plugin JAR, there must also be a corresponding configuration file to specify which artifacts can use the plugins. The file name must match the name of the JAR, except it must have the .json extension instead of the .jar extension. For example, if your JAR file is named custom-transforms-1.0.0.jar, there must be a corresponding custom-transforms-1.0.0.json file. If your custom-transforms-1.0.0.jar contains transforms that can be used by both the cdap-data-pipeline and cdap-data-streams artifacts, custom-transforms-1.0.0.json would contain:

{
  "parents": [ "cdap-data-pipeline[6.2.0,6.2.0]", "cdap-data-streams[6.2.0,6.2.0]" ]
}

This file specifies that the plugins in custom-transforms-1.0.0.jar can be used by version 6.2.0 of the cdap-data-pipeline and cdap-data-streams artifacts. You can also specify a wider range of versions that can use the plugins, with square brackets [ ] indicating an inclusive version and parentheses ( ) indicating an exclusive version. For example:

{
  "parents": [ "cdap-data-pipeline[3.5.0,4.0.0)", "cdap-data-streams[3.5.0,4.0.0)" ]
}

specifies that these plugins can be used by versions 3.5.0 (inclusive) to 4.0.0 (exclusive) of the cdap-data-pipeline and cdap-data-streams artifacts.

If the artifact contains third-party plugins, you can explicitly list them in the config file. For example, you may want to deploy a JDBC driver contained in a third-party JAR. In these cases, you have no control over the code to annotate the classes that should be plugins, so you need to list them in the configuration:

{
  "parents": [ "cdap-data-pipeline[3.5.0,4.0.0)", "cdap-data-streams[3.5.0,4.0.0)" ],
  "plugins": [
    {
      "name": "mysql",
      "type": "jdbc",
      "className": "com.mysql.jdbc.Driver"
    }
  ]
}

Once your JARs and matching configuration files are in place, a CDAP CLI command (load artifact) or a Microservices call to load system artifacts can be made to load the artifacts. As described in the documentation on Artifacts, only snapshot artifacts can be re-deployed without requiring that they first be deleted.

Alternatively, the CDAP Sandbox should be restarted for this change to take effect in local sandbox mode, and cdap-master services should be restarted in the Distributed mode.

Deploying as a User Artifact

To deploy the artifact as a user artifact, use the Artifact Microservices Add Artifact or the CLI.

When using the Microservices, you will need to specify the Artifact-Extends header. Unless the artifact's version is defined in the manifest file of the JAR file you upload, you will also need to specify the Artifact-Version header.

When using the CLI, a configuration file exactly like the one described in the Deploying as a System Artifact must be used.

For example, to deploy custom-transforms-1.0.0.jar using the Microservices:

Linux

$ curl -w"\n" -X POST "localhost:11015/v3/namespaces/default/artifacts/custom-transforms" \ -H "Artifact-Extends: system:cdap-data-pipeline[6.2.0, 6.2.0]/system:cdap-data-streams[6.2.0, 6.2.0]" \ --data-binary @/path/to/custom-transforms-1.0.0.jar

Windows

> curl -X POST "localhost:11015/v3/namespaces/default/artifacts/custom-transforms" ^ -H "Artifact-Extends: system:cdap-data-pipeline[6.2.0, 6.2.0]/system:cdap-data-streams[6.2.0, 6.2.0]" ^ --data-binary @/path/to/custom-transforms-1.0.0.jar

Using the CLI

cdap > load artifact /path/to/custom-transforms-1.0.0.jar config-file /path/to/config.json

where config.json contains:

{
"parents": [ "system:cdap-data-pipeline[6.2.0,6.2.0]", "system:cdap-data-streams[6.2.0,6.2.0]" ]
}

Note that when deploying a user artifact that extends a system artifact, you must prefix the parent artifact name with 'system:'. This is in the event there is a user artifact with the same name as the system artifact. If you are extending a user artifact, no prefix is required.

You can deploy third-party JARs in the same way except the plugin information needs to be explicitly listed. As described in the documentation on Artifacts, only snapshot artifacts can be re-deployed without requiring that they first be deleted.

Using the Microservices (note that if the artifact version is not in the JAR manifest file, it needs to be set explicitly, as the JAR contents are uploaded without the filename):

Linux

$ curl -w"\n" -X POST "localhost:11015/v3/namespaces/default/artifacts/mysql-connector-java" \ -H 'Artifact-Plugins: [ { "name": "mysql", "type": "jdbc", "className": "com.mysql.jdbc.Driver" } ]' \ -H "Artifact-Version: 5.1.35" \ -H "Artifact-Extends: system:cdap-data-pipeline[6.2.0, 6.2.0]/system:cdap-data-streams[6.2.0, 6.2.0]" \ --data-binary @mysql-connector-java-5.1.35.jar

Windows

> curl -X POST "localhost:11015/v3/namespaces/default/artifacts/mysql-connector-java" ^ -H "Artifact-Plugins: [ { \"name\": \"mysql\", \"type\": \"jdbc\", \"className\": \"com.mysql.jdbc.Driver\" } ]" ^ -H "Artifact-Version: 5.1.35" ^ -H "Artifact-Extends: system:cdap-data-pipeline[6.2.0, 6.2.0]/system:cdap-data-streams[6.2.0, 6.2.0]" ^ --data-binary @mysql-connector-java-5.1.35.jar

Using the CLI (note that the artifact version, if not explicitly set, is derived from the JAR filename):

cdap > load artifact /path/to/mysql-connector-java-5.1.35.jar config-file /path/to/config.json

where config.json contains:

Deployment Verification

You can verify that a plugin artifact was added successfully by using the Artifact Microservices to retrieve artifact details. For example, to retrieve detail about our custom-transforms artifact:

Linux

$ curl -w"\n" -X GET "localhost:11015/v3/namespaces/default/artifacts/custom-transforms/versions/1.0.0?scope=[system | user]

Windows

> curl -X GET "localhost:11015/v3/namespaces/default/artifacts/custom-transforms/versions/1.0.0?scope=[system | user]

Using the CLI

cdap > describe artifact properties custom-transforms 1.0.0 [system | user]

If you deployed the custom-transforms artifact as a system artifact, the scope is system. If you deployed the custom-transforms artifact as a user artifact, the scope is user.

You can verify that the plugins in your newly-added artifact are available to its parent by using the Artifact Microservices to list plugins of a specific type. For example, to check if cdap-data-pipeline can access the plugins in the custom-transforms artifact:

Linux

$ curl -w"\n" -X GET "localhost:11015/v3/namespaces/default/artifacts/cdap-data-pipeline/versions/6.2.0/extensions/transform?scope=system"

Windows

> curl -X GET "localhost:11015/v3/namespaces/default/artifacts/cdap-data-pipeline/versions/6.2.0/extensions/transform?scope=system"

Using the CLI

cdap > list artifact plugins cdap-data-pipeline 6.2.0 transform system

You can then check the list returned to see if your transforms are in the list. Note that the scope here refers to the scope of the parent artifact. In this example it is the system scope because cdap-data-pipeline is a system artifact. This is true even if you deployed custom-transforms as a user artifact because the parent is still a system artifact.

Deploying Third-Party JARs

Prebuilt JARs: In a case where you'd like to use pre-built third-party JARs (such as a JDBC driver) as a plugin, you will need to create a JSON file to describe the JAR.

For information on the format of the JSON, please refer to the sections on Third-Party Plugins and Plugin Deployment in Plugins.

A sample JDBC Driver Plugin configuration:

Managing Multiple Versions

Different versions of the same plugin (or artifact) can be loaded and available at the same time. These will appear in the Pipeline Studio as possible choices when selecting a plugin or creating a plugin template. If no version is specified for a plugin in the configuration file used to create an application, the highest version currently available in the system will be used.

Deleting Plugins

Plugins can be deleted using either the Artifact Microservices or the CDAP CLI.

In the case of the CDAP CLI, only plugins that have been deployed in the user scope can be deleted by the CDAP CLI. Both system and user scope plugins can be deleted using the Microservices by using the appropriate calls.

Note that in all cases, the actual files (JARs and JSON files) associated with the plugins are not deleted. Instead, the references to them are deleted in the CDAP system. If the files are not removed after these references are deleted, then, in the case of the system scope plugins, the artifacts will be reloaded the next time CDAP is restarted, as they are automatically loaded at startup from the appropriate directory.