Data Pipeline Plugins

Owned by Robin Rielley
Last updated: Feb 18, 2021

CDAP has the following types of plugins:

  • System plugins

  • User plugins

System plugins are installed by default with both CDAP Sandbox and Distributed CDAP. System plugins are available to all namespaces in a CDAP instance.

User plugins are available for download in the Hub. User plugins are available in the namespace where the user downloads them from the Hub. For example, if you have a namespace called test and download the Amazon SQS Plugins from the Hub, the Amazon SQS Sink will be available in the test namespace, but not in any other namespaces.

Types of Plugins

The following types of plugins are available in CDAP, both in the CDAP Sandbox and Distributed CDAP:

  • Batch Source Plugins

  • Realtime Source Plugins

  • Transform Plugins

  • Analytics Plugins

  • Sink Plugins

  • Conditions and Actions

  • Error Handlers and Alerts

  • Pipeline Alerts (also called Post-run Action plugins)

Plugin Notes

  • Plugins are grouped here based on how they are displayed in the Pipeline Studio UI. Each plugin is of a particular type.

  • The batch sources can write to any batch sinks that are available and real-time sources can write to any real-time sinks. Transformations work with either sinks or sources. Transformations can use validators to test data and check that it follows user-specified rules.

    Other plugin types may be restricted as to which plugin (and artifact) that they work with, depending on the particular functionality they provide.

  • Certain plugins, such as the JavaScript transform, are designed to be customized by end-users with their own code, either from within Pipeline Studio or in a configuration file.

  • Action plugins (supported only in pipelines based on the cdap-data-pipeline artifact) can be added to run either before a source or after a sink. A "post-run" action plugin can be specified that runs after the entire pipeline has run.

  • Developers can create and add their own plugins and plugin types.

Exploring Plugin Details

Details on the available plugins and the required properties for sources, analytics, transformations (transforms), sinks, and other plugin types can be obtained and explored using:

  • Pipeline Studio

  • CDAP CLI

  • Artifact Microservices

  • Individual plugin documentation for conditions and actions, sources, analytics, transforms, sinks, error handlers and alerts, and post-run plugins

Creating Custom Plugins

If these plugins don't meet your requirements, you can create a custom plugin.

If you are creating a custom plugin that extends the existing system artifacts, its name should not collide with existing names, for ease-of-use in the CDAP UI and Pipeline Studio. You can create your own plugin and plugin-type, depending on the functionality you are adding or requiring.

Created in 2020 by Google Inc.