Data Pipeline Plugin Reference

Owned by Robin Rielley
Last updated: Mar 17, 2021
2 min read

These plugins are shipped with CDAP, both in the CDAP Sandbox and Distributed CDAP:

  • Batch Source

  • Realtime Source

  • Transformation

  • Sink

  • Analytics

  • Conditions and Actions

  • Error Handlers and Alerts

  • Post Run Actions

Some plugins are installed by default and others are available in the Hub. Plugins installed by default are system plugins and available to all namespaces. Plugins deployed from the Hub are user plugins and are available in the namespace where they were deployed.

Plugin notes

  • Plugins are grouped here based on how they are displayed in the Pipeline Studio. Each plugin is of a particular type, and all plugins that are not sources, sinks, or actions are grouped in transformation plugins. Each page describing a plugin shows its type and version at the bottom of the page.

  • The batch sources can write to any batch sinks that are available and real-time sources can write to any real-time sinks. Transformations work with either sinks or sources. Transformations can use validators to test data and check that it follows user-specified rules.

    Other plugin types may be restricted as to which plugin (and artifact) that they work with, depending on the particular functionality they provide.

  • Certain plugins—such as the JavaScript and Python Evaluator transformations—are designed to be customized by end-users with their own code, either from within Pipeline Studio or in a configuration file.

  • Action plugins (supported only in batch data pipelines based on the cdap-data-pipeline artifact) can be added to run either before a source or after a sink. A "post-run" action plugin can be specified that runs after the entire pipeline has run.

  • Additional types of plugins are under development, and developers can create and add their own plugins and plugin types.

Exploring plugin details

Details on the available plugins and the required properties for sources, analytics, transformations (transforms), sinks, and other plugin types can be obtained and explored using:

  • Pipeline Studio

  • CDAP CLI

  • Artifact Microservices

  • Individual plugin documentation for actions, sources, analytics, transformations, sinks, shared and post-run plugins

Creating custom plugins

If these plugins don't meet your requirements, you can create a custom plugin.

If you are creating a custom plugin that extends the existing system artifacts, its name should not collide with existing names, for ease-of-use in the CDAP UI and Pipeline Studio. You are free to create your own plugin and plugin-type, depending on the functionality you are adding or requiring.

Created in 2020 by Google Inc.