Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 18 Current »

These plugins (from CDAP Pipelines Version 2.5.0-SNAPSHOT) are shipped with CDAP, both in the CDAP Sandbox and Distributed CDAP:

  • Source

  • Transform

  • Sink

  • Analytics

  • Conditions and Actions

  • Error Handlers and Alerts

  • Post Run Actions

Plugin notes

  • Plugins are grouped here based on how they are displayed in the Pipeline Studio. Each plugin is of a particular type, and all plugins that are not sources, sinks, or actions are grouped in transform plugins. Each page describing a plugin shows its type and version at the bottom of the page.

  • The batch sources can write to any batch sinks that are available and real-time sources can write to any real-time sinks. Transformations work with either sinks or sources. Transformations can use validators to test data and check that it follows user-specified rules.

    Other plugin types may be restricted as to which plugin (and artifact) that they work with, depending on the particular functionality they provide.

  • Certain plugins—such as the JavaScript, Python Evaluator, and Validator transforms—are designed to be customized by end-users with their own code, either from within Pipeline Studio or in a configuration file. For instance, you can create your own data validators either by using the functions supplied in the CoreValidator plugin or by implementing and supplying your own custom validation function.

  • Action plugins (supported only in pipelines based on the cdap-data-pipeline artifact) can be added to run either before a source or after a sink. A "post-run" action plugin can be specified that runs after the entire pipeline has run.

  • Additional types of plugins are under development, and developers can create and add their own plugins and plugin types.

Exploring plugin details

Details on the available plugins and the required properties for sources, analytics, transformations (transforms), sinks, and other plugin types can be obtained and explored using:

  • Pipeline Studio

  • CDAP CLI

  • Artifact Microservices

  • Individual plugin documentation for actions, sources, analytics, transforms, sinks, shared and post-run plugins

Creating custom plugins

If these plugins don't meet your requirements, you can create a custom plugin.

If you are creating a custom plugin that extends the existing system artifacts, its name should not collide with existing names, for ease-of-use in the CDAP UI and Pipeline Studio. You are free to create your own plugin and plugin-type, depending on the functionality you are adding or requiring.

  • No labels

Created in 2020 by Google Inc.