Versions Compared

Old Version 18

changes.mady.by.user Robin Rielley

Saved on

compared with

New Version Current

changes.mady.by.user Robin Rielley

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

These plugins (from CDAP Pipelines Version 2.5.0-SNAPSHOT) are shipped with CDAP, both in the CDAP Sandbox and Distributed CDAP:

  • Batch Source

  • Realtime Source

  • Transform Transformation

  • Sink

  • Analytics

  • Conditions and Actions

  • Error Handlers and Alerts

  • Post Run Actions

Some plugins are installed by default and others are available in the Hub. Plugins installed by default are system plugins and available to all namespaces. Plugins deployed from the Hub are user plugins and are available in the namespace where they were deployed.

Plugin notes

  • Plugins are grouped here based on how they are displayed in the Pipeline Studio. Each plugin is of a particular type, and all plugins that are not sources, sinks, or actions are grouped in transformtransformation plugins. Each page describing a plugin shows its type and version at the bottom of the page.

  • The batch sources can write to any batch sinks that are available and real-time sources can write to any real-time sinks. Transformations work with either sinks or sources. Transformations can use validators to test data and check that it follows user-specified rules.

    Other plugin types may be restricted as to which plugin (and artifact) that they work with, depending on the particular functionality they provide.

  • Certain plugins—such as the JavaScript,  and Python Evaluator, and Validator transforms—are transformations—are designed to be customized by end-users with their own code, either from within Pipeline Studio or in a configuration file. For instance, you can create your own data validators either by using the functions supplied in the CoreValidator plugin or by implementing and supplying your own custom validation function.

  • Action plugins (supported only in batch data pipelines based on the cdap-data-pipeline artifact) can be added to run either before a source or after a sink. A "post-run" action plugin can be specified that runs after the entire pipeline has run.

  • Additional types of plugins are under development, and developers can create and add their own plugins and plugin types.

Exploring plugin details

Details on the available plugins and the required properties for sources, analytics, transformations (transforms), sinks, and other plugin types can be obtained and explored using:

  • Pipeline Studio

  • CDAP CLI

  • Artifact Microservices

  • Individual plugin documentation for actions, sources, analytics, transforms transformations, sinks, shared and post-run plugins

Creating custom plugins

If these plugins don't meet your requirements, you can create a custom plugin.

...