Plugin Presentation

When a plugin is displayed in the CDAP UI, its properties are represented by widgets in the Pipeline Studio. Each property of a plugin is represented, by default, as a simple textbox in the user interface. By including an appropriate plugin widget JSON file, you can customize that presentation to use a different interface, as desired.

The available widgets that can be used are described below. Note that some of the plugins available for CDAP may include use of additional widgets that are not listed here. We recommend only using the list of plugins shown here, as these other widgets are internal to CDAP and may not be supported in a future release.

This document describes version 1.5 of the plugin specification. Changes to the specification are described in the Specification Changes and should be checked if you are using a version of the specification earlier than the current.

Plugin Widget JSON

To customize the plugin display, a plugin can include a widget JSON file that specifies the particular widgets and the sets of widget attributes used to display the plugin properties in the CDAP UI.

The widget JSON is composed of:

a map of metadata
a list of property configuration groups
a map of inputs
a list of outputs
a map of dataset jumps

Each configuration group consists of a list of the individual properties and the widgets to be used. For example:

{
  "metadata": {
    ...
  },
  "display-name": "Plugin Display Name",
  "icon": {
    "type": "builtin|link|inline",
    "arguments": {
       "url": "http://localhost/myicon.png",
       "data": "data:image/png;base64,..."
    }
  },
  "configuration-groups": [
    {"label": "Group 1",
      "properties": [
      ...
      ]
    },
    {"label": "Group 2",
      "properties": [
      ...
      ]
    },
    ...
  ],
  "inputs": {
    ...
  },
  "outputs": [
    {"output-property-1"},
    {"output-property-2"},
    ...
  ],
  "jump-config": {
    ...
  }
}

Metadata

Metadata refers to the top-level information about a plugin. The only information required is the a map consisting of the spec-version, the version of the specification which the JSON follows.

Current version: 1.5. For example:

{
  "metadata": {
    "spec-version" : "1.5"
  },
  ...
}

Display Name

The display-name field specifies the name of the plugin as it is displayed on the CDAP UI. A display name can be different from the plugin name, and is not guaranteed to be unique. A display name can contain whitespace. If not specified, the plugin name is used as the display name.

Icon

The icon field allows users to specify custom icons for their plugins. Users can specify icons for their plugins in three different ways, depending on the value of the type field contained inside the icon field:

builtin: The icon type builtin indicates that the icon for the plugin is defined in CDAP, and is not specified in the plugin's widget JSON file. This is the default icon type, if an icon is not specified. If such an icon is not found, the CDAP UI will use a default icon type.
link: The icon type link indicates that the icon for the plugin should be fetched from a URL specified in the arguments map. When the type of icon is link, the user is expected to provide a link to the icon image in a url attribute inside the arguments map.
inline: The icon type inline indicates that the icon for the plugin should be decoded from a base64 encoded image specified in the arguments map. When the type of icon is inline, the user is expected to provide a base64 encoded image in a data attribute inside the arguments map.

Icon Image Specification - The image used for an icon for a CDAP plugin should be in PNG format (http://www.libpng.org/pub/png/). The size of the image should be 50 X 50 pixels. The image background should be transparent.

Configuration Groups

Configuration groups are a simple grouping of properties of a plugin. A configuration group is represented as a JSON object with a label and an ordered list of plugin properties for that group.

For example, in a Batch Source plugin, properties such as name, basePath, duration, and delay could be grouped into a Batch Source Configuration.

Setting the hideByDefault property of a configuration group results in the group properties to be hidden by default.

In the case of a Batch Source plugin, it could look like this:

{
  "configuration-groups": [
    {
      "label": "Batch Source Configuration",
      "properties": [
        {
          "name": "name",
          ...
        },
        {
          "name": "basePath",
          ...
        },
        {
          "name": "duration",
          ...
        },
        {
          "name": "delay",
          ...
        }
      ]
    }
  ]
  ...
}

Once a group is established, you can configure how each of the individual properties inside the group is represented in the CDAP UI.

Property Configuration

Each individual property of the plugin is represented by a configuration, composed of:

name: Name of the field (as supplied by the CDAP server for the artifact).
label: Text string displayed in the CDAP UI beside the widget.
widget-type: The type of widget to be used to represent this property.
widget-attributes: A map of attributes that the widget type requires to be defined in order to render the property in the CDAP UI. The attributes required depend on the widget type used.
plugin-function: An optional map of plugin method and its widget attributes that can be applied to a particular plugin property.

Note: All properties and property values are case-sensitive.

To find the available field names, you can use the Artifact Microservices to retrieve plugin details for an artifact, which will include all the available field names. (If the artifact is your own, you will already know the available field names from your source code.)

In this example of a Batch Source plugin and its configuration-groups, four different properties are defined; three use a textbox widget, while one uses a dataset-selector. The groupByFields property includes a plugin function, used to fetch an output schema for the widget:

{
  "configuration-groups": [
    {
      "label": "Batch Source",
      "properties": [
        {
          "name": "name",
          "label": "Dataset",
          "widget-type": "dataset-selector"
        },
        {
          "name": "basePath",
          "label": "Base Path",
          "widget-type": "textbox"
        },
        {
          "name": "groupByFields",
          "label": "Group By Fields",
          "widget-type": "textbox",
          "plugin-function": {
            "method": "POST",
            "widget": "outputSchema",
            "plugin-method": "outputSchema",
            "required-fields": ["groupByFields", "aggregates"],
            "missing-required-fields-message":
              "'Group By Fields' & 'Aggregates' properties are required to fetch schema."
          }
        },
        {
          "name": "duration",
          "widget-type": "textbox"
        },
        ...
      ]
    }
  ]
}

Plugin Widgets

A widget in the CDAP UI represents a component that will be rendered and used to set the value of a property of a plugin. These are the different widgets, their type, their attributes (if any), their output data type, a description, sample JSON, that we support in CDAP pipelines as of version 6.2.0.

Widget Type	Widget Attributes	Output Data Type	Description	Example Widget JSON

Widget Type	Widget Attributes	Output Data Type	Description	Example Widget JSON
`csv`	`delimiter`: the delimiter between each set of values `value-placeholder`: placeholder for each row	Comma-separated `string`	Comma-separated values; each value is entered in a separate box	`{ "name": "property-csv", "widget-type": "csv", "widget-attributes": { "delimiter": ",", "value-placeholder": "enter a value" } }`
`daterange`	No attributes	`string`	Selector for a date and time range using graphical calendar	`{ "name": "daterange-selector", "widget-type": "daterange", "widget-attributes": {} }`
`datetime`	No attributes	`string`	Selector for a single date and time using graphical calendar	`{ "name": "datetime-selector", "widget-type": "datetime", "widget-attributes": {} }`
`dataset-selector`	No attributes	`string`	A type-ahead textbox with a list of datasets from the CDAP instance	`{ "name": "property-dataset-selector", "widget-type": "dataset-selector", "widget-attributes": {} }`
`ds-multiplevalues`	`delimiter`: the delimiter between each set of values `numValues`: number of values (number of delimiter-separated values) `placeholders`: array of placeholders for each value's textbox `values-delimiter`: the delimiter between each value	`string`	A delimiter-separated values widget that allows specifying lists of values separated by delimiters	`{ "name": "property-ds-multiplevalues", "widget-type": "ds-multiplevalues", "widget-attributes": { "delimiter": ",", "values-delimiter": ":", "numValues": "3", "placeholders": [ "Input Field", "Lookup", "Output Field" ] } }`
`dsv`	`delimiter`: delimiter used to separate the values	Delimiter-separated `string`	Delimiter-separated values; each value is entered in a separate box	`{ "name": "property-dsv", "widget-type": "dsv", "widget-attributes": { "delimiter": ":" } }`
`hidden`	`default`: default `string` value for the widget	`string`	This "hidden" widget allows values to be set for a property but hidden from users. A default can be supplied that will be used as the value for the property.	`{ "name": "property-hidden", "widget-type": "hidden", "widget-attributes": { "default": "defaultValue" } }`
`input-field-selector`	No attributes `multiselect`	`string`	A dropdown widget with a list of columns taken from the input schema. Selecting sets the input column for that plugin property.	`{ widget-type": "input-field-selector", "label": "Unique Fields", "name": "uniqueFields", "widget-attributes":{ "multiselect": "true" }`
`javascript-editor`	`default`: default `string` value for the widget	`string`	An editor to write JavaScript code as a value of a property	`{ "name": "property-javascript-editor", "widget-type": "javascript-editor", "widget-attributes": { "default": "function transform(input, emitter, context) {\ \\n emitter.emit(input);\\n}" } }`
`json-editor`	`default`: default serialized JSON value for the widget	`string`	A JSON editor that pretty-prints and auto-formats JSON while it is being entered	`{ "name": "property-json-editor", "widget-type": "json-editor", "widget-attributes": { "default": "{ \"p1\": \"value\" }" } }`
`keyvalue`	`delimiter`: delimiter for the key-value pairs `kv-delimiter`: delimiter between key and value `key-placeholder`: placeholder for the key input `value-placeholder`: placeholder for the value input	`string`	A key-value editor for constructing maps of key-value pairs	`{ "name": "property-keyvalue", "widget-type": "keyvalue", "widget-attributes": { "delimiter": ",", "kv-delimiter": ":", "key-placeholder": "Enter key", "value-placeholder": "Enter value" } }`
`multi-select`	`delimiter`: delimiter to concatenate multiple values to form a single string `options`: array of options where each option is defined by - `id`: value to be set for the plugin property - `label`: a label to be rendered in UI for that specific option `default`: a string containing default value(s) to use when rendering the widget for the first time. You must use the specified delimiter to separate values.	`string`	Multi-select dropdown to choose one or more from a list of available options.	`{ "name": "property-keyvalue", "widget-type": "multi-select", "widget-attributes": { "delimiter": ",", "default": "value1,value2", "options": [ { "id": "value1", "label": "Value One" }, { "id": "value2", "label": "Value two" } ] } }`
`radio-group`	`layout`: Layout for the radio buttons to render. Defaults to block to render one below the other. Another option is inline to render side-by-side `options`: an array of options where each option is defined by - `id`: value to be set for the plugin property - `label`: a label to be rendered in UI for that specific option `default`: default value to use when no value is specified	`string`	A group of radio buttons to choose one among a list of available options.	`{ "name": "property-keyvalue", "widget-type": "radio-group", "widget-attributes": { "layout": "inline", "default": "value1", "options": [ { "id": "value1", "label": "Value One" }, { "id": "value2", "label": "Value two" } ] } }`
`keyvalue-dropdown`	`delimiter`: delimiter for the key-value pairs `dropdownOptions`: list of drop-down options to display `kv-delimiter`: delimiter between key and value	`string`	Similar to keyvalue widget, but with a drop-down value list	`{ "name": "property-keyvalue-dropdown", "widget-type": "keyvalue-dropdown", "widget-attributes": { "delimiter": ",", "kv-delimiter": ":", "dropdownOptions": [ "Option1", "Option2"] } }`
`non-editable-schema-editor`	`schema`: schema that will be used as the output schema for the plugin	`string`	A non-editable widget for displaying a schema	`{ "name": "property-non-editable-schema-editor", "widget-type": "non-editable-schema-editor", "widget-attributes": {} }`
`number`	`default`: default value for the widget `max`: maximum value for the number box `min`: minimum value for the number box	`string`	Default HTML number textbox that only accepts valid numbers	`{ "name": "property-number", "widget-type": "number", "widget-attributes": { "default": "1", "min": "1", "max": "100" } }`
`password`	No attributes	`string`	Default HTML password entry box	`{ "name": "property-password", "widget-type": "password", "widget-attributes": {} }`
`plugin-list`	`plugin-type`: The type of plugin that will be fetched	`string`	A drop-down with a list of plugins	`{ "name": "property-plugin-list", "widget-type": "plugin-list", "widget-attributes": { "plugin-type": "jdbc" } }`
`python-editor`	`default`: default `string` value for the widget	`string`	An editor to write Python code as a value of a property	`{ "name": "property-python-editor", "widget-type": "python-editor", "widget-attributes": { "default": "def transform(input, emitter, context):\ \\n emitter.emit(input)\\n" } }`
`schema`	`schema-default-type`: default type for each newly-added field in the schema `schema-types`: list of schema types for each field from which the user can chose when setting the schema	`string`	A four-column, editable table for representing the schema of a plugin	`{ "name": "property-schema", "widget-type": "schema", "widget-attributes": { "schema-default-type": "string", "schema-types": [ "boolean", "int", "long", "float", "double", "bytes", "string", "map<string, string>" ] } }`
`select`	`default`: default value from the list `values`: list of values for the drop-down	`string`	An HTML drop-down with a list of values; allows one choice from the list	`{ "name": "property-select", "widget-type": "select", "widget-attributes": { "default": "Bananas", "values": ["Apples", "Oranges", "Bananas"] } }`
`textarea`	`default`: default value for the widget `rows`: height of the `textarea`	`string`	An HTML `textarea` element which accepts a default value attribute and a height in rows	`{ "name": "property-textarea", "widget-type": "textarea", "widget-attributes": { "default": "Default text.", "rows": "1" } }`
`textarea-validate`	`placeholder`: placeholder text for the textarea `validate-endpoint`: plugin function endpoint to hit to validate the contents of the textarea `validate-button-text`: label of the validate button `validate-success-message`: message to display when validation succeeds	`string`	An HTML `textarea` element with a button to validate its contents using a plugin function endpoint	`{ "name": "property-to-validate", "widget-type": "textarea-validate", "widget-attributes": { "placeholder": "E.g. ((token['Data Quality']['error'] / token['File']['output']) * 100) > runtime['error_percentage']", "validate-endpoint": "validate", "validate-button-text": "Validate", "validate-success-message": "Expression is valid" } }`
`textbox`	`default`: default value for the widget	`string`	An HTML textbox, used to enter any string, with a default value attribute	`{ "name": "property-textbox", "widget-type": "textbox", "widget-attributes": { "default": "Default text." } }`
`toggle`	`on`: 'On' state of the toggle, defined by `value` - value to be set for the plugin property, and `label` - a label to be rendered in UI, limited to 64 characters. `off`: 'Off' state of the toggle, defined by `value` - value to be set for the plugin property, and `label` - a label to be rendered in UI, limited to 64 characters. `default`: default value for the widget	`string`	A toggle widget that allows toggling between 'on' and 'off' states	`{ "name": "property-toggle", "widget-type": "toggle", "widget-attributes": { "on": { "value": "on", "label": "On" }, "off": { "value": "off", "label": "Off" }, "default": "on" } }`

Plugin Function

A plugin function is a method exposed by a particular plugin that can be used for a specific task, such as fetching an output schema for a plugin.

These fields need to be configured to use the plugin functions in the CDAP UI:

method: Type of request to make when calling the plugin function from the CDAP UI (for example: GET or POST)
widget: Type of widget to use to import output schema
plugin-method: Name of the plugin method to call (as exposed by the plugin)
required-fields: Fields required to call the plugin method
missing-required-fields-message: A message for the user as to why the action is disabled in the CDAP UI, displayed when required fields are missing values

The last two properties (required-fields and missing-required-fields-message) are solely for use by the CDAP UI and are not required for all widgets. However, the first four fields are required fields to use a plugin method of the plugin in the CDAP UI.

With plugin functions, if the widget is not supported in the CDAP UI or the plugin function map is not supplied, the user will not see the widget in the CDAP UI.

Widget Type	Widget attributes	Description	Example Widget JSON

Widget Type	Widget attributes	Description	Example Widget JSON
`outputSchema`	`label`: Label for the button `btnClass`: bootstrap css class to add to the button `multiple-inputs`: boolean to indicate if there are multiple input schemas	Widget to populate output schema for a plugin. This is specifically used to populate output schema and not any other property.	`{ "widget-type": "csv", "label": "Group by fields", "name": "groupByFields", "widget-attributes": { "delimiter": ",", "value-placeholder": "Field Name" }, "plugin-function": { "method": "POST", "widget": "outputSchema", "label": "Get output schema", "button-class": "btn-primary", "plugin-method": "outputSchema", "required-fields": ["groupByFields", "aggregates"], "missing-required-fields-message": "'Group By Fields' & 'Aggregates' properties are required to fetch schema." } }`
`getPropertyValue`	`label`: Label for the button	Widget to populate any property in a plugin. This widget should be used in the property which needs to fetch its value from a plugin function implemented in the plugin backend	`{ "widget-type": "select", "label": "Format", "name": "format", "widget-attributes": { "values": [ "avro", "blob", "csv", "delimited", "json", "parquet", "text", "tsv" ], "default": "text" }, "plugin-function": { "method": "POST", "widget": "getPropertyValue", "widget-attributes": { "label": "Get Schema Value" }, "required-fields": ["path"], "missing-required-fields-message": "Please provide path field", "plugin-method": "getSchema" } }`

Example Plugin

In the case of a Batch Source plugin example, the configuration-groups, with additional widgets to show the groupByFields and aggregates properties and using a plugin-function, could be represented by:

{
  "configuration-groups": [
    {
      "label": "Batch Source",
      "properties": [
        {
          "name": "name",
          "widget-type": "dataset-selector"
        },
        {
          "name": "basePath",
          "widget-type": "textbox"
        },
        {
          "name": "groupByFields",
          "widget-type": "textbox",
          "plugin-function": {
            "method": "POST",
            "widget": "outputSchema",
            "plugin-method": "outputSchema",
            "required-fields": ["groupByFields", "aggregates"],
            "missing-required-fields-message":
              "Both 'Group By Fields' and 'Aggregates' properties are required to fetch the schema."
          }
        },
        {
          "name": "aggregates",
          "widget-type": "textbox"
        },
        {
          "name": "duration",
          "widget-type": "textbox"
        },
        {
          "name": "duration",
          "widget-type": "textbox"
        },
        ...
      ]
    }
  ]
}

Inputs

Beginning with version 1.2 of the specification, a plugin can accept multiple input schemas and from them generate a single output schema. Using the field multipleInputs and setting it to true tells the CDAP UI to show the multiple input schemas coming into a specific plugin, instead of assuming that all of the schemas coming in from different plugins are identical.

This is an optional object, and if it is not present, it is assumed that all of the schemas coming in from any connected plugins are identical. Currently, only one value (multipleInputs) is accepted.

For example:

{
  "metadata": {
    ...
  },
  "configuration-groups": [
    ...
  ],
  "inputs": {
    "multipleInputs": true
  },
  "outputs": [
    ...
  ]
}

Outputs

The outputs is a list of plugin properties that represent the output schema of a particular plugin.

The output schema for a plugin can be represented in two different ways, either:

via an explicit schema using a named property; or
via an implicit schema

Output properties are configured in a similar manner as individual properties in configuration groups. They are composed of a name and a widget-type, one of either schema (for an explicit schema) or non-editable-schema-editor (for an implicit schema).

With the schema widget type, a list of widget attributes can be included; with the non-editable-schema-editor, a schema to be displayed is added instead.

An explicit schema using a property can be defined as the output schema and then will be editable through the CDAP UI.

For example, a "Batch Source" plugin could have a configurable output schema named data-format, displayed for editing with the schema widget-type, with a default type of string, and a list of the types that are available:

{
  "outputs": [
    {
      "name": "data-format",
      "widget-type": "schema",
      "widget-attributes": {
        "schema-default-type": "string",
        "schema-types": [
          "boolean",
          "int",
          "long",
          "float",
          "double",
          "string",
          "map<string, string>"
        ]
      }
    }
  ]
}

An implicit schema is a pre-determined output schema for a plugin that the plugin developer enforces. The implicit schema is not associated with any properties of the plugin, but instead shows the output schema of the plugin, to be used for visual display only.

An example of this is from the KeyValueTable Batch Source plugin:

{
  "outputs": [
    {
      "widget-type": "non-editable-schema-editor",
      "schema": {
        "name": "etlSchemaBody",
        "type": "record",
        "fields": [
          {
            "name": "key",
            "type": "bytes"
          },
          {
            "name": "value",
            "type": "bytes"
          }
        ]
      }
    }
  ]
}

Widget types for output properties are limited to ensure that the schema that is propagated across different plugins in the CDAP UI is consistent. For output properties, CDAP uses use either schema-editor for editable schemas and non-editable-schema-editor for non-editable schemas.

Dataset Jumps

Beginning with version 1.3 of the specification, a plugin can be specified (using jump-config) with a map of dataset "jumps". They specify which plugin property names are either a dataset that can be used, in the CDAP UI, to directly jump to a detailed view of the dataset.

This is an optional object, and if it is not present, no jump links will be created in the CDAP UI. Jump links are not active in the Pipeline Studio.