Plugin Presentation

When a plugin is displayed in the CDAP UI, its properties are represented by widgets in the Pipeline Studio. Each property of a plugin is represented, by default, as a simple textbox in the user interface. By including an appropriate plugin widget JSON file, you can customize that presentation to use a different interface, as desired.

The available widgets that can be used are described below. Note that some of the plugins available for CDAP may include use of additional widgets that are not listed here. We recommend only using the list of plugins shown here, as these other widgets are internal to CDAP and may not be supported in a future release.

This document describes version 1.5 of the plugin specification. Changes to the specification are described in the Specification Changes and should be checked if you are using a version of the specification earlier than the current.

Plugin Widget JSON

To customize the plugin display, a plugin can include a widget JSON file that specifies the particular widgets and the sets of widget attributes used to display the plugin properties in the CDAP UI.

The widget JSON is composed of:

a map of metadata
a list of property configuration groups
a map of inputs
a list of outputs
a map of dataset jumps

Each configuration group consists of a list of the individual properties and the widgets to be used. For example:

{
  "metadata": {
    ...
  },
  "display-name": "Plugin Display Name",
  "icon": {
    "type": "builtin|link|inline",
    "arguments": {
       "url": "http://localhost/myicon.png",
       "data": "data:image/png;base64,..."
    }
  },
  "configuration-groups": [
    {"label": "Group 1",
      "properties": [
      ...
      ]
    },
    {"label": "Group 2",
      "properties": [
      ...
      ]
    },
    ...
  ],
  "inputs": {
    ...
  },
  "outputs": [
    {"output-property-1"},
    {"output-property-2"},
    ...
  ],
  "jump-config": {
    ...
  }
}

Metadata

Metadata refers to the top-level information about a plugin. The only information required is the a map consisting of the spec-version, the version of the specification which the JSON follows.

Current version: 1.5. For example:

{
  "metadata": {
    "spec-version" : "1.5"
  },
  ...
}

Display Name

The display-name field specifies the name of the plugin as it is displayed on the CDAP UI. A display name can be different from the plugin name, and is not guaranteed to be unique. A display name can contain whitespace. If not specified, the plugin name is used as the display name.

Icon

The icon field allows users to specify custom icons for their plugins. Users can specify icons for their plugins in three different ways, depending on the value of the type field contained inside the icon field:

builtin: The icon type builtin indicates that the icon for the plugin is defined in CDAP, and is not specified in the plugin's widget JSON file. This is the default icon type, if an icon is not specified. If such an icon is not found, the CDAP UI will use a default icon type.
link: The icon type link indicates that the icon for the plugin should be fetched from a URL specified in the arguments map. When the type of icon is link, the user is expected to provide a link to the icon image in a url attribute inside the arguments map.
inline: The icon type inline indicates that the icon for the plugin should be decoded from a base64 encoded image specified in the arguments map. When the type of icon is inline, the user is expected to provide a base64 encoded image in a data attribute inside the arguments map.

Icon Image Specification - The image used for an icon for a CDAP plugin should be in PNG format (http://www.libpng.org/pub/png/). The size of the image should be 50 X 50 pixels. The image background should be transparent.

Configuration Groups

Configuration groups are a simple grouping of properties of a plugin. A configuration group is represented as a JSON object with a label and an ordered list of plugin properties for that group.

For example, in a Batch Source plugin, properties such as name, basePath, duration, and delay could be grouped into a Batch Source Configuration.

Setting the hideByDefault property of a configuration group results in the group properties to be hidden by default.

In the case of a Batch Source plugin, it could look like this:

{
  "configuration-groups": [
    {
      "label": "Batch Source Configuration",
      "properties": [
        {
          "name": "name",
          ...
        },
        {
          "name": "basePath",
          ...
        },
        {
          "name": "duration",
          ...
        },
        {
          "name": "delay",
          ...
        }
      ]
    }
  ]
  ...
}

Once a group is established, you can configure how each of the individual properties inside the group is represented in the CDAP UI.

Property Configuration

Each individual property of the plugin is represented by a configuration, composed of:

name: Name of the field (as supplied by the CDAP server for the artifact).
label: Text string displayed in the CDAP UI beside the widget.
widget-type: The type of widget to be used to represent this property.
widget-attributes: A map of attributes that the widget type requires to be defined in order to render the property in the CDAP UI. The attributes required depend on the widget type used.
plugin-function: An optional map of plugin method and its widget attributes that can be applied to a particular plugin property.

Note: All properties and property values are case-sensitive.

To find the available field names, you can use the Artifact Microservices to retrieve plugin details for an artifact, which will include all the available field names. (If the artifact is your own, you will already know the available field names from your source code.)

In this example of a Batch Source plugin and its configuration-groups, four different properties are defined; three use a textbox widget, while one uses a dataset-selector. The groupByFields property includes a plugin function, used to fetch an output schema for the widget:

Plugin Widgets

A widget in the CDAP UI represents a component that will be rendered and used to set the value of a property of a plugin. These are the different widgets, their type, their attributes (if any), their output data type, a description, sample JSON, that we support in CDAP pipelines as of version 6.2.0.

Widget Type	Widget Attributes	Output Data Type	Description	Example Widget JSON

Widget Type	Widget Attributes	Output Data Type	Description
`csv`	`delimiter`: the delimiter between each set of values `value-placeholder`: placeholder for each row	Comma-separated `string`	Comma-separated values; each value is entered in a separate box
`daterange`	No attributes	`string`	Selector for a date and time range using graphical calendar
`datetime`	No attributes	`string`	Selector for a single date and time using graphical calendar
`dataset-selector`	No attributes	`string`	A type-ahead textbox with a list of datasets from the CDAP instance
`ds-multiplevalues`	`delimiter`: the delimiter between each set of values `numValues`: number of values (number of delimiter-separated values) `placeholders`: array of placeholders for each value's textbox `values-delimiter`: the delimiter between each value	`string`	A delimiter-separated values widget that allows specifying lists of values separated by delimiters
`dsv`	`delimiter`: delimiter used to separate the values	Delimiter-separated `string`	Delimiter-separated values; each value is entered in a separate box
`hidden`	`default`: default `string` value for the widget	`string`	This "hidden" widget allows values to be set for a property but hidden from users. A default can be supplied that will be used as the value for the property.
`input-field-selector`	No attributes `multiselect`	`string`	A dropdown widget with a list of columns taken from the input schema. Selecting sets the input column for that plugin property.
`javascript-editor`	`default`: default `string` value for the widget	`string`	An editor to write JavaScript code as a value of a property
`json-editor`	`default`: default serialized JSON value for the widget	`string`	A JSON editor that pretty-prints and auto-formats JSON while it is being entered
`keyvalue`	`delimiter`: delimiter for the key-value pairs `kv-delimiter`: delimiter between key and value `key-placeholder`: placeholder for the key input `value-placeholder`: placeholder for the value input	`string`	A key-value editor for constructing maps of key-value pairs
`multi-select`	`delimiter`: delimiter to concatenate multiple values to form a single string `options`: array of options where each option is defined by - `id`: value to be set for the plugin property - `label`: a label to be rendered in UI for that specific option `default`: a string containing default value(s) to use when rendering the widget for the first time. You must use the specified delimiter to separate values.	`string`	Multi-select dropdown to choose one or more from a list of available options.
`radio-group`	`layout`: Layout for the radio buttons to render. Defaults to block to render one below the other. Another option is inline to render side-by-side `options`: an array of options where each option is defined by - `id`: value to be set for the plugin property - `label`: a label to be rendered in UI for that specific option `default`: default value to use when no value is specified	`string`	A group of radio buttons to choose one among a list of available options.
`keyvalue-dropdown`	`delimiter`: delimiter for the key-value pairs `dropdownOptions`: list of drop-down options to display `kv-delimiter`: delimiter between key and value	`string`	Similar to keyvalue widget, but with a drop-down value list
`non-editable-schema-editor`	`schema`: schema that will be used as the output schema for the plugin	`string`	A non-editable widget for displaying a schema
`number`	`default`: default value for the widget `max`: maximum value for the number box `min`: minimum value for the number box	`string`	Default HTML number textbox that only accepts valid numbers
`password`	No attributes	`string`	Default HTML password entry box
`plugin-list`	`plugin-type`: The type of plugin that will be fetched	`string`	A drop-down with a list of plugins
`python-editor`	`default`: default `string` value for the widget	`string`	An editor to write Python code as a value of a property
`schema`	`schema-default-type`: default type for each newly-added field in the schema `schema-types`: list of schema types for each field from which the user can chose when setting the schema	`string`	A four-column, editable table for representing the schema of a plugin
`select`	`default`: default value from the list `values`: list of values for the drop-down	`string`	An HTML drop-down with a list of values; allows one choice from the list
`textarea`	`default`: default value for the widget `rows`: height of the `textarea`	`string`	An HTML `textarea` element which accepts a default value attribute and a height in rows
`textarea-validate`	`placeholder`: placeholder text for the textarea `validate-endpoint`: plugin function endpoint to hit to validate the contents of the textarea `validate-button-text`: label of the validate button `validate-success-message`: message to display when validation succeeds	`string`	An HTML `textarea` element with a button to validate its contents using a plugin function endpoint
`textbox`	`default`: default value for the widget	`string`	An HTML textbox, used to enter any string, with a default value attribute
`toggle`	`on`: 'On' state of the toggle, defined by `value` - value to be set for the plugin property, and `label` - a label to be rendered in UI, limited to 64 characters. `off`: 'Off' state of the toggle, defined by `value` - value to be set for the plugin property, and `label` - a label to be rendered in UI, limited to 64 characters. `default`: default value for the widget	`string`	A toggle widget that allows toggling between 'on' and 'off' states

Plugin Function

A plugin function is a method exposed by a particular plugin that can be used for a specific task, such as fetching an output schema for a plugin.

These fields need to be configured to use the plugin functions in the CDAP UI:

method: Type of request to make when calling the plugin function from the CDAP UI (for example: GET or POST)
widget: Type of widget to use to import output schema
plugin-method: Name of the plugin method to call (as exposed by the plugin)
required-fields: Fields required to call the plugin method
missing-required-fields-message: A message for the user as to why the action is disabled in the CDAP UI, displayed when required fields are missing values

The last two properties (required-fields and missing-required-fields-message) are solely for use by the CDAP UI and are not required for all widgets. However, the first four fields are required fields to use a plugin method of the plugin in the CDAP UI.

With plugin functions, if the widget is not supported in the CDAP UI or the plugin function map is not supplied, the user will not see the widget in the CDAP UI.

Widget Type	Widget attributes	Description	Example Widget JSON

Widget Type	Widget attributes	Description	Example Widget JSON
`outputSchema`	`label`: Label for the button `btnClass`: bootstrap css class to add to the button `multiple-inputs`: boolean to indicate if there are multiple input schemas	Widget to populate output schema for a plugin. This is specifically used to populate output schema and not any other property.
`getPropertyValue`	`label`: Label for the button	Widget to populate any property in a plugin. This widget should be used in the property which needs to fetch its value from a plugin function implemented in the plugin backend

Example Plugin

In the case of a Batch Source plugin example, the configuration-groups, with additional widgets to show the groupByFields and aggregates properties and using a plugin-function, could be represented by:

Inputs

Beginning with version 1.2 of the specification, a plugin can accept multiple input schemas and from them generate a single output schema. Using the field multipleInputs and setting it to true tells the CDAP UI to show the multiple input schemas coming into a specific plugin, instead of assuming that all of the schemas coming in from different plugins are identical.

This is an optional object, and if it is not present, it is assumed that all of the schemas coming in from any connected plugins are identical. Currently, only one value (multipleInputs) is accepted.

For example:

Outputs

The outputs is a list of plugin properties that represent the output schema of a particular plugin.

The output schema for a plugin can be represented in two different ways, either:

via an explicit schema using a named property; or
via an implicit schema

Output properties are configured in a similar manner as individual properties in configuration groups. They are composed of a name and a widget-type, one of either schema (for an explicit schema) or non-editable-schema-editor (for an implicit schema).

With the schema widget type, a list of widget attributes can be included; with the non-editable-schema-editor, a schema to be displayed is added instead.

An explicit schema using a property can be defined as the output schema and then will be editable through the CDAP UI.

For example, a "Batch Source" plugin could have a configurable output schema named data-format, displayed for editing with the schema widget-type, with a default type of string, and a list of the types that are available:

An implicit schema is a pre-determined output schema for a plugin that the plugin developer enforces. The implicit schema is not associated with any properties of the plugin, but instead shows the output schema of the plugin, to be used for visual display only.

An example of this is from the KeyValueTable Batch Source plugin:

Widget types for output properties are limited to ensure that the schema that is propagated across different plugins in the CDAP UI is consistent. For output properties, CDAP uses use either schema-editor for editable schemas and non-editable-schema-editor for non-editable schemas.

Dataset Jumps

Beginning with version 1.3 of the specification, a plugin can be specified (using jump-config) with a map of dataset "jumps". They specify which plugin property names are either a dataset that can be used, in the CDAP UI, to directly jump to a detailed view of the dataset.

This is an optional object, and if it is not present, no jump links will be created in the CDAP UI. Jump links are not active in the Pipeline Studio.

For example:

In this example, the datasetName field of the dataset-selector will have a "jump" link added in the CDAP UI.

Example Widget JSON

Based on the above specification, we can write a widget JSON for a Batch Source plugin (with the properties of name, basePath, duration, delay, groupByFields, aggregates, and an editable output explicit schema) as:

Dynamic Plugin Filters

The CDAP plugin UI allows plugin developers to accompany logic to show/hide plugin properties based on the plugin configuration. For instance, if the developer of a plugin has “Authentication method” as one of the plugin properties, and based on the authentication method the developer wants only a specific set of properties to be configured (say service account or authentication token or oauth token), they can add filters to widget json to add logic to show/hide specific properties based on the value of the Authentication method.

Filters to plugin’s widget json can be added under filters property:

Each filter under the filters contains three required properties.

name - Name of the filter. This is solely added here to name every filter for easy readability.
condition - This specifies the condition to be evaluated when plugin properties change to show/hide specific properties.
show - Properties/Groups to show in plugin UI.

Note: The property show is named intentionally that it will show the properties/groups only if the condition is true. So, by default those properties will be hidden in UI. Each time a plugin property is changed the conditions are evaluated. When the condition evaluates to true, the UI shows the properties. The property to show or hide needs to be nullable (with @Nullable when defining the java attribute).

The condition property under the filter can be of two formats: simple operator and jexl expression.

Simple operator: The CDAP UI provides some basic operators for use when specifying filter conditions. Those include:

exists
does not exist
equal to
not equal to.

A simple operator has two required properties and an optional property:

operator - name of the operator (mentioned above)
property - name of the property to apply the operator
value - If the the operator is binary, the value to compare it against. (greater than, less than etc.,)

jexl expression: Is a javascript expression that evaluates to true/false.

The javascript expression will only consider numbers, strings, and boolean while evaluating conditions. Since context for evaluating the expression will be the plugin properties, we expect the condition to be composed of numbers, boolean, and strings. This requires that we convert the values of the plugin properties to appropriate types instead of having everything as string.

The expression is represented in javascript expression language. This is not to be confused with Jexl defined as part of the Apache foundation for java language. For more information about the javascript expression language, see Jexl Github repo.

A jexl expression has one required property:

expression: Any valid javascript expression

Every time an expression is evaluated, the plugin properties are provided as context for evaluating the expression. This allows plugin developers to use plugin properties in expressions. Say for instance the plugin properties look like this:

The plugin developer could add a filters section that could use one of the properties to evaluate the expression:

Based on the above filter, if numSplits property is set to a value greater than 1, the UI will show boundingQuery and splitBy properties.

Macros: Even though the plugin developer can specify filters to show/hide properties, when the pipeline developer starts to use macros for plugin properties, all the filter logic will be ignored. Since the UI cannot resolve the macros during development time, it is not possible for the UI to evaluate the expressions.

Error validation (Required property): Since the plugin developer can configure multiple filters as part of the widget json, there could be a case where the plugin developer hides a plugin property that is a required for plugin’s functionality. Since the UI knows that a specific plugin property is annotated as required by the plugin developer, it, by default, gets shown when rendered. Even though the plugin developer has hidden it under a filter condition, the UI has to show as it is required.

Specification Changes

These changes describe changes added with each version of the specification.

1.1: Initial version of the specification.
1.2: Added multiple inputs for a plugin.
1.3: Added jump-config to specify which property names are to be connected in the CDAP UI to a detailed view of a dataset.
1.4: Added widget-type: hidden.