Specification Audit

Purpose

To help any hydrator plugin developer write a configuration JSON for representing properties of a hydrator plugin, group them and configure the output(s) (the schema of data going out of a plugin) if any required.

Why do we need this?

What this JSON achieves is to help in configuring the configurations of a plugin to be specific components in UI so that overhead of giving the right format of value for a configuration is removed. By default if no configuration is provided for a plugin all the properties appear as text-boxes in UI. It will be difficult to fill in the values for them if it requires it in specific format (for instance JSON or select one from multiple choices etc.,).

What does it have - Structure?

Before we dive into the structure it is imperative we clear up certain terms.

• Configuring (Configurations) – A plugin in hydrator has configurations that needs to be set with appropriate value, for the plugin to function. This configuration could be a string or a number or a JSON blob or a Javascript/Python snippet etc., Configuring here refers to setting plugin configurations with appropriate value and for hydrator developers to do that we need to map configurations to widget to set a value.
•  Output –  Every plugin will have data emitted out of it (passing it to subsequent plugins or dump to a database). An output of a plugin specifies the schema of data that is emitted out of the plugin.
•  Input –  Similar to output every plugin will have an input data coming into the plugin (except for source plugins)

Configuration JSON for a plugin is structured as,

• metadata

• list of groups of configurations

• list of output configurations

‘Groups’ refers to collection of configurations of a plugin. An example structure of a configuration JSON would look like this - Sample JSON

{
 "metadata": {...},
 "inputs": null,
 "configuration-groups": [
   {group1},
   {group2},
   ...
 ],
 "output": [
   {configuration-output-1},
   {configuration-output-2},
   ...
 ]
}

Metadata

Metadata refers to the top level information of a plugin. Metadata map includes,

• label - Label for the plugin in hydrator
• type - Type of the plugin
• name - Name of the plugin (not sure if name and label should be separate)
• artifact-version - Version of the plugin (artifact).
• spec-version - Version of SPEC based on which the json is generated.

Following is an example of the metadata section:

{
 "metadata": {
   "spec-version" : "2.1",
   "label" : "Azure Blob Store"
 },
 ...
}

Groups

A group is a meaningful collection of configurations where related configs are grouped together to view and edit in hydrator. For instance in a Stream plugin we group together the stream, Processing time window & delay together [more explanation is needed as to why they are grouped together]. A group can essentially have a label , representing the label of the group and a list of fields inside the group. These are represented as ‘label’ and ‘fields’ inside the group map. Based on these information a sample JSON for a custom plugin would look something like this,

[
 {
   "label": "My Group Title",
   "properties": [
 {property1},
 {property2},
 ...
    ]
 }
]

Here ‘My Group Title’ is the label of the group that will be grouping fields 1 & 2 under it(as they are related). A field inside a group is nothing but the configuration of the plugin.

Fields

In the above mentioned sample JSON, fields 1 & 2 are configurations of CustomPlugin defined in the backend. We could now configure them in the JSON so that it gets easier to use view/edit them in hydrator.

Configurations for a field

Configurations of a field are the set of properties that we define in the JSON, that governs what/how that particular plugin field gets rendered in hydrator. 

•  ‘How’ refers how should it be displayed - like a textbox, passwordbox, dropdown, JSON editor etc., and 
•  ‘What’ refers to what should it default to. For instance if the property is rendered as a select widget and if we know that the possible list of values for it would be [‘Male’, ‘Female’], then we could configure that particular plugin property to have those values for the dropdown.

Required

• name - Name of the field as specified in the backend.  

• widget-type - Type of widget to represent the field.

Optional

• label - Label for the field. For instance a field coming from the backend like time.ticks.sec could be labelled as ‘Time in Seconds’.

• widget-attributes - Attribute/values required by the widget. For instance, list of values for select widget or a default value for a number-textbox. More details should be available in widgets list section.

• plugin-function - Function available for the plugin. For instance, Database batchsource plugin have a getSchema endpoint method. output-property signify the plugin property that will get updated from this plugin-function action. 

From the above explanation we could enhance the above mentioned sample JSON to something like this,

{
 "configuration-groups": [
   {
     "label": "My Group Title",
     "properties": [
       {
         "name": "Field1",
         "widget-type": "numberbox",
         "widget-attributes": {
           "default": "1"
         },
		 "plugin-function": {
		   "method": "POST",
		   "endpoint": "getSchema",
		   "output-property": "schema"
		 }
       },
       {
         "name": "Field2",
         "widget-type": "json-editor"
       }
     ]
   }
 ],
 "output": [
   {output-property1} 
   {output-property2}
   ...
 ]
}

Widgets

A widget in UI represents a component that we could use to set the value of a property of a plugin. It could be anything representable in HTML. For instance if our CustomPlugin has a property that requires a JSON value, we could use a json-editor to set value to it. A widget, in general, is a component that we can use to represent a property of a plugin to attach a value to it. The following are the list of widgets that we right now have in CDAP that could be readily be used.

{
  "name": "property1",
  "label": "My Property Label",
  "widget-type": "textbox",
  "widget-attributes": {
    "default": "Some Text"
  }
}

{
  "name": "property1",
  "label": "My Property Label",
  "widget-type": "number",
  "widget-attributes": {
    "default": "1",
    "min": "1",
    "max": "100"
  }
}

{
  "name": "property1",
  "label": "My Property Label",
  "widget-type": "password",
  "widget-attributes": {}
}

{
  "name": "property1",
  "label": "My Property Label",
  "widget-type": "datetime",
  "widget-attributes": {
    "default": "current",
    "format": "MM-DD-YYYY"
  }
}

{
  "name": "property1",
  "label": "My Property Label",
  "widget-type": "csv",
  "widget-attributes": {}
}

{
  "name": "property1",
  "label": "My Property Label",
  "widget-type": "dsv",
  "widget-attributes": {
    "delimiter": ":"
  }
}

{
  "name": "property1",
  "label": "My Property Label",
  "widget-type": "json-editor",
  "widget-attributes": {
    "default": "{ \"p1\": \"value\" }"
  }
}

{
  "name": "property1",
  "label": "My Property Label",
  "widget-type": "javascript-editor",
  "widget-attributes": {
    "default": "function transform(input, emitter, context) {\n  emitter.emit(input);\n}"
  }
}

{
  "name": "property1",
  "label": "My Property Label",
  "widget-type": "keyvalue",
  "widget-attributes": {
    "delimiter": ",",
    "kv-delimiter": ":"
  }
}

{
  "name": "property1",
  "label": "My Date Property",
  "widget-type": "keyvalue-dropdown",
  "widget-attributes": {
    "delimiter": ",",
    "kv-delimiter": ":",
    "dropdownOptions": [ "Option1", "Option2"]
  }
}

{
  "name": "property1",
  "label": "My Property Label",
  "widget-type": "select",
  "widget-attributes": {
    "values": ["Apples", "Oranges", "Bananas"],
    "default": "Bananas"
  }
}

{
  "name": "property1",
  "label": "My Property Label",
  "widget-type": "dataset-selector",
  "widget-attributes": {}
}

{
  "name": "property1",
  "label": "My Property Label",
  "widget-type": "ds-multiplevalues",
  "widget-attributes": {
    "delimiter": ",",
    "values-delimiter": ":",
    "numValues": "3",
    "placeholders": ["Input Field", "Lookup", "Output Field"]
  }
}

The versioning of plugin config json is closely tied to plugin deployment and maintenance. Every time a hydrator plugin developer deploys a plugin(artifact) he(she) will couple the plugin JARs with config JSON and documentation. The CLI or UI that is deploying the artifact (plugin) will add the config JSON and documentation with it. The config JSON and documentation will be added as properties of an artifact. Each plugin under an artifact will be added as a property to the artifact and each property (plugin) will have JSON as its value with doc and widgets as keys. an example structure would look something like this,

GET: /namespaces/:namespaceId/artifact/:artifactId/properties[/:propertyId]

properties: {
 plugin1: {
   doc: 'Some Doc',
   widgets: /* config JSON */
 },
 plugin2: {
   doc: 'Some Doc for plugin2',
   widgets: /* config JSON for plugin2*/
 }
}

Why do we need this?

This will ease out the plugin deployment process whenever the plugin developer wants to update the plugin (artifact). The maintenance of plugin, with doc and ui config, would be in single place and could be consistent since cdap (or hydrator) manages different versions of artifacts (plugins) and a doc and config for each version.

FAQs 

• • How are the groups ordered?

    The configuration groups appear in the order that they are in the ‘configurations’ array.

• • How are the configurations ordered?

    The same way as groups.

• • What happens if I specify a configuration that is not coming in from CDAP backend?

    The hydrator UI won’t show that particular configuration. Configurations that come from backend that has configurations mapped will appear in UI.

• • What happens when backend sends a configuration for a plugin that is not configured in the configurations JSON file?

    In that case the hydrator will create a separate group at the bottom of the configurations named ‘Others’ and add all non-matching configurations for that plugin as textboxes.

• • What happens when I enter a widget that is not available?

    Hydrator defaults it to textbox

• • What should be done to have a schema that I know will be the schema for the output data? An implicit schema of data emitted out of a plugin.

    The output of plugin can have list of configurations and if we configure it to use ‘noneditable-schema-editor’ widget without a label and name then hydrator will use the schema under the field’s widget attributes. An example of an implicit schema would look something like this,

Example:

{
 "configurations": [
   {group1}
   {group2}
   ...
 ],
 "output": [
   {
     "widget-type": "noneditable-schema-editor",
     "widget-attributes": {
       "schema": {
         "id": { "type": "long" },
         "message": { "type": "string" },
         "lang": { "type": "string", "isNullable": true },
         "time": { "type": "long", "isNullable": true },
         "favCount": {"type": "int"},
         "rtCount": {"type": "int"},
         "source": { "type":"string", "isNullable": true },
         "geoLat": { "type":"double", "isNullable": true },
         "geoLong": { "type":"double", "isNullable": true},
         "isRetweet": { "type": "boolean" }
       }
     }
   }
 ]
}