Proposal II

To have a separate section for filters at the same level as configuration-groups. An example of how this will look like would be,

INITIAL PROPOSAL
================
{
  "outputs": [],
  "display-name": "",
  "configuration-groups": [
    {
      "label": "Basic",
      "properties": [
        { 
          "widget-type": "textbox",
          "name": "propertyZ",
          "label": "Property Z",
          "widget-attributes": {
            "placeholder": "Name used to identify this source for lineage"
          }
        },
        {
          "widget-type": "textbox",
          "name": "propertyW",
          "label": "Property W",
          "widget-attributes": {
            "placeholder": "Name used to identify this source for lineage"
          }
        }
      ]
    },
    {
      "Label": "Group 2",
      "properties": [
        {
          "name": "propertyJ",
          "Label": "Property J",
          "widget-type": "textbox",
          "widget-attributes": {
            "default": "auto-detect"
         },
        },
        {
          "name": "propertyK",
          "Label": "Property K",
          "widget-type": "textbox",
          "widget-attributes": {
            "default": "auto-detect"
          }
        }
      ],
    }
  ],
  "filters": [
    {
      "condition": {
        "property": "propertyX",
        "operator": "equal",
        "value": "Yes"
      },
      "show": [
		{
		  type: "property" | "group",
		  name: "propertyW"
		},
	    {
		  type: "property" | "group",
		  name: "propertyZ"
		}
	  ],
    }
  ]
}

Things to note,

Things to note: 

• Each condition object under a filter specifies the condition based on which the properties under "show" gets shown or hidden.
• Expression:
  • A plugin developer can instead choose to define a javascript expression that evaluates to a boolean under condition.
  • The javascript expression will only consider numbers, strings and boolean while evaluating conditions. Since context for evaluating the expression will be the plugin properties we expect the condition to be composed of numbers, boolean and strings. This requires we convert the values of the plugin properties to appropriate types instead of having everything as string.
  • The expression is represented in javascript expression language. This not to be confused with Jexl defined as part of the Apache foundation for java language
  • More reference to the javascript expression language can be found here. Jexl Github repo
  • Even though the documentation talks about using expressions that evaluates to some value, the CDAP UI will only be looking for boolean results to show/hide properties under a filter. Say for instance if the plugin developer chooses to ternary that evaluates to a string (splitBy > 1 ? "value1" : "value2") in this case both are truth'y responses and will show the properties all the time. It is imperative that the expressions have to evaluate to a boolean for UI to correctly show/hide plugin properties as part of the filter.
  • The context for the expression would be the map of plugin properties. As an example,

  Database plugin:
  {
    "jdbcPluginType": "jdbc",
    "numSplits": "2",
    "enableAutoCommit": "false",
    "columnNameCase": "No change",
    "transactionIsolationLevel": "TRANSACTION_SERIALIZABLE",
    "referenceName": "ref1",
    "jdbcPluginName": "mysql",
    "connectionString": "jdbc:mysql://localhost/mydb?useLegacyDatetimeCode=false",
    "importQuery": "select * from query_table where $CONDITION",
    "boundingQuery": "select * from query_table",
    "splitBy": "date",
    "user": "username",
    "password": "password"
  }

  And the plugin developer can choose to show splitBy property if the numSplits is greater than 1
  The following JSON shows the usage of plugin properties inside the expression.

  Example with expression under condition:
  "filters": [
    {
      "name": "showBoundingQuery",
      "condition": {
        "expression": "numSplits > 1"
      },
      "show": [
        {
          "type": "property",
          "name": "boundingQuery"
        },
        {
          "type": "property",
          "name": "splitBy"
        }
      }
    }
  ]

  
  Since we don't have a way for the plugin developer to restrict the usage of expression, if the plugin widget spec has filters with both operator-value-property combination and an expression the precedence will be given to the expression and the rest of the properties in the condition object will be ignored.

• On a high level we could see this format as a list of filters with specific names with a condition to show or hide certain properties.
• The name is purely used for readability for plugin developer. This will not be surfaced in UI or anywhere else.

• Default operator: In the condition object above operator property under each filter is optional. If not specified it is by default equal . We need to decide on what other operators in the future would make sense depending on the type of data we allow for a plugin property.

• Operator list: As we have restricted to only show when a condition is satisfied, it is imperative we need to provide operators that involves more than just equality check. Currently we should support,

• equal to

• not equal to

• exists

• does not exist

• Handling required properties: If a required property is set to be hidden the UI should default it to be never hidden. There could be a case where a plugin developer could potentially configure their widget JSON to hide a required property which will make deploying the pipeline to fail.

• default case: We will only "show" properties when the conditions are matched for any specific filter. This means the "default" case for the UI is to hide all the properties across all the filters. This would also mean the UI will look through all the filters and hide all those properties to start with unless the condition is true. In the above example properties "PropertyW, PropertyZ, PropertyA, PropertyB, PropertyC & PropertyD" will hidden when the user opens the plugin configuration for the first time. Once the propertyX is set with an appropriate value, only those properties whose condition evaluates to true gets shown. We will not be having a "hide" property under individual filters.

• When the plugin user resets the toggle property (in the above example propertyX) values for properties that are now hidden based on this action will be reset. The user will have to provide value after every reset.

• Macro usage: When user provides a macros (${macroValue}) then UI should invalidate all the filters concerning that specific property. At this point UI cannot determine if it can show or hide anything based on the value of the toggle property. Example, if PropertyX from the above example is given a macro then all the properties "PropertyW, PropertyZ, PropertyA, PropertyB, PropertyC & PropertyD" will become visible as then its up to the user to determine which one to configure.

Error Validation:

This opens up a lot of corner cases during stage level error validation. One of the possible scenarios would be, 

Intersecting condition

1. 1. Plugin developer specifies PropertyC to be shown only when PropertyX is 30

   2. Plugin developer is also expecting(implicitly required) PropertyC when both PropertyX and PropertyL have some value.

   3. Plugin user specifies a value of 20 to PropertyC and provides a value to PropertyL. 

   4. Now during stage level validation, the backend will error out saying missing value for PropertyC.

   5. At this point the user would be confused because they wouldn't even be seeing PropertyC as it is hidden.
      

At this point the UI should show the hidden property as it is required by some logic set by the plugin developer during stage level validation (once user clicks on validate button in plugin modal).

Already set hidden property

This means if the property has a value the UI will be forced to show the property even if it should be hidden based on the filter conditions. 

A hidden property could have a value,

• • if the user either imports the pipeline JSON or,

  • if the user clones an older pipeline or,

  • if it gets populated by the "browse" functionality that we recently introduced.

Pros

Proposal III

Provide the ability to subgroup related fields under a condition. This way the user experience becomes,

if propertyX is value1 then  
  all the properties that gets shown form subgroup1
if propertyX is value2 then
  all the properties that gets shown form subgroup2
default
  show none

Based on this assumption the example widget JSON would like this,

{
  "outputs": [],
  "display-name": "",
  "configuration-groups": [{
      "label": "Basic",
      "properties": [{
          "widget-type": "condition-branch",
          "name": "propertyZ",
          "label": "Property Z",
          "widget-attributes": {
            "placeholder": "Name used to identify this source for lineage",
            "type": "select",
            "options": ["yes", "no"]
          },
          "sub-groups": [
            {
              "branch": "yes",
              "properties": [
                {
                  "name": "propertyW",
                  "label": "Property W",
                  "widget-type": "textbox",
                  "widget-attributes": {
                    "placeholder": "auto-detect"
                  }
                },
                {
                  "name": "propertyZ",
                  "label": "Property Z",
                  "widget-type": "textbox",
                  "widget-attributes": {
                    "placeholder": "auto-detect"
                  }
                }
              ]
            },
            {
              "branch": "no",
              "properties": [
                {
                  "name": "propertyJ",
                  "label": "Property J",
                  "widget-type": "textbox",
                  "widget-attributes": {
                    "placeholder": "auto-detect"
                  }
                },
                {
                  "name": "propertyK",
                  "label": "Property K",
                  "widget-type": "textbox",
                  "widget-attributes": {
                    "placeholder": "auto-detect"
                  }
                }
              ]
            }
          ]
        },
        {
          "widget-type": "textbox",
          "name": "propertyW",
          "label": "Property W",
          "widget-attributes": {
            "placeholder": "Name used to identify this source for lineage"
          }
        }
      ]
    },
    {
      "Label": "Group 2",
      "properties": [{
          "name": "propertyJ",
          "Label": "Property J",
          "widget-type": "textbox",
          "widget-attributes": {
            "default": "auto-detect"
          }
        },
        {
          "name": "propertyK",
          "Label": "Property K",
          "widget-type": "textbox",
          "widget-attributes": {
            "default": "auto-detect"
          }
        }
      ]
    }
  ],
}

Pros