Page Comparison

Table of Contents

...

As a framework Apache HttpComponents HttpClient is be used, a successor of Commons HttpClient.

It seems the most widely used/supported by community framework. It is very simple to find all kind of solutions and workaround already implemented, which makes plugin development and maintenance easy. Framework has a built in support for compession, https tunneling, digest auth and lot of other functions.

...

2.1 STEP 1 - Get XML by XPath

XML parsing is done by default Java DOM parser. Which is able to get items by a specified XPath. XPath is super flexible it allows user to get nodes by attribute value, as well as to group nodes from different parents into single result, as well as chose nodes conditionally etc. etc. 

Some XPath examples:

/bookstores/bookstore/book[position()<3]
//title[@lang]
//title[@lang='en']
/bookstores/bookstore/book/price[text()] # convert all subelements to string
/bookstores/bookstore/book[price>35.00]/title

...

Will use the functionality from cdap-formats/TextRecordFormat.java  to validate schema and convert input text to StructuredRecord. 

OAuth2

1.1 General concepts

Before using OAuth2 usually user has to create an application via service site (e.g. Twitter) and register redirect uri, than receive a client_id and client_secret. Which will be used by an application during authentication.

The further steps are shown in the diagram below:

Image Removed

We are implementing grant types "Authorization Code Grant" and "Refresh Token". Other types are not suitable or rarely used. Click here for some context.

Properties:

...

CDAP will start a local server on the given url. Only localhost urls are allowed. For more info click here.

This is a URL where service callbacks with authCode after user enters username and password and agrees to grant the permissions.

...

Moved design information into a separate doc: Plugin OAuth2 Common Module

Properties:

Note: OAuth2 implementation will reside in one of CDAP core modules, since this will need to be re-used by different plugins.

1.2 UI Changes Required. Need to expose an authentication dialog pop-up window to a user

We need to show authentication dialog from the service where user enters his username and passwords, as well as agrees to grant a certain access to our application.

This will require change to UI. We can implement this as plugin-function:

"plugin-function": {
  "method": "POST",
  "widget": "showPageToUser",
  "output-property": "void",
  "plugin-method": "showPageToUser"
}

...

1. Once the user clicks the button, UI runs showPageToUser to get url and headers from plugin using post-function. Example of return from plugin:

   Code Block
   url = https://www.facebook.com/v3.3/dialog/oauth? client_id=3MVG96_7YM2sI9wT6c13RTPp6RDeRBPFc0F5sYfIrKBZPdTK2Yr7jiTwq8u3ykXyBHtlf3lnNMWSN1rqfjA_y &redirect_uri=http://localhost:27435 headers = {} # empty

   On the side note: The url syntax is established by an RFC: https://tools.ietf.org/html/rfc6749#section-4.1.1

   IMPORTANT: During this call plugin will also start a callback server (for more information click here)

2. UI shows the page.
3. Once the page is closed, ui does another call to plugin.
   During this call plugin waits for a callback to be complete. Than exchanges received authCode to pair accessToken, refreshToken and return it back to UI.
4. UI populates the widget field refreshToken with the value. 

refreshToken usually has a permanent lifetime (unlike accessToken), unless invalidated manually by user. With this token during every pipeline run we can ask for an accessToken from tokenUrl. No need for callback server or UI/user involvement at this point.

If OAuth2 is enabled and refreshToken field is not populated. Pipeline deployment will fail. So effectively user will be asked to authenticate once and the information for further authentication will be saved in widget.

...

We will need to start an HTTP callback server on localhost. Let's say http://localhost:27435. The port should be statically configured via widget, we cannot get a random ephemeral port, since callback_url needs to be constant. It is saved on service provider (e.g. Facebook) as a static configuration of OAuth2 application.

After user enters his credentials on authentication page (of let's say Facebook), browser will redirect the response authCode to this server. Since this request is done by client browser (not remote server), this address is not required to be public IP address and we can safely use localhost.

1.4 Refreshing access tokens

Since we save refresh token. instead of access token token (which are short-lived). During every pipeline we will need to get an access token. This is a very simple process. We have to execute only one request. Which will return an access token.

POST /token HTTP/1.1
Host: server.example.com
Content-Type: application/x-www-form-urlencoded

grant_type=refresh_token&refresh_token=tGzv3JOkF0XG5Qx2TlKWIA
&client_id=s6BhdRkqt3&client_secret=7Fjfp0ZBr1KtDRbnfVdmIw

Facebook case

...

TLS/SSL

Some general definitions for more context:

...

...