Page Comparison

Table of Contents

...

As a framework Apache HttpComponents HttpClient is be used, a successor of Commons HttpClient.

It seems the most widely used/supported by community framework. It is very simple to find all kind of solutions and workaround already implemented, which makes plugin development and maintenance easy. Framework has a built in support for compession, https tunneling, digest auth and lot of other functions.

Properties:

[1] Unfortunately we cannot do validation here. Even though most commonly body in API requests is a JSON for JSON APIs or an XML for XML SOAP APIs. Theoretically it can be anything.

[2] Pagination types, where next page url is on previous taken from the previous page, are the one which do not support SEND_TO_ERROR or IGNORE.

...

Parallelization

There are two reasons why we should not parallelize the requests:

...

The above is a bit messy cause we cannot dynamically change the content of widget depending on pagination type. Which makes it a mix of properties for different pag_types. Is not super user-friendly for end-user. For now I will a placeholder which says which pagination type property coresponds to.

Pagination type is none

Plugin will request a single page.

...

The plugin stops reading when a page returns no records or 404. Or when reached max_index (if it's not empty) 

...

Different APIs use very different styles of pagination. In the simple cases they return link in header or some field of response JSON.

...

Pagination by next page token

Here's an example of pagination from youtube API. NextPageToken field contains a token, which should be included in url to get next page. "&page_token=CAEQAA"

...

${url}
${url}&nextPageToken=${nextPageToken1}
${url}&nextPageToken=${nextPageToken2}
...

...

Anchor
custom_pagination custom_pagination

Custom pagination

Different APIs use very different styles of pagination. In the simple cases they return link in header or some field of response JSON.

• For example API where user wants to paginate by time in the following way: &start_time={something}&end_time={something+10000}. Two dependent variables are involved here. It would be very problematic to give ability to configure something like this via widget.  
• Let's images another case. User wants to download a webserver directory. So "pages" in this case are files on webserver. Let's say he analyses/backups a whole site. So we need to paginate based on results from parsing HTML.

• Let's assume another example. User wants to skip certain pages in API. Let's say the API pagination is time based, meaning something like this is appended to url "&start_time=1389075585". But he only wants to get pages for the weekends.

...

1 JSON format

JSON entries are converted into StructuredRecord using StructuredRecordStringConverter.java

To specify where we should take record from user needs to specify JSON Result Path.

Example:

...

1 JSON format

JSON entries are converted into StructuredRecord using StructuredRecordStringConverter.java

To specify where we should take record from user needs to specify JSON Result Path.

Example:

{
 "pageInfo": {
  "totalResults": 208,
  "resultsPerPage": 2
 },
 "items": [
  {
   "kind": "youtube#searchResult",
   "etag": "\"Bdx4f4ps3xCOOo1WZ91nTLkRZ_c/yrJNwvacPS7tA7BQCQmeIZr7fg8\"",
   "id": {
    "kind": "youtube#channel",
    "channelId": "UCfkRcekMTa5GA2DdNKba7Jg"
   },
   "snippet": {
    "publishedAt": "2015-02-12T22:12:43.000Z",
    "channelId": "UCfkRcekMTa5GA2DdNKba7Jg",
    "title": "Cask",
    "description": "Founded by developers for developers, Cask is an open source big data software company focused on developers. Cask's flagship offering, the Cask Data ...",
    "thumbnails": {
     ...
    },
    "channelTitle": "Cask",
    "liveBroadcastContent": "upcoming"
   }
  },
  {
   "kind": "youtube#searchResult",
   "etag": "\"Bdx4f4ps3xCOOo1WZ91nTLkRZ_c/uv6u8PSG0DsOqN9m77o06Jl4LnA\"",
   "id": {
    "kind": "youtube#video",
    "videoId": "ntOXeYecj7o"
   },
   "snippet": {
    "publishedAt": "2016-12-21T19:32:03.000Z",
    "channelId": "UCfkRcekMTa5GA2DdNKba7Jg",
    "title": "Cask Product Tour",
    "description": "In this video, we take you on a product tour of CDAP, CDAP extensions and the Cask Market. Cask Data Application Platform (CDAP) is the first Unified ...",
    "thumbnails": {
     ...
    },
    "channelTitle": "Cask",
    "liveBroadcastContent": "none"
   }
  }
 ]
}

...

We may add functionality for XML parsing to https://github.com/cdapio/cdap/blob/release/6.0/cdap-formats/src/main/java/io/cdap/cdap/format/ similarly to what we have for JSON. to separate project so other projects can re-use that.

XML below will be used as basis for examples in this section.

...

2.1 STEP 1 - Get XML by XPath

XML parsing is done by default Java DOM parser. Which is able to get items by a specified XPath. XPath is super flexible it allows user to get nodes by attribute value, as well as to group nodes from different parents into single result, as well as chose nodes conditionally etc. etc. 

Some XPath examples:

/bookstores/bookstore/book[position()<3]
//title[@lang]
//title[@lang='en']
/bookstores/bookstore/book/price[text()] # convert all subelements to string
/bookstores/bookstore/book[price>35.00]/title

...

{"bookstores": {"bookstore": [
    {
        "book": [
            {
                "year": 2005,
                "author": "Giada De Laurentiis",
                "price": {
                    "value": 15,
                    "policy": "Discount up to 50%"
                },
                "category": "cooking",
                "title": {
                    "lang": "en",
                    "content": "Everyday Italian"
                }
            },
            {
                "year": 2003,
                "author": [
                    "James McGovern",
                    "Per Bothner"
                ],
                "price": {
                    "value": 49.99,
                    "policy": "No discount"
                },
                "category": "web",
                "title": {
                    "lang": "en",
                    "content": "XQuery Kick Start"
       
        }             },
           
...         ],
        "id": 1
    },
    {
        ...
        "id": 2
    }
]}}

On a side note: look at author fields, they are of different type in the above JSON. This will be handled. If schema has field type = union, and there is a value not a list in place, we consider it as a list with a single element.

2.3 STEP 3 - Generate StructuredRecord from JSON

Converting JSON into Structured?ecord is a simple task. We do this via StructuredRecordStringConverter.java. Example:

{
    "year": 2003,
    "author": [
        "James McGovern",
        "Per Bothner"
    ],
    "price": {
        "value": 49.99,
        "policy": "No discount"
    },
    "category": "web",
    "title": {
        "lang": "en",
        "content": "XQuery Kick Start"
    }
}

will yield records compatible with below schema:

year: string
author: union
price: record
   - value:double
   - policy:string
category: string
title: record
   - lang:string
   - content:string

3 Delimited format

Will use the functionality from cdap-formats/DelimitedStringsRecordFormat.java to validate schema and convert input TCS/CSV to StructuredRecord.  The class supports a columns mapping as last field of schema. 

4 Text format

Will use the functionality from cdap-formats/TextRecordFormat.java  to validate schema and convert input text to StructuredRecord. 

OAuth2

1.1 General concepts

Before using OAuth2 usually user has to create an application via service site (e.g. Twitter) and register redirect uri, than receive a client_id and client_secret. Which will be used by an application during authentication.

The further steps are shown in the diagram below:

Image Removed

We are implementing grant types "Authorization Code Grant" and "Refresh Token". Other types are not suitable or rarely used. Click here for some context.

Properties:

...

CDAP will start a local server on the given url. Only localhost urls are allowed. For more info click here.

This is a URL where service callbacks with authCode after user enters username and password and agrees to grant the permissions.

This URL is also usually configured when registering the OAuth2 application in the service (e.g. Twitter). If the URL registered there is not equal to the one we send, OAuth2 will get denied.

...

A page, where the user is directed to enter his credentials.

Example: https://www.facebook.com/dialog/oauth

...

A page, where CDAP can exchange authCode for accessToken and refreshToken. Or refresh the accessToken.

Example: https://graph.facebook.com/v3.3/oauth/access_token

...

User should obtain this when registering the OAuth2 application in the service (e.g. Twitter).

...

This is optional.

Scope is a mechanism in OAuth 2.0 to limit an application's access to a user's account. An application can request one or more scopes, this information is then presented to the user in the consent screen, and the access token issued to the application will be limited to the scopes granted.

...

Note: OAuth2 implementation will reside in one of CDAP core modules, since this will need to be re-used by different plugins.

1.2 UI Changes Required. Need to expose an authentication dialog pop-up window to a user

We need to show authentication dialog from the service where user enters his username and passwords, as well as agrees to grant a certain access to our application.

This will require change to UI. We can implement this as plugin-function:

"plugin-function": {
  "method": "POST",
  "widget": "showPageToUser",
  "output-property": "void",
  "plugin-method": "showPageToUser"
}

...

1. Once the user clicks the button, UI runs showPageToUser to get url and headers from plugin using post-function. Example of return from plugin:

   Code Block
   url = https://www.facebook.com/v3.3/dialog/oauth? client_id=3MVG96_7YM2sI9wT6c13RTPp6RDeRBPFc0F5sYfIrKBZPdTK2Yr7jiTwq8u3ykXyBHtlf3lnNMWSN1rqfjA_y &redirect_uri=http://localhost:27435 headers = {} # empty

   On the side note: The url syntax is established by an RFC: https://tools.ietf.org/html/rfc6749#section-4.1.1

   IMPORTANT: During this call plugin will also start a callback server (for more information click here)

2. UI shows the page.
3. Once the page is closed, ui does another call to plugin.
   During this call plugin waits for a callback to be complete. Than exchanges received authCode to pair accessToken, refreshToken and return it back to UI.
4. UI populates the widget field refreshToken with the value. 

refreshToken usually has a permanent lifetime (unlike accessToken), unless invalidated manually by user. With this token during every pipeline run we can ask for an accessToken from tokenUrl. No need for callback server or UI/user involvement at this point.

If OAuth2 is enabled and refreshToken field is not populated. Pipeline deployment will fail. So effectively user will be asked to authenticate once and the information for further authentication will be saved in widget.

...

We will need to start an HTTP callback server on localhost. Let's say http://localhost:27435. The port should be statically configured via widget, we cannot get a random ephemeral port, since callback_url needs to be constant. It is saved on service provider (e.g. Facebook) as a static configuration of OAuth2 application.

After user enters his credentials on authentication page (of let's say Facebook), browser will redirect the response authCode to this server. Since this request is done by client browser (not remote server), this address is not required to be public IP address and we can safely use localhost.

1.4 Refreshing access tokens

Since we save refresh token. instead of access token token (which are short-lived). During every pipeline we will need to get an access token. This is a very simple process. We have to execute only one request. Which will return an access token.

POST /token HTTP/1.1
Host: server.example.com
Content-Type: application/x-www-form-urlencoded

grant_type=refresh_token&refresh_token=tGzv3JOkF0XG5Qx2TlKWIA
&client_id=s6BhdRkqt3&client_secret=7Fjfp0ZBr1KtDRbnfVdmIw

Facebook case

All the APIs I checked: Google APIs, PayPal, WordPress, Microsoft Azure, Okta support refreshing access token. Actually this is parf of RFC. The only API which does not is Facebook. Instead they use concept they have made up called fb_exchange_token. Here's more info. Since facebook is widely spread, I suggest we just add ugly check "if url contains facebook.com" (or talking in fancy Java terms create a factory class, which creates oauth2 handlers depending on url provided) than save long-lived-access token instead of refreshToken and do a refresh the way facebook wants. The factory can than be used to implement behavior for other services with non-default oauth2 implementations.

...

   },
            ...
        ],
        "id": 1
    },
    {
        ...
        "id": 2
    }
]}}

On a side note: look at author fields, they are of different type in the above JSON. This will be handled. If schema has field type = union, and there is a value not a list in place, we consider it as a list with a single element.

2.3 STEP 3 - Generate StructuredRecord from JSON

Converting JSON into Structured?ecord is a simple task. We do this via StructuredRecordStringConverter.java. Example:

{
    "year": 2003,
    "author": [
        "James McGovern",
        "Per Bothner"
    ],
    "price": {
        "value": 49.99,
        "policy": "No discount"
    },
    "category": "web",
    "title": {
        "lang": "en",
        "content": "XQuery Kick Start"
    }
}

will yield records compatible with below schema:

year: string
author: array
price: record
   - value:double
   - policy:string
category: string
title: record
   - lang:string
   - content:string

3 Delimited format

Will use the functionality from cdap-formats/DelimitedStringsRecordFormat.java to validate schema and convert input TCS/CSV to StructuredRecord.  The class supports a columns mapping as last field of schema. 

4 Text format

Will use the functionality from cdap-formats/TextRecordFormat.java  to validate schema and convert input text to StructuredRecord. 

OAuth2

Moved design information into a separate doc: Plugin OAuth2 Common Module

Properties:

SSL/TLS

Some general definitions for more context:

...

Should we provide an option for user to skip identity check during HTTPs connection? This is not recommended anywhere you read about it, but it might be useful in case user is testing some API which is in development stage.If url starts with "https" the plugin by default will try to use TLS.

...