Page Comparison

Table of Contents

...

As a framework Apache HttpComponents HttpClient is be used, a successor of Commons HttpClient.

It seems the most widely used/supported by community framework. It is very simple to find all kind of solutions and workaround already implemented, which makes plugin development and maintenance easy. Framework has a built in support for compession, https tunneling, digest auth and lot of other functions.

Properties:

Section

Name

Description

Default

Widget

Validations

General

URL

The url we will request. "{pagination.index}" can be included into url to represent a changing part needed for some pagination strategies.

E.g:

https://my.api.com/api/v1/user?maxResults=10&name=John&pageNumber={pagination_index}

Text Box

HTTP Method

Possible values:

GET
PUT
POST
DELETE
HEAD

GET

Radio group

Headers

Key-value map of headers

KeyValue

Dropdown


Request Body			Text Area	No validation [1]

Connect TimeoutRead Timeout

Maximum seconds to wait for data. (seconds)

0 - wait forever

120Text BoxIf is_ number and >=0

Maximum seconds to connect to server. (seconds)

0 - wait forever

120Text BoxIf is_ number and >=0

Error Handling

Error

HTTP Errors Handling

Per Status

This is a map in which user can define which error status codes produce which results. Possible values are: RETRY, FAIL, SKIP, SEND_TO_ERROR

.

, ALERT

Example:

500: RETRY

404: SEND_TO_ERROR

*: FAIL

Wildcard (*) means "otherwise" or "for all other codes do ..".

If the field is empty. Any status_code>=400 will yield a pipeline failure.

KeyValue Dropdown

If using SEND_TO_ERROR or SKIP or SEND_TO_ALERTS and current pagination type does not support it throw a validation error. [2]

Non-HTTP Error Handling

Handling of type casting and any other unhandled exceptions thrown during transformation of a record:

Possible values are:

"Skip on error" - ignores any errors
"Stop on error" - fails pipeline
"Send to error" - send to error handler

Stop on error

Dropdown list

If using "Send to error" or "Skip on error" and current pagination type does not support it throw a validation error. [2]

Retry

Type

Policy

Possible values are:

- Exponential
- Linear

Exponential

Radio group

Linear Retry Interval

The interval between retries (seconds)

30

Text Box

If is_ number and >=0

Number
Max retry

timeIf is_ number and >=0

duration

Max seconds it takes to do retries

600

Text Box

Number
Connect Timeout	Maximum seconds to connect to server. (seconds) 0 - wait forever	120	Number
Read Timeout	Maximum seconds to wait for data. (seconds) 0 - wait forever	120	Number
Basic authentication	Username	Used for basic authentication.		Text Box
Basic authentication	Password	Used for basic authentication.		Password
HTTP Proxy:	Proxy

URI

URL

Example: http://proxy.com:8080

Note for me: test this with https proxies.

Text Box

Username

Text Box

Password

[1] Unfortunately we cannot do validation here. Even though most commonly body in API requests is a JSON for JSON APIs or an XML for XML SOAP APIs. Theoretically it can be anything.

...

Name	Description	Default	Widget	Validations
Pagination type	Possible values are: None Link in response header Link in response body Increment an index Token in Response Body Custom	None	Dropdown listSelect	"Link in response body": Next Page field is set "Increment an index": {pagination.index} is in url, start index, increment are set. Custom: python code is set.
Start Index	Initial value for index which replaces {pagination.index} in url. See example here		Text Box	If set and pagination type is not "Increment an index", fail. If set and no {pagination.index} in url, fail. Assert if is_number
Max Index	Max value for index which replaces {pagination.index} in url. If this is empty, plugin will load pages until no results or 404 is returned. Also plugin may stop iteration before max index is reached, if no more records.		Text Box	If set and pagination type is not "Increment an index", fail. If set and no {pagination.index} in url, fail. Assert if is_number
Index Increment	Increment value for index which replaces {pagination.index} in url.		Text Box	If set and pagination type is not "Increment an index", fail. If set and no {pagination.index} in url, fail. Assert if is_number
Next Page JSON/XML Field Path	Link to a field in JSON or an XML containing next page url. See an example here		Text Box	If set and pagination type is not "Link in response body", fail. If the content type is not XML or JSON, fail.
Next Page token AddressToken	Link to a field in JSON or an XML containing next page token.			If set and pagination type is not "Token in Response Body", fail. If the content type is not XML or JSON, fail.
Custom Pagination Python Code	A code fragment which determines how next page url is generated and also when to finish iteration. For more info see Custom Pagination		Python code	If set and pagination type is not "Custom" fail.
Wait time between pages	The number of milliseconds to wait before requesting the next page.	1000	Text BoxNumber	Assert if is_number and > 0. If not set and Pagination type is non 'None' fail.

The above is a bit messy cause we cannot dynamically change the content of widget depending on pagination type. Which makes it a mix of properties for different pag_types. Is not super user-friendly for end-user. For now I will a placeholder which says which pagination type property coresponds to.

Pagination type is none

Plugin will request a single page.

...

2.1 STEP 1 - Get XML by XPath

XML parsing is done by default Java DOM parser. Which is able to get items by a specified XPath. XPath is super flexible it allows user to get nodes by attribute value, as well as to group nodes from different parents into single result, as well as chose nodes conditionally etc. etc.

Some XPath examples:

Code Block
/bookstores/bookstore/book[position()<3] //title[@lang] //title[@lang='en'] /bookstores/bookstore/book/price[text()] # convert all subelements to string /bookstores/bookstore/book[price>35.00]/title

...

Name	Description	Default	Widget	Validations
OAuth2 Enabled	True or false.	false	Radio group
Auth URL	A page, where the user is directed to enter his credentials. Example: https://www.facebook.com/dialog/oauth		Text Box	Assert to be empty if OAuth2 is disabled and the not empty if enabled.
Token URL	A page, where CDAP can exchange authCode for accessToken and refreshToken. Or refresh the accessToken. Example: https://graph.facebook.com/v3.3/oauth/access_token		Text Box	Assert to be empty if OAuth2 is disabled and the not empty if enabled.
Client ID	User should obtain this when registering the OAuth2 application in the service (e.g. Twitter).		Text Box	Assert to be empty if OAuth2 is disabled and the not empty if enabled.
Client Secret	User should obtain this when registering the OAuth2 application in the service (e.g. Twitter).		Password	Assert to be empty if OAuth2 is disabled and the not empty if enabled.
Scope	This is optional. Scope is a mechanism in OAuth 2.0 to limit an application's access to a user's account. An application can request one or more scopes, this information is then presented to the user in the consent screen, and the access token issued to the application will be limited to the scopes granted.		Text Box	Assert to be empty if OAuth2 is disabled.
Refresh Token	This is populated by the button "Login via OAuth 2.0". Since we save Refresh Token (not an access token which is short lived), this should be done only once, during initial pipeline deployment. For more information click here. UI should put an actual value into secure store and put macro function ${secure(key)} a value for extra safety.			Fail is empty and OAuth2 is enabled.

SSL/TLS

...

Some general definitions for more context:
...
Name Description Default Widget Validations
Keystore File Path to a keystore file
Text Box Check if file exists
Keystore Type
According to Oracle docs. There are 3 supported keystore types.
Possible values:
Java KeyStore (JKS)
Java Cryptography Extension KeyStore (JCEKS)
PKCS #12
JKS Radio Group
Keystore Password Leave empty if keystore is not password protected
Password Try to load keystore with given password
Keystore Key Algorithm SunX509 is default in Java. SunX509 Text Box
TrustStore File Path to a truststore file. If empty use default Java truststores.
Text Box Check if file exists
TrustStore Type
According to Oracle docs. There are 3 supported truststore types.
Possible values:
Java KeyStore (JKS)
Java Cryptography Extension KeyStore (JCEKS)
PKCS #12
JKS Radio Group
TrustStore Password Leave empty if keystore is not password protected
Password Try to load truststore with given password
Truststore Trust Algorithm
SunX509 Text Box
Transport Protocols User can add multiple protocols. Which will be offered by client during handshake. TLSv1.2 Array Validate if names are correct
Cipher Suites
User can add multiple cipher suites. They will be offered by client during handshake.
If empty use default cipher suites.
This is textBox with comma separated list of ciphers. Since sometimes there can be 20, 30 or more ciphers it is not usable for user to add every one of them manually into an array.

Text Box
Validate if supported by current java implementation
...

Versions Compared

Old Version 69

New Version 70

Key

Pagination type is none

2.1 STEP 1 - Get XML by XPath

SSL/TLS

Name	Description	Default	Widget	Validations
Keystore File	Path to a keystore file		Text Box	Check if file exists
Keystore Type	According to Oracle docs. There are 3 supported keystore types. Possible values: Java KeyStore (JKS) Java Cryptography Extension KeyStore (JCEKS) PKCS #12	JKS	Radio Group
Keystore Password	Leave empty if keystore is not password protected		Password	Try to load keystore with given password
Keystore Key Algorithm	SunX509 is default in Java.	SunX509	Text Box
TrustStore File	Path to a truststore file. If empty use default Java truststores.		Text Box	Check if file exists
TrustStore Type	According to Oracle docs. There are 3 supported truststore types. Possible values: Java KeyStore (JKS) Java Cryptography Extension KeyStore (JCEKS) PKCS #12	JKS	Radio Group
TrustStore Password	Leave empty if keystore is not password protected		Password	Try to load truststore with given password
Truststore Trust Algorithm		SunX509	Text Box
Transport Protocols	User can add multiple protocols. Which will be offered by client during handshake.	TLSv1.2	Array	Validate if names are correct
Cipher Suites	User can add multiple cipher suites. They will be offered by client during handshake. If empty use default cipher suites. This is textBox with comma separated list of ciphers. Since sometimes there can be 20, 30 or more ciphers it is not usable for user to add every one of them manually into an array.		Text Box	Validate if supported by current java implementation