Page Comparison

The url we will request. "{pagination.index}" can be included into url to represent a changing part needed for some pagination strategies.

E.g:

https://my.api.com/api/v1/user?maxResults=10&name=John&pageNumber={pagination_index}

Possible values:

• GET
• PUT
• POST
• DELETE
• HEAD

This is a map in which user can define which error status codes produce which results. Possible values are: RETRY, FAIL, SKIP, SEND_TO_ERROR, ALERT

Example:

500: RETRY

404: SEND_TO_ERROR

*: FAIL

Wildcard (*) means "otherwise" or "for all other codes do ..".

If the field is empty. Any status_code>=400 will yield a pipeline failure.

If using SEND_TO_ERROR or SKIP or SEND_TO_ALERTS and current pagination type does not support it throw a validation error. [2]

Handling of type casting and any other unhandled exceptions thrown during transformation of a record:

Possible values are:

• "Skip on error" - ignores any errors
• "Stop on error" - fails pipeline
• "Send to error" - send to error handler

Possible values are:

• • Exponential
  • Linear

Exponential

• if not set and retryPolicy is linear, fail.

Maximum seconds to connect to server. (seconds)

0 - wait forever

Maximum seconds to wait for data. (seconds)

0 - wait forever

Example: http://proxy.com:8080

Note for me: test this with https proxies.

Pagination type

Possible values are:

• None
• Link in response header
• Link in response body
• Increment an index
• Token in Response Body
• Custom

• "Link in response body": Next Page field is set
• "Token in Response Body": "Next Page Token Path" and "Next Page Url Parameter" are set.
• "Increment an index": {pagination.index} is in url, start index, increment are set.
• Custom: python code is set.

• If set and pagination type is not "Increment an index", fail.
• If set and no {pagination.index} in url, fail.
• Assert if is_number

Max value for index which replaces {pagination.index} in url.

If this is empty, plugin will load pages until no results or 404 is returned.

Also plugin may stop iteration before max index is reached, if no more records.

• If set and pagination type is not "Increment an index", fail.
• If set and no {pagination.index} in url, fail.
• Assert if is_number

• If set and pagination type is not "Increment an index", fail.
• If set and no {pagination.index} in url, fail.
• Assert if is_number

Link to a field in JSON or an XML containing next page url. See an example here

• If set and pagination type is not "Link in response body", fail.
• If the content type is not XML or JSON, fail.

• If set and pagination type is not "Token in Response Body", fail.
• If the content type is not XML or JSON, fail.
• Validate to have at least one element

• If set and pagination type is not "Token in Response Body", fail.

• Assert if is_number and > 0.
• If not set and Pagination type is non 'None' fail.

Possible values:

• JSON
• XML
• TSV
• CSV
• Text
• Blob

For JSON a simple slash separated path is used e.g. /library/books/items.

For XML an XPath is used.

For JSON a simple slash separated path is used e.g. /library/books/items.

For XML an XPath is used.

Mapping of schema field name to jsonPath (past the result path).

Example (Jira API):

Schema fields which are not in the map, will use fieldName:/fieldName mapping.

• if key is not present in schema fail
• if used for non JSON/XML fail

A page, where the user is directed to enter his credentials.

Example: https://www.facebook.com/dialog/oauth

A page, where CDAP can exchange authCode for accessToken and refreshToken. Or refresh the accessToken.

Example: https://graph.facebook.com/v3.3/oauth/access_token

User should obtain this when registering the OAuth2 application in the service (e.g. Twitter).

This is optional.

Scope is a mechanism in OAuth 2.0 to limit an application's access to a user's account. An application can request one or more scopes, this information is then presented to the user in the consent screen, and the access token issued to the application will be limited to the scopes granted.

This is populated by the button "Login via OAuth 2.0". Since we save Refresh Token (not an access token which is short lived), this should be done only once, during initial pipeline deployment. For more information click here.

UI should put an actual value into secure store and put macro function ${secure(key)} a value for extra safety.

According to Oracle docs. There are 3 supported keystore types.

Possible values:

• Java KeyStore (JKS)
• Java Cryptography Extension KeyStore (JCEKS)
• PKCS #12
  

According to Oracle docs. There are 3 supported truststore types.

Possible values:

• Java KeyStore (JKS)
• Java Cryptography Extension KeyStore (JCEKS)
• PKCS #12

User can add multiple cipher suites. They will be offered by client during handshake.

If empty use default cipher suites.

This is textBox with comma separated list of ciphers. Since sometimes there can be 20, 30 or more ciphers it is not usable for user to add every one of them manually into an array.

Text Box