Table of Contents
| Table of Contents |
|---|
...
As a framework Apache HttpComponents HttpClient is be used, a successor of Commons HttpClient.
It seems the most widely used/supported by community framework. It is very simple to find all kind of solutions and workaround already implemented, which makes plugin development and maintenance easy. Framework has a built in support for compession, https tunneling, digest auth and lot of other functions.
...
| Section | Name | Description | Default | Widget | Validations |
|---|---|---|---|---|---|
| General | URL | The url we will request. "{pagination.index}" can be included into url to represent a changing part needed for some pagination strategies. E.g: https://my.api.com/api/v1/user?maxResults=10&name=John&pageNumber={pagination_index} | Text Box | Validate it contains protocol. | |
| HTTP Method | Possible values:
| GET | Radio group | ||
| Headers | Key-value map of headers | KeyValue | |||
| Request Body | Text Area | No validation [1] | |||
| Error Handling | HTTP Errors Handling | This is a map in which user can define which error status codes produce which results. Possible values are: RETRY, FAIL, SKIP, SEND_TO_ERROR, ALERT Example: 500: RETRY 404: SEND_TO_ERROR *: FAIL Wildcard (*) means "otherwise" or "for all other codes do ..". If the field is empty. Any status_code>=400 will yield a pipeline failure. | KeyValue Dropdown | If using SEND_TO_ERROR or SKIP or SEND_TO_ALERTS and current pagination type does not support it throw a validation error. [2] | |
| Non-HTTP Error Handling | Handling of type casting and any other unhandled exceptions thrown during transformation of a record: Possible values are:
| Stop on error | Dropdown list | If using "Send to error" or "Skip on error" and current pagination type does not support it throw a validation error. [2] | |
| Retry Policy | Possible values are:
| Exponential | Radio group | ||
| Linear Retry Interval | The interval between retries (seconds) | 30 | Number |
| |
| Max retry duration | Max seconds it takes to do retries | 600 | Number | ||
| Connect Timeout | Maximum seconds to connect to server. (seconds) 0 - wait forever | 120 | Number | ||
| Read Timeout | Maximum seconds to wait for data. (seconds) 0 - wait forever | 120 | Number | ||
| Basic authentication | Username | Used for basic authentication. | Text Box | ||
| Password | Used for basic authentication. | Password | |||
| HTTP Proxy: | Proxy URL | Example: http://proxy.com:8080 Note for me: test this with https proxies. | Text Box | ||
| Username | Text Box | ||||
| Password | Password |
...
| Section | Name | Description | Default | Widget | Validations | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Format | Format | Possible values:
| Dropdown list | JSON/XML Result Path | For JSON a simple slash separated path is used e.g. /library/books/items. For XML an XPath is used. | Text Box | Fail if used with non JSON/XML formatlist | |||||||||||
| JSON/XML Result Path | For JSON a simple slash separated path is used e.g. /library/books/items. For XML an XPath is used. | Text Box | Fail if used with non JSON/XML format | |||||||||||||||
| JSON/XML Fields Mapping | Mapping of schema field name to jsonPath (past the result path). Example (Jira API):
Schema fields which are not in the map, will use fieldName:/fieldName mapping. |
|
1 JSON format
JSON entries are converted into StructuredRecord using StructuredRecordStringConverter.java
...
2.1 STEP 1 - Get XML by XPath
XML parsing is done by default Java DOM parser. Which is able to get items by a specified XPath. XPath is super flexible it allows user to get nodes by attribute value, as well as to group nodes from different parents into single result, as well as chose nodes conditionally etc. etc.
Some XPath examples:
| Code Block |
|---|
/bookstores/bookstore/book[position()<3] //title[@lang] //title[@lang='en'] /bookstores/bookstore/book/price[text()] # convert all subelements to string /bookstores/bookstore/book[price>35.00]/title |
...
| Name | Description | Default | Widget | Validations |
|---|---|---|---|---|
| OAuth2 Enabled | True or false. | false | Radio group | |
| Auth URL | A page, where the user is directed to enter his credentials. Example: https://www.facebook.com/dialog/oauth | Text Box | Assert to be empty if OAuth2 is disabled and the not empty if enabled. | |
| Token URL | A page, where CDAP can exchange authCode for accessToken and refreshToken. Or refresh the accessToken. | Text Box | Assert to be empty if OAuth2 is disabled and the not empty if enabled. | |
| Client ID | User should obtain this when registering the OAuth2 application in the service (e.g. Twitter). | Text Box | Assert to be empty if OAuth2 is disabled and the not empty if enabled. | |
| Client Secret | User should obtain this when registering the OAuth2 application in the service (e.g. Twitter). | Password | Assert to be empty if OAuth2 is disabled and the not empty if enabled. | |
| Scope | This is optional. Scope is a mechanism in OAuth 2.0 to limit an application's access to a user's account. An application can request one or more scopes, this information is then presented to the user in the consent screen, and the access token issued to the application will be limited to the scopes granted. | Text Box | Assert to be empty if OAuth2 is disabled. | |
| Refresh Token | This is populated by the button "Login via OAuth 2.0". Since we save Refresh Token (not an access token which is short lived), this should be done only once, during initial pipeline deployment. For more information click here. UI should put an actual value into secure store and put macro function ${secure(key)} a value for extra safety. | Fail is empty and OAuth2 is enabled. |
...
| Name | Description | Default | Widget | Validations |
|---|---|---|---|---|
| Verify HTTPs Trust Certificates | If false will allow connection to untrusted https sources. | true | ||
| Keystore File | Path to a keystore file | Text Box | Check if file exists | |
| Keystore Type | According to Oracle docs. There are 3 supported keystore types. Possible values:
| JKS | Radio Group | |
| Keystore Password | Leave empty if keystore is not password protected | Password | Try to load keystore with given password | |
| Keystore Key Algorithm | SunX509 is default in Java. | SunX509 | Text Box | |
| TrustStore File | Path to a truststore file. If empty use default Java truststores. | Text Box | Check if file exists | |
| TrustStore Type | According to Oracle docs. There are 3 supported truststore types. Possible values:
| JKS | Radio Group | |
| TrustStore Password | Leave empty if keystore is not password protected | Password | Try to load truststore with given password | |
| Truststore Trust Algorithm | SunX509 | Text Box | ||
| Transport Protocols | User can add multiple protocols. Which will be offered by client during handshake. | TLSv1.2 | Array | Validate if names are correct |
| Cipher Suites | User can add multiple cipher suites. They will be offered by client during handshake. If empty use default cipher suites. This is textBox with comma separated list of ciphers. Since sometimes there can be 20, 30 or more ciphers it is not usable for user to add every one of them manually into an array. | Text Box | Validate if supported by current java implementation |
...