...
Compute config
Pipeline config
Engine config
Resources
Pipeline alert
ELT Transformation pushdown
Compute config
You can change the compute profile you want to use to run this pipeline. For example, you want to run the pipeline against an existing Dataproc cluster rather than the default Dataproc cluster.
...
Property | Description |
|---|---|
Run Condition | When to run the action. Must be completion, success, or failure. Defaults to completion. If set to completion, the action will be performed regardless of whether the pipeline run succeeds or fails. If set to success, the action will only be performed if the pipeline run succeeds. If set to failure, the action will only be performed if the pipeline run fails. |
URL | Required. The URL to fetch data from. |
HTTP Method | Required. The http request method. Choose from: DELETE, GET, HEAD, OPTIONS, POST, PUT. Defaults to POST. |
Request body | The http request body. |
Number of Retries | The number of times the request should be retried if the request fails. Defaults to 0 |
Should Follow Redirects | Whether to automatically follow redirects. Defaults to true. |
Request Headers | Request headers to set when performing the http request. |
Generic - Connection Timeout | Sets the connection timeout in milliseconds. Set to 0 for infinite. Default is 60000 (1 minute). |
...
Transformation pushdown
Starting in CDAP 6.5.0, you can enable ELT Transformation pushdown to have BigQuery process joins. For more information, see Using ELT Transformation pushdown.