Columns with null data leads to pipeline failure due to error in wrangler parsing
Description
Scenario - Batch pipeline - GCS --> Wrangler(SQL precondition) ---> Wrangler (a non SQL stage) --> Wrangler(SQL precondition) ---> BQ
Preview error :
Pipeline execution error:
It appears that Wrangler is trying to parse values in phone column to type of age column.
BigQuery schema for the input table which has null data for last_name, profession
Pipeline json
Release Notes
None
Attachments
3
Activity
Show:
Vanathi Ganeshraj November 9, 2023 at 7:37 AM
Fixed as part of schema changes done in
Sagar Ahire March 31, 2023 at 4:11 PM
Just to note: the null columns being dropped from the schema is a known issue documented in .
Sagar Ahire March 31, 2023 at 4:10 PM
On enabling Transformation Pushdown, the pipeline fails with the following error:
Notably the error is thrown in the BQ sink this time and not in Wrangler.
Divya B March 31, 2023 at 1:07 PM
Edited
I removed the preconditions step in both Wrangler plugin and ran preview. I had to remove preconditions step because of the known bug that it fails in Preview. Preview ran fine and data is mapped right until the sink.
Pipeline succeeded and data was mapped right in the BQ as well.
This issue seems to be a bug introduced in Filter pushdown. I am attaching the pipeline json(SS4-cdap-data-pipeline-updated.json) that ran successfully.
Fixed
Pinned fields
Click on the next to a field label to start pinning.
Scenario - Batch pipeline - GCS --> Wrangler(SQL precondition) ---> Wrangler (a non SQL stage) --> Wrangler(SQL precondition) ---> BQ
Preview error :
Pipeline execution error:
It appears that Wrangler is trying to parse values in phone column to type of age column.
BigQuery schema for the input table which has null data for last_name, profession
Pipeline json