Inconsistencies in decimal datatype handling between Wrangler and Pipelines

Description

Data types between BigQuery and Wrangler are not consistent. The BigQuery source table has a column of type NUMERIC but Wrangler pulls the value in as BIGDECIMAL which adds unwanted padding with zeros and a decimal point.

Furthermore, there are issues with converting string values back to integer/long even though those are the initial types of those values. Example error: java.lang.Exception: Stage:Wrangler - Reached error threshold 1, terminating processing due to error : Error encountered while executing 'set-type' : Column 'CNTRCT_LN_RECURRING_AMT' cannot be converted to a 'long'.

Release Notes

None

Assignee

Unassigned

Reporter

Bhooshan Mogal

Labels

Dev Complete Date

None

Publish Date

None

Reviewer

None

Components

Sprint

Fix versions

Priority

Major