BigQuery replication plugin does not handle datetime correctly
Description
Release Notes
Fix issue in BigQuery replication plugin which caused pipeline to fail when replicating datetime from sources that are more precise than microsecond, for example datetime2 data type in sql server.
clones
Activity
Show:

Sumit Jain October 21, 2022 at 12:07 PM
Fixed
Pinned fields
Click on the next to a field label to start pinning.
Created October 21, 2022 at 12:05 PM
Updated November 29, 2022 at 5:24 AM
Resolved October 26, 2022 at 4:29 PM
The CDAP
datetime
logical type allows setting aLocalDateTime
java object, which goes up to nanosecond precision. This is more precise than BigQuerydatetime
, which only goes up to microsecond precision ( ).This causes issues when reading datetimes from sources that are more precise than microsecond. For example, sql server type
datetime2
has 7 digits of precision, which will cause the BigQuery load jobs to fail with errors like: