BigQuery replication plugin does not handle datetime correctly

Description

The CDAP datetime logical type allows setting a LocalDateTime java object, which goes up to nanosecond precision. This is more precise than BigQuery datetime, which only goes up to microsecond precision ( ).

This causes issues when reading datetimes from sources that are more precise than microsecond. For example, sql server type datetime2 has 7 digits of precision, which will cause the BigQuery load jobs to fail with errors like:

Release Notes

Fix issue in BigQuery replication plugin which caused pipeline to fail when replicating datetime from sources that are more precise than microsecond, for example datetime2 data type in sql server.

Activity

Show:

Sumit Jain October 21, 2022 at 12:07 PM

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Affects versions

Triaged

Yes

Components

Fix versions

Priority

Created October 21, 2022 at 12:05 PM
Updated November 29, 2022 at 5:24 AM
Resolved October 26, 2022 at 4:29 PM