Schema nullability is ignored in BigQuery Sink plugin

Description

We observed a couple of issues with single root-cause:

Case 1:
BigQuery table does not exist, BigQuery Sink always creates table with nullable fields, though some fields from source might have been required.
Source schema: a int not null
BigQuery schema: a int

Case 2:
BigQuery table exists and has at least one column which is required. Does not matter which nullability such column has in source, pipeline fails.
Source schema: a int not null
BigQuery schema: a int not null

Some basic STR:

  • Create BigQuery Table (see schema sample)

  • Create pipeline that use this schema (see attached sample)

  • Execute and found that BiqQuery sink plugin ignore nullable checkbox in configured schema (see attached log)

Release Notes

Fixed in the scope of CDAP-15235.

Attachments

3

relates to

Activity

Show:
Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Components

Priority

Created April 24, 2019 at 4:06 PM
Updated July 17, 2020 at 12:28 AM
Resolved May 8, 2019 at 6:43 PM