We're updating the issue view to help you get more done. 

BQ sink validation should fail when setting both truncation and schema update

Description

BQ sink validation fails to catch the incompatible config setting: 

"

The Data Fusion BQ Multisink plugin has two configuration options: 'truncate table' and 'Update Table Schema' (ALLOW_FIELD_ADDITION).

Currently if set both 'Truncate Table' and 'Update Table Schema' to True, they conflict and during the BQ loading phase of a pipeline will fail with the following message:

"Could not commit Job: Error occurred while importing data to BigQuery 'Schema update options should only be specified with WRITE_APPEND disposition, or with WRITE_TRUNCATE disposition on a table partition.' 

"

 

Release Notes

'Truncate table' and 'update schema' options if set together, will apply only WRITE_TRUNCATE to BQ job.

Activity

Show:
Greeshma Swaminathan
June 24, 2020, 6:52 PM

The validation will not enforce choosing one of "Truncate table" or "Update Schema" as this may be confusing to the user who is expecting to clear schema and write. If both are selected together, only WRITE_TRUNCATE option will be set and not schema update options - this will ensure there is no exception while executing the job.

Greeshma Swaminathan
June 26, 2020, 1:27 AM

Validated on Jun 25th staging build.

Fixed

Assignee

Greeshma Swaminathan

Reporter

Wangyuan Zhang

Labels

None

Docs Impact

None

UX Impact

None

Fix versions

Priority

Major
Configure