We're updating the issue view to help you get more done. 

BQ sink: should skip schema validation when truncate table is set

Description

Problem:

BQ sink validate output table schema even though truncating table is set, while ideally (please verify) the validation should be skipped when truncating the table. 

 

Bug:

We[1] should skip this whole section [1] when  [2] truncating is true

 [1]https://github.com/data-integrations/google-cloud/blob/bf09d9d4b77c11629ecc1ed61904b4b65a6f3297/src/main/java/io/cdap/plugin/gcp/bigquery/sink/AbstractBigQuerySink.java#L306

[2] https://github.com/data-integrations/google-cloud/blob/bf09d9d4b77c11629ecc1ed61904b4b65a6f3297/src/main/java/io/cdap/plugin/gcp/bigquery/sink/AbstractBigQuerySinkConfig.java#L77

 

 

 

Release Notes

Removed schema validation from BQ sink when 'truncate table' option is set.

Activity

Show:
Wangyuan Zhang
June 8, 2020, 6:35 PM

com/google/cloud/bigquery/LoadConfiguration.java:200

comment from BQ java client. 

/**

  • [Experimental] Returns options allowing the schema of the destination table to be updated as a

  • side effect of the load job. Schema update options are supported in two cases: when

  • writeDisposition is WRITE_APPEND; when writeDisposition is WRITE_TRUNCATE and the destination

  • table is a partition of a table, specified by partition decorators. For normal tables,

  • WRITE_TRUNCATE will always overwrite the schema.
    */
    List<SchemaUpdateOption> getSchemaUpdateOptions();

 

Greeshma Swaminathan
June 26, 2020, 1:30 AM

Validated on staging on Jun 25th rc10.

Fixed

Assignee

Greeshma Swaminathan

Reporter

Wangyuan Zhang

Labels

None

Docs Impact

None

UX Impact

None

Fix versions

Priority

Major
Configure