We're updating the issue view to help you get more done. 

Update/Upsert operations in BigQuery sink should support only specifying mandatory columns in the input schema

Description

Let's say I have a BigQuery table with fields f1, f2 and f3, out of which f1 and f2 are nullable, and f3 is not.

I should be able to have a BigQuery sink in my pipeline that updates this table, but only has the field f3 in its input schema. Validation should not fail if optional fields are absent in the schema, and the pipeline should succeed at execution time also.

Release Notes

None

Activity

Show:
Ajdin Idrizi
December 17, 2020, 2:43 PM

tested in cdap sandbox 6.3 6.2 and 6.1,

  1. The target table had three columns Identifier, First_name,Last-name(optional)

  2. File source avro with output schema with only Identifier and First_name

  3. Press validate in BQ sink and it was successful

  4. Run those pipelines and it worked fine

Vinisha Shah
December 28, 2020, 6:20 PM
  1. Was bigquery table already created on Bigquery side while reproducing the issue?

  2. Also, was the operation on bigquery sink “Update“ OR “Upsert“?

Flakrim Jusufi
December 29, 2020, 11:20 AM

 

  1. The bigquery table was already created on bigquery side.

  2. We tested this operation with both Update and Upsert and we verified that both operations are working fine.

Tested in cdap sandbox 6.3.0, 6.1.4 and 6.2.3

Assignee

Ajdin Idrizi

Reporter

Bhooshan Mogal

Labels

None

Product Requirement Doc

None

Reviewer

None

Dev Complete Date

None

Publish Date

None

Docs Impact

None

UX Impact

None

Sprint

Cirus Sprint 4

Priority

Major
Configure