BigQuery plugins temporary bucket location enhancements

Description

The BigQuery plugins use a temporary GCS bucket for intermediate storage. If one is not given, it will auto-create a bucket. However, the auto-created bucket is located in the US, which causes problems if the BQ table is in the EU, resulting in errors like:

java.io.IOException: Cannot read and write in different locations: source: EU, destination: US

The plugins should create buckets in the same zone as the BQ table when auto-creating.

If a bucket is given, they should also validate that the locations are valid to fail at pipeline deployment instead of at runtime.

Release Notes

Fixed the BigQuery plugins to create temporary GCS buckets in the same location as the BigQuery dataset.

Activity

Show:
Albert Shau
May 23, 2019, 12:07 AM
Fixed

Assignee

Albert Shau

Reporter

Albert Shau

Labels

Docs Impact

None

UX Impact

None

Components

Fix versions

Priority

Major
Configure