The BigQuery plugins use a temporary GCS bucket for intermediate storage. If one is not given, it will auto-create a bucket. However, the auto-created bucket is located in the US, which causes problems if the BQ table is in the EU, resulting in errors like:
java.io.IOException: Cannot read and write in different locations: source: EU, destination: US
The plugins should create buckets in the same zone as the BQ table when auto-creating.
If a bucket is given, they should also validate that the locations are valid to fail at pipeline deployment instead of at runtime.
Release Notes
Fixed the BigQuery plugins to create temporary GCS buckets in the same location as the BigQuery dataset.
The BigQuery plugins use a temporary GCS bucket for intermediate storage. If one is not given, it will auto-create a bucket. However, the auto-created bucket is located in the US, which causes problems if the BQ table is in the EU, resulting in errors like:
java.io.IOException: Cannot read and write in different locations: source: EU, destination: US
The plugins should create buckets in the same zone as the BQ table when auto-creating.
If a bucket is given, they should also validate that the locations are valid to fail at pipeline deployment instead of at runtime.