If you write to a BigQuery table that is in a different zone than the dataproc cluster, the pipeline fails with:
The job is not found because it has id lbg-obd:europe-west2.direct-bigqueryhelper-import-a77c79e4-4e07-4373-b8dd-3a905940876c and not lbg-obd:direct-bigqueryhelper-import-a77c79e4-4e07-4373-b8dd-3a905940876c
Moving the datasets and temp storage to the same zone as the dataproc profile causes this issue to go away, but that is clearly not a good solution. I would assume changing the zone for the dataproc profile would also be a workaround.
Release Notes
Fixed a bug in the BigQuery sink that would cause pipelines to fail when writing to a dataset in a different region.
If you write to a BigQuery table that is in a different zone than the dataproc cluster, the pipeline fails with:
The job is not found because it has id
lbg-obd:europe-west2.direct-bigqueryhelper-import-a77c79e4-4e07-4373-b8dd-3a905940876c
and not
lbg-obd:direct-bigqueryhelper-import-a77c79e4-4e07-4373-b8dd-3a905940876c
Moving the datasets and temp storage to the same zone as the dataproc profile causes this issue to go away, but that is clearly not a good solution. I would assume changing the zone for the dataproc profile would also be a workaround.