We're updating the issue view to help you get more done. 

BigQuery source does not delete the temporary tables created while exporting data to GCS

Description

BigQuery source exports data to temporary tables in the source dataset and then from these temporary tables to the GCS bucket. This workflow is followed (i.e. temporary table creation in source dataset) when one of the following configurations in source is used:

1. partition
2. filter
3. view

For view we need to create temporary table because of the limitation of bigquery view from where data cant be exported directly to GCS. However we need to check if exporting data to temporary tables is really required for partition and filter cases.

Also these temporary tables should be deleted as soon as data is lands in GCS otherwise they can cost users.

Release Notes

None

Activity

Show:
Bajram Bojku
December 4, 2020, 8:33 PM

Flakrim Jusufi
December 30, 2020, 8:57 AM

PR:

Flakrim Jusufi
3 days ago

Cherrypicks:

release/0.14 -->
release/0.15 -->
release/0.16 -->

Assignee

Ajdin Idrizi

Reporter

Sagar Kapare

Labels

None

Dev Complete Date

None

Publish Date

None

Reviewer

None

Components

Sprint

Cirus Sprint 4

Priority

Critical
Configure