Replication pipeline configured with more than 1000 tables fails with BigQuery "maximum number of tables per query" quota issue.

Description

When replicating more than 1000 tables we hit quota limit from the BQ - "Maximum number of tables referenced per query — 1,000"

It seems like query from BigQuery target to find the maximum sequence number from all existing bq tables is causing this and it can be batched.

Release Notes

Fixed an issue in Replication that caused jobs to fail if more than 1000 tables are selected for replication.

Activity

Show:

Robin Rielley July 15, 2021 at 6:40 PM

Thanks Sagar. This one is done!

Sagar Kapare July 15, 2021 at 6:09 PM

minor nit:

“Fixed an issue in Replication that caused jobs to fail if more than 1000 tables are selected for replication.”

 

Thanks Robin!

Robin Rielley July 15, 2021 at 5:47 PM

This looks like we might want to add it to the 6.5.0 release notes. How about “Fixed an issue in Replication that caused jobs to fail is the source had more than 1000 tables.”

Sanket Sahu July 14, 2021 at 7:58 PM

Fix PR :
Cherrypick PR :

Sagar Kapare April 9, 2021 at 6:37 PM

Workaround is to split the tables into multiple replication pipelines so that each pipeline process no more than 1000 tables.

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Sanket Sahu

Reporter

Affects versions

Components

Fix versions

Priority

Created April 9, 2021 at 5:56 PM
Updated July 30, 2021 at 4:34 PM
Resolved July 14, 2021 at 11:22 PM
Loading...