Best way to transfer a large amount of data from a US dataset to an EU dataset in BigQuery?

Question

I have many TBs of approximately 1 million tables in a single BigQuery project located in several datasets located in the USA. I need to transfer all this data to datasets hosted in the EU. What is my best option for this?

• I would export tables to Google Cloud Storage and reimport using boot tasks, but the load limit is 10 thousand per project per day
• I would do this as w / "allow big results" queries and save the destination table, but this does not work in the cross scope

The only option I see now is to re-insert all the data using the BQ streaming API, which will be prohibitive.

What is the best way to move large amounts of data in many cross-section tables in BigQuery?

Answer 1

You have several options:

• Use download tasks and contact Google Cloud to request a quota exemption. They will probably provide 100 thousand or so on a temporary basis (if not, contact me tigani@google, and I can do it).
• Use jobs with combined queries. That is, move the data to a GCS bucket in the EU, and then re-import the data through BigQuery queries with GCS data sources. More details here .

I will also consider whether we can increase this quota limit in all directions.

Answer 2

, BigQuery Copy Dataset (/-). . " " . . , .

Answer 3

, , . :

1. Cloud Composer GCS. .
2. Cloud Dataflow GCS. .

Disclaimer: I wrote an article for option 2 (using Cloud Dataflow).