Geek Answers Handbook

Getting Bigquery to automatically detect schemas

I am trying to load a csv file and BigQuery automatically creates a schema.

bq load --source_format CSV -F '\t' --skip_leading_rows=1 voterdb.voters gs://[[ redacted bucket ]]/extract.csv BigQuery error in load operation: Error processing job 'tokyo-unity-87516:bqjob_r3682474e46ce720f_0000015421aee065_1': No schema specified on job or table.

According to https://cloud.google.com/bigquery/federated-data-sources , automatic creation / output of the scheme is supported.

+7
google-bigquery
source share
3 answers

As indicated in the previous answer, automatic schema discovery is part of the queries to the federated data source. This is not explicitly stated in the CSV section and the JSON automatic detection scheme that this does not apply to bq load . If you feel that this is unclear in the documentation, I would strongly suggest clicking Send Feedback in the upper right corner of the documentation page and describe this ambiguity in detail.

As for the bq load command, according to the bq load documentation , the table schema is a required parameter. Omitting this, you will receive an error message that you encounter.

EDIT Thanks polleyg for the update. In this blog post, it was announced that the circuit should also be detected at boot time. As Chris Sears noted, the --autodetect flag should meet your needs.

+3
source share

The --autodetect flag is probably what you want. It works for JSON input files for CSV and (with a line extension).

For example...

 bq load --source_format=NEWLINE_DELIMITED_JSON --autodetect yourdataset.yourtable inputfile.json

See here: https://cloud.google.com/bigquery/bq-command-line-tool#creatingtablefromfile

Note that this has nothing to do with join data sources.

+4
source share

The link you are linking to - BigQuery supports automatic schema creation / output when you directly query a data source.
The error you get is on load from GCS

+1
source share

More articles:


All Articles