Convert JSON schema to Google BigQuery schema
This includes the ability to:
- Create tables
- Patch tables
Further enhancements are planned: delete tables (dev only), create dataset, set data ACLs
Note that some features involve bespoke interpretation of schema details suited to our environment.
npm install jsonschema-bigquery
jsbq -p <gbq project> -d <gbq dataset> -j <json schema file> --preventAdditionalObjectProperties --continueOnError
For embedded usage the following will allow and support runtime schema conversion and table maintenance:
const jsonSchemaBigquery = require('jsonschema-bigquery')
const bigquerySchema = jsonSchemaBigquery.run(jsonSchemaObject, options)
Please ensure that the input JSON schema is dereferenced so that all external references have been resolved. json-schema-ref-parser can do this, prior to using this module.
{
preventAdditionalObjectProperties: true,
continueOnError: false
}
preventAdditionalObjectProperties- boolean, check for additional object properties in schemas.continueOnError- boolean, continues conversion if problem JSON is encountered. Problems will be excluded from resulting schema.
Google maintains a CLI tool called bq which allows the management of BigQuery tables:
https://cloud.google.com/bigquery/docs/reference/bq-cli-reference
To create a table from a JSON schema using other options that our jsbq does not support, use the following commands:
First, output the generated GBQ schema to a file:
npx jsbq -j test/integration/samples/complex/input.json > /tmp/schema.json
Then run the bq command to create a table:
bq mk --schema=/tmp/schema.json test_dataset.test_table
Please see the bq reference for table creation options:
https://cloud.google.com/bigquery/docs/reference/bq-cli-reference#bq_mk
npm test