Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test BQ schema changes in CI #317

Open
fbertsch opened this issue Apr 22, 2019 · 1 comment
Open

Test BQ schema changes in CI #317

fbertsch opened this issue Apr 22, 2019 · 1 comment
Assignees

Comments

@fbertsch
Copy link
Contributor

This is a bit of a difficult task, but the request is basically the following:

  1. Create BQ schema from before-commit and after-commit
  2. Create table in BQ with before-commit schema (bq mk)
  3. Update table in BQ with after-commit schema (bq update)
  4. Insert the sample data from validation directory

None of these steps should fail, and the validation data should be fully accessible.

@fbertsch
Copy link
Contributor Author

@jklukas I think this issue is more akin to what you mentioned in #316. Long-term it would be lovely if we could do some integration tests on schema updates:

  • Validate some landfill data + validation data against the JSON schema
  • Generate Avro data using the ingestion Beam job
  • Check the schema update (as above)
  • Load the avro data into the updated BQ table

This is a long-term goal though and will eventually probably be it's own proposal. For now let's keep this as checking for BQ compatible schema updates only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants