Skip to content

Commit

Permalink
feat: Add check for schema read compatibility (delta-io#554)
Browse files Browse the repository at this point in the history
## What changes are proposed in this pull request?
This PR introduces the `schema::compare` module to handle schema
compatibility comparisons. This module introduces the `SchemaComparison`
trait that adds the `can_read_as` method. This checks that for data
written with schema `schema`, whether it can be read with the
`read_schema`. This check is useful for implementing schema evolution
checks in CDF.

Closes delta-io#523 
## How was this change tested?
Schema compatibility tests are added that check the following: 
- `can_read_as` is reflexive
- adding a nullable column to each the key and value of a map succeeds.
- changing a map value from nullable to non-nullable fails
- same schema with different field name case fails
- changing column type from long to integer fails.
- Setting nullability from false to true succeeds
- Setting nullability from true to false fails
- Adding a nullable column succeeds
- Adding a non-nullable column fails

---------

Co-authored-by: Oussama Saoudi <[email protected]>
Co-authored-by: OussamaSaoudi <[email protected]>
  • Loading branch information
3 people authored Jan 23, 2025
1 parent d999b5c commit 6751838
Show file tree
Hide file tree
Showing 2 changed files with 403 additions and 0 deletions.
Loading

0 comments on commit 6751838

Please sign in to comment.