Skip to content

Commit

Permalink
feature: Implement Shape::widen which enables widening a Shape to…
Browse files Browse the repository at this point in the history
… fit a provided `AsNode`. This sets the groundwork for performantly keeping track of a running inferred schema in the combiner.

Of note, the schemas "inferred" by `widen()` are maximally strict:
* By default, newly inferred objects have have `additionalProperties: false`.
* Object fields initially have `required: true` until we encounter a document missing that field, at which point it will be downgraded to `required: false`

The next piece of work is implementing the stubbed-out `enforce_field_count_limits`. With that, we should have everything we need to implement the running inferred schema and emit it to the ops logs. Also of note is that the `reduce: flow-inferred-schema-merge` reduction annotation implementation can and should also use `enforce_field_count_limits`, as it's possible for itra-transaction documents to not exceed the limits while inter-transaction documents do, and we care about limiting both of those cases.
  • Loading branch information
jshearer committed Jul 26, 2023
1 parent 9cab979 commit 7196176
Show file tree
Hide file tree
Showing 4 changed files with 430 additions and 2 deletions.
23 changes: 23 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,8 @@ tonic-build = "0.9"

warp = "0.3.3"

pretty_assertions = "1.4.0"

[profile.release]
incremental = true
debug = 0 # Set this to 1 or 2 to get more useful backtraces in debugger.
Expand Down
3 changes: 2 additions & 1 deletion crates/doc/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,9 @@ rand_distr = { workspace = true }
serde-transcode = { workspace = true }
serde_json = { workspace = true }
serde_yaml = { workspace = true }
pretty_assertions = { workspace = true }

[features]
default = ["combine"]

combine = ["lz4"]
combine = ["lz4"]
Loading

0 comments on commit 7196176

Please sign in to comment.