Schema inference: Shape::widen()
#1126
Merged
Commits on Aug 4, 2023
-
feature: Implement
Shape::widenwhich enables widening aShapeto…… fit a provided `AsNode`. This sets the groundwork for performantly keeping track of a running inferred schema in the combiner. Of note, the schemas "inferred" by `widen()` are maximally strict: * By default, newly inferred objects have have `additionalProperties: false`. * Object fields initially have `required: true` until we encounter a document missing that field, at which point it will be downgraded to `required: false` The next piece of work is implementing the stubbed-out `enforce_field_count_limits`. With that, we should have everything we need to implement the running inferred schema and emit it to the ops logs. Also of note is that the `reduce: flow-inferred-schema-merge` reduction annotation implementation can and should also use `enforce_field_count_limits`, as it's possible for itra-transaction documents to not exceed the limits while inter-transaction documents do, and we care about limiting both of those cases.
-
* Factor out `ObjShape::widen` * Refactor to be zero-cost by default * Ensure ObjShape.properties stays sorted * Get rid of the helper function `Shape::widen_inner` * Set `is_required` for new fields properly based on whether they have always been present or not
-
-
* Infer string formats following similar logic to `is_required`: the first string gets inferred, subsequent ones get checked, and after a non-conforming string causes the format to drop off, don't re-infer it * Reduce some nesting in `ObjShape::widen` * Widen array min and max length * Correctly detect `integer`, and `fractional` types * Tests
-
-
fix: Recur in
Shape::enforce_field_count_limits()even when we're a……lready in a location that's getting squashed because we also need to ensure that the newly-squashed `additionalProperties` isn't _also_ excessively large
-
-
-
-
Update logic to handle
patternProperties, and to widen explicit pro……perties first, even if `additionalProperties` is defined.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.