[red-knot] Property tests

[sharkdp]

Summary

This is something I played with, wondering if it could be useful. It found a few bugs so far:

• [red-knot] Minor: fix Literal[True] <: int #14177
• [red-knot] Fix intersection simplification for ~Any/~Unknown #14195
• [red-knot] Fix is_assignable_to for unions #14196
• [red-knot] Fix is_disjoint_from for class literals #14210

If we want to merge this, there are a few things to do / to consider:

• The tests are extremely slow at the moment (> 30 ms per iteration in debug mode; > 3 ms in release mode) . The reason for this is our incredible naive way of setting up the TestDb in every iteration. This is hopefully easy to fix by caching the TestDb. The tests are now three orders of magnitude faster.
• Even if this resolved, we probably don't want to run those tests alongside the normal unit tests. For one, they can be arbitrary slow, depending on the number of iterations. And second, they are not deterministic. Even if we fix both of these issues (e.g. make the tests opt-in via a special config + fix random seeds), I'm not sure we would want them to run in CI? I'm afraid they would be a constant source of flakiness because even after fixing seeds, their behavior might change as soon as we make changes to the (test) code. One option could be to only run them offline from time to time, using a special config.
• There are a lot of "spurious" failures that turn out to be caused:
  • insufficient understanding of equality of types (we currently don't understand that int | str is equal to str | int, there is an open TODO in the code)
  • no "propagation" of Never. For example: tuple[Never, int] is equivalent to Never, but we currently don't simplify this anywhere.
  • other limitations of our current code, like the fact that is_disjoint_from/is_subtype_of can produce false negative answers.
• The current shrinking implementation is very naive, which leads to counterexamples that are very long (str & Any & ~tuple[Any] & ~tuple[Unknown] & ~Literal[""] & ~Literal["a"] | str & int & ~tuple[Any] & ~tuple[Unknown]), requiring the developer to simplify manually
• The code needs improvements

Test Plan

QUICKCHECK_TESTS=1000 cargo test --release -p red_knot_python_semantic property_tests

[sharkdp]

If we want to move forward with this, we could probably auto-generate this boilerplate in each test using a macro.

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

[sharkdp]

There must be a better way to do this. If only we had the author of this library on our team...

[MichaReiser]

CC: @BurntSushi

[BurntSushi]

I think this looks reasonable to me?

[carljm]

This is awesome! Thank you for doing this, I've been wanting to explore this. I think property testing is very well suited to testing type relation invariants, and I do think we should move forward with actually landing this.