-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Release GIL as much as possible in PySeries and PyDataFrame methods #19382
Comments
The ideal would be to have the code calling into Rayon for the Polars release the GIL, so that it would only need to happen once in the code. |
This is plausibly doable by having conditional creation of POOL in |
No, we should just release the GIL on the edge, not deep in Polars on every threadpool access. We should just go to the chore on time and then we're done. |
I am happy to do it the manual widespread small tweaks way, it's just that it feels like a systemic invariant, and putting the burden of a systemic invariant on every single caller (and there are many!) is a recipe for mistakes. Like, someone adds 3 methods in 4 months, and now there's a new place with the problem. Whereas doing it in one place where it's always done by default gives you a systemic solution to a systemic invariant. The performance cost would be a But again, will do it manual update-all-the-functions way if you prefer. |
Description
Motivation:
pl.Object
series #19358 for an example, but I have seen other cases where this happens.The text was updated successfully, but these errors were encountered: