Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve locking of certain dicts (e.g. __globals__) #30

Open
JustAMan opened this issue Jun 7, 2016 · 0 comments
Open

Improve locking of certain dicts (e.g. __globals__) #30

JustAMan opened this issue Jun 7, 2016 · 0 comments

Comments

@JustAMan
Copy link

JustAMan commented Jun 7, 2016

From my profiling experience of x.py (as shown at the sprint at PyCon'16) a lot of time was spent in locking the global dictionary due to contention.

Implementing a RW lock for an arbitraty dictionary is really hard because one would have to somehow implement a RW lock which can recurse in any way (e.g. R+W+R, R+R+W, etc.) due to the fact that comparison operator for dictionary keys can execute just anything (example might be an object which has overriden __eq__ method which inserts something in the same dictionary it was being used as a key - weird but possible).

But my profiling also suggests that most problems (at least in certain benchmark cases) are caused by contention on dictionaries of certain type, that is, on dictionaries with strings as keys.
Also when looking at dictobject.c implementation we can see that Python dicts have a special case for those exact same dicts which does not call custom compare, but rather compares memory directly.

That said, I think it's possible to implement more efficient locking for those objects using RW locks, and lock all other dicts as before.

I'll try prototyping that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant