Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: get() method not releasing GIL #324

Open
awelzel opened this issue Mar 27, 2023 · 0 comments
Open

Python: get() method not releasing GIL #324

awelzel opened this issue Mar 27, 2023 · 0 comments

Comments

@awelzel
Copy link
Contributor

awelzel commented Mar 27, 2023

I think this came up in Slack before. When calling subscriber.get() and blocking, other Python threads do not continue to run. The broker bindings should release the Python GIL before blocking.

https://pybind11.readthedocs.io/en/stable/advanced/misc.html#global-interpreter-lock-gil

import threading
import time

import broker

def poll():
    while True:
        print(time.time(), "thread polling")
        time.sleep(0.3)

def main():
    t = threading.Thread(target=poll, daemon=True)
    t.start()
    time.sleep(2)

    ep = broker.Endpoint()
    ep.listen("127.0.0.1", 60000)
    with ep.make_status_subscriber(True) as ss:
        while True:
            print(time.time(), "ep.get() blocks thread from running")
            status = ss.get()

            print(time.time(), "got a status")
            time.sleep(1)

if __name__ == '__main__':
    main()

The output is as follows, showing how the thread polling stops once ep.get() is called. Connecting another peer shortly allows to do more polling as there's a time.sleep() which yields the GIL when sleeping.

$ PYTHONPATH=.:/opt/zeek-dev-prod/lib/zeek/python python3 threaded.py 
1679943976.7454154 thread polling
1679943977.0458524 thread polling
1679943977.3463733 thread polling
1679943977.6468718 thread polling
1679943977.947265 thread polling
1679943978.2476246 thread polling
1679943978.548081 thread polling
1679943978.7587862 ep.get() blocks thread from running
1679943983.725927 thread polling
1679943983.7260008 got a status
1679943984.0264978 thread polling
1679943984.327149 thread polling
1679943984.6278195 thread polling
1679943984.727265 ep.get() blocks thread from running
1679943984.727514 got a status
1679943984.928495 thread polling
1679943985.2289653 thread polling
1679943985.5294564 thread polling
1679943985.7281764 ep.get() blocks thread from running


^Cterminate called without an active exception
Aborted (core dumped)
@awelzel awelzel changed the title Python: get() method not releasing GIL (?) Python: get() method not releasing GIL Mar 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants