BentoML and the GIL

[jondoering]

Hi,

I was wondering, how bentoML is working with the GIL when you have multiple workers /threads - is there one share GIL or are independent workers running in true subprocesses and there is no scalability problem with the GIL? Would be great to hear some insights.
Thanks,
Jonathan

[parano]

Hi @jondoering, there is no scalability issue with GIL in BentoML. BentoML uses gunicorn to manage multiple processes of model backend workers. In BentoML, the frontend micro-batching layer(the MarshalServer in the code) is built with asyncio and aiohttp. It sends batched requests to the model backend, which runs a Flask app in a Gunicorn server.

[jondoering]

Hi @parano Thanks for your fast reply. So if I understand correctly - every worker will run its own python-process with its own GIL?

[bojiang]

GIL matters only when using multi-threads to handle incoming requests. Bentoml uses coroutines to handle&hold requests and processes to perform inference tasks, there is no GIL problem on both sides. And it is far more efficient than the multi-threaded model.