Exceptions when shutting down with no model loaded

[awatuna]

OS

Linux

GPU Library

CUDA 12.x

Python version

3.12

Describe the bug

If you start the server without a model, or unload the model first before stopping the server, it will throw exceptions, with or without uvloop.

Reproduction steps

start tabby without model loaded, or use api to unload model first before shutting down

Expected behavior

graceful shutdown

Logs

with uvloop

INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8088(Press CTRL+C to quit)
WARNING:  Immediately terminating all jobs. Clients will have their requests cancelled.

INFO:     Model unloaded.
INFO:     192.168.20.11:56638 - "POST /v1/model/unload HTTP/1.1" 200
^CINFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [9330]
WARNING:  Shutdown signal called. Exiting gracefully.
ERROR:    Task exception was never retrieved
ERROR:    future: \<Task finished name='Task-5' coro=\<signal_handler_async() done, defined at /home/llm/apps/tabbyapi/common/signals.py:28>
exception=SystemExit(0)>
ERROR:    Traceback (most recent call last):
ERROR:      File "uvloop/loop.pyx", line 474, in uvloop.loop.Loop._on_idle
ERROR:      File "uvloop/cbhandles.pyx", line 83, in uvloop.loop.Handle._run
ERROR:      File "uvloop/cbhandles.pyx", line 61, in uvloop.loop.Handle._run
ERROR:      File "/home/llm/apps/tabbyapi/common/signals.py", line 38, in signal_handler_async
ERROR:        sys.exit(0)
ERROR:    SystemExit: 0
Traceback (most recent call last):
  File "/home/llm/miniforge3/envs/tabby/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/llm/miniforge3/envs/tabby/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1511, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1504, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1377, in uvloop.loop.Loop.run_forever
  File "uvloop/loop.pyx", line 555, in uvloop.loop.Loop._run
  File "uvloop/loop.pyx", line 474, in uvloop.loop.Loop._on_idle
  File "uvloop/cbhandles.pyx", line 83, in uvloop.loop.Handle._run
  File "uvloop/cbhandles.pyx", line 61, in uvloop.loop.Handle._run
  File "/home/llm/apps/tabbyapi/common/signals.py", line 38, in signal_handler_async
    sys.exit(0)
SystemExit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/llm/apps/tabbyapi/main.py", line 171, in <module>
    entrypoint()
  File "/home/llm/apps/tabbyapi/main.py", line 167, in entrypoint
    asyncio.run(entrypoint_async())
  File "/home/llm/miniforge3/envs/tabby/lib/python3.12/asyncio/runners.py", line 193, in run
    with Runner(debug=debug, loop_factory=loop_factory) as runner:
  File "/home/llm/miniforge3/envs/tabby/lib/python3.12/asyncio/runners.py", line 62, in __exit__
    self.close()
  File "/home/llm/miniforge3/envs/tabby/lib/python3.12/asyncio/runners.py", line 72, in close
    loop.run_until_complete(
  File "uvloop/loop.pyx", line 1515, in uvloop.loop.Loop.run_until_complete
RuntimeError: Event loop stopped before Future completed.

without uvloop

INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8088 (Press CTRL+C to quit)
WARNING:  Immediately terminating all jobs. Clients will have their requests cancelled.

INFO:     Model unloaded.
INFO:     192.168.20.11:34298 - "POST /v1/model/unload HTTP/1.1" 200
^CINFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [9526]
WARNING:  Shutdown signal called. Exiting gracefully.
Traceback (most recent call last):
  File "/home/llm/miniforge3/envs/tabby/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/llm/miniforge3/envs/tabby/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/llm/miniforge3/envs/tabby/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/home/llm/miniforge3/envs/tabby/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/home/llm/miniforge3/envs/tabby/lib/python3.12/asyncio/base_events.py", line 1986, in _run_once
    handle._run()
  File "/home/llm/miniforge3/envs/tabby/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/home/llm/apps/tabbyapi/common/signals.py", line 38, in signal_handler_async
    sys.exit(0)
SystemExit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/llm/apps/tabbyapi/main.py", line 171, in <module>
    entrypoint()
  File "/home/llm/apps/tabbyapi/main.py", line 167, in entrypoint
    asyncio.run(entrypoint_async())
  File "/home/llm/miniforge3/envs/tabby/lib/python3.12/asyncio/runners.py", line 193, in run
    with Runner(debug=debug, loop_factory=loop_factory) as runner:
  File "/home/llm/miniforge3/envs/tabby/lib/python3.12/asyncio/runners.py", line 62, in __exit__
    self.close()
  File "/home/llm/miniforge3/envs/tabby/lib/python3.12/asyncio/runners.py", line 72, in close
    loop.run_until_complete(
  File "/home/llm/miniforge3/envs/tabby/lib/python3.12/asyncio/base_events.py", line 685, in run_until_complete
    raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.

Additional context

Add an admin api to do graceful shutdown?

And is there any way to speed up the model loading for quick swap of models?
Ie keeping loaded models in ram, copy to vram on demand.

Loading a fully cached models is about 2GB/s to the vram, bottlenecked by cpu single thread performance by the looks of it.

Acknowledgements

• I have looked for similar issues before submitting this one.
• I have read the disclaimer, and this issue is related to a code bug. If I have a question, I will use the Discord server.
• I understand that the developers have lives and my issue will be answered when possible.
• I understand the developers of this program are human, and I will ask my questions politely.

[bdashore3]

This is not a bug. These errors pop up due to the SystemExit call which cancels all pending tasks when a user calls to shut down the program. As such, I'm removing the bug status.

Agreed, there should be a way to see what's happening here and how to silence these errors since the user wants to shut down.

[bdashore3]

Fixed in d5e4285