Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] - sqlalchemy.exc.TimeoutError: QueuePool limit reached #4793

Open
5 of 6 tasks
floli opened this issue Dec 30, 2024 · 7 comments
Open
5 of 6 tasks

[BUG] - sqlalchemy.exc.TimeoutError: QueuePool limit reached #4793

floli opened this issue Dec 30, 2024 · 7 comments
Labels
bug Something isn't working triage

Comments

@floli
Copy link

floli commented Dec 30, 2024 •
edited
Loading

First Check

  • This is not a feature request.
  • I added a very descriptive title to this issue (title field is above this).
  • I used the GitHub search to find a similar issue and didn't find it.
  • I searched the Mealie documentation, with the integrated search.
  • I already read the docs and didn't find an answer.
  • This issue can be replicated on the demo site (https://demo.mealie.io/).

What is the issue you are experiencing?

During normal usage, two users, not a significant number of (simultaneous) requests, Mealie frequently stops serving requests and recovers after some minutes. Probably related issue is #4604, which focuses more on lock up due to high load. Number of total recipes is around 30.

Steps to Reproduce

Viewing the timeline seems to trigger it mostly, see the log.

Please provide relevant logs

mealie-1  | INFO     2024-12-30T13:37:11 - [172.19.0.1:0] 200 OK "GET /api/households/self HTTP/1.1"
mealie-1  | INFO     2024-12-30T13:37:11 - [172.19.0.1:0] 200 OK "GET /api/recipes/timeline/events?page=1&perPage=32&orderBy=timestamp&orderDirection=desc&queryFilter=(recipe.group_id%3D%22c2b1d4d1-871a-4f6b-85e8-dc7c5b6bb488%22)+AND+eventType+IN+[%22info%22,+%22system%22,+%22comment%22] HTTP/1.1"
mealie-1  | ERROR    2024-12-30T13:37:41 - Unknown Error on recipe controller action
mealie-1  | ERROR    2024-12-30T13:37:41 - Unknown Error on recipe controller action
mealie-1  | ERROR    2024-12-30T13:37:41 - QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)
mealie-1  | Traceback (most recent call last):
mealie-1  |   File "/app/mealie/routes/recipe/recipe_crud_routes.py", line 302, in get_one
mealie-1  |     recipe = self.service.get_one(slug)
mealie-1  |              ^^^^^^^^^^^^
mealie-1  |   File "/usr/local/lib/python3.12/functools.py", line 995, in __get__
mealie-1  |     val = self.func(instance)
mealie-1  | ERROR    2024-12-30T13:37:41 - QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)
mealie-1  | Traceback (most recent call last):
mealie-1  |   File "/app/mealie/routes/recipe/recipe_crud_routes.py", line 302, in get_one
mealie-1  |     recipe = self.service.get_one(slug)
mealie-1  |              ^^^^^^^^^^^^
mealie-1  |   File "/usr/local/lib/python3.12/functools.py", line 995, in __get__
mealie-1  |           ^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/app/mealie/routes/recipe/_base.py", line 53, in service
mealie-1  |     return RecipeService(self.repos, self.user, self.household, translator=self.translator)
mealie-1  |                                                 ^^^^^^^^^^^^^^
mealie-1  |   File "/app/mealie/routes/_base/base_controllers.py", line 164, in household
mealie-1  |     return self.repos.households.get_one(self.household_id)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/app/mealie/repos/repository_generic.py", line 163, in get_one
mealie-1  |     result = self.session.execute(q).unique().scalars().one_or_none()
mealie-1  |              ^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2362, in execute
mealie-1  |     return self._execute_internal(
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2237, in _execute_internal
mealie-1  |     conn = self._connection_for_bind(bind)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2106, in _connection_for_bind
mealie-1  |     return trans._connection_for_bind(engine, execution_options)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "<string>", line 2, in _connection_for_bind
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
mealie-1  |     ret_value = fn(self, *arg, **kw)
mealie-1  |                 ^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1189, in _connection_for_bind
mealie-1  |     conn = bind.connect()
mealie-1  |            ^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 3278, in connect
mealie-1  |     return self._connection_cls(self)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 146, in __init__
mealie-1  |     self._dbapi_connection = engine.raw_connection()
mealie-1  |                              ^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 3302, in raw_connection
mealie-1  |     return self.pool.connect()
mealie-1  |            ^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 449, in connect
mealie-1  |     return _ConnectionFairy._checkout(self)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 1263, in _checkout
mealie-1  |     fairy = _ConnectionRecord.checkout(pool)
mealie-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
mealie-1  |     rec = pool._do_get()
mealie-1  |           ^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
mealie-1  |     raise exc.TimeoutError(
mealie-1  | sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)
mealie-1  |     val = self.func(instance)
mealie-1  |           ^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/app/mealie/routes/recipe/_base.py", line 53, in service
mealie-1  |     return RecipeService(self.repos, self.user, self.household, translator=self.translator)
mealie-1  |                                                 ^^^^^^^^^^^^^^
mealie-1  |   File "/app/mealie/routes/_base/base_controllers.py", line 164, in household
mealie-1  |     return self.repos.households.get_one(self.household_id)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/app/mealie/repos/repository_generic.py", line 163, in get_one
mealie-1  |     result = self.session.execute(q).unique().scalars().one_or_none()
mealie-1  |              ^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2362, in execute
mealie-1  |     return self._execute_internal(
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2237, in _execute_internal
mealie-1  |     conn = self._connection_for_bind(bind)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2106, in _connection_for_bind
mealie-1  |     return trans._connection_for_bind(engine, execution_options)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "<string>", line 2, in _connection_for_bind
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
mealie-1  |     ret_value = fn(self, *arg, **kw)
mealie-1  |                 ^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1189, in _connection_for_bind
mealie-1  |     conn = bind.connect()
mealie-1  |            ^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 3278, in connect
mealie-1  |     return self._connection_cls(self)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 146, in __init__
mealie-1  |     self._dbapi_connection = engine.raw_connection()
mealie-1  |                              ^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 3302, in raw_connection
mealie-1  |     return self.pool.connect()
mealie-1  |            ^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 449, in connect
mealie-1  |     return _ConnectionFairy._checkout(self)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 1263, in _checkout
mealie-1  |     fairy = _ConnectionRecord.checkout(pool)
mealie-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
mealie-1  |     rec = pool._do_get()
mealie-1  |           ^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
mealie-1  |     raise exc.TimeoutError(
mealie-1  | sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)
mealie-1  | ERROR    2024-12-30T13:37:41 - Unknown Error on recipe controller action
mealie-1  | ERROR    2024-12-30T13:37:41 - Unknown Error on recipe controller action
mealie-1  | ERROR    2024-12-30T13:37:41 - QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)
mealie-1  | Traceback (most recent call last):
mealie-1  |   File "/app/mealie/routes/recipe/recipe_crud_routes.py", line 302, in get_one
mealie-1  |     recipe = self.service.get_one(slug)
mealie-1  |              ^^^^^^^^^^^^
mealie-1  |   File "/usr/local/lib/python3.12/functools.py", line 995, in __get__
mealie-1  |     val = self.func(instance)
mealie-1  |           ^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/app/mealie/routes/recipe/_base.py", line 53, in service
mealie-1  |     return RecipeService(self.repos, self.user, self.household, translator=self.translator)
mealie-1  |                                                 ^^^^^^^^^^^^^^
mealie-1  |   File "/app/mealie/routes/_base/base_controllers.py", line 164, in household
mealie-1  |     return self.repos.households.get_one(self.household_id)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/app/mealie/repos/repository_generic.py", line 163, in get_one
mealie-1  |     result = self.session.execute(q).unique().scalars().one_or_none()
mealie-1  |              ^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2362, in execute
mealie-1  |     return self._execute_internal(
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2237, in _execute_internal
mealie-1  |     conn = self._connection_for_bind(bind)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2106, in _connection_for_bind
mealie-1  |     return trans._connection_for_bind(engine, execution_options)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "<string>", line 2, in _connection_for_bind
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
mealie-1  |     ret_value = fn(self, *arg, **kw)
mealie-1  |                 ^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1189, in _connection_for_bind
mealie-1  |     conn = bind.connect()
mealie-1  |            ^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 3278, in connect
mealie-1  |     return self._connection_cls(self)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 146, in __init__
mealie-1  |     self._dbapi_connection = engine.raw_connection()
mealie-1  |                              ^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 3302, in raw_connection
mealie-1  |     return self.pool.connect()
mealie-1  |            ^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 449, in connect
mealie-1  |     return _ConnectionFairy._checkout(self)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 1263, in _checkout
mealie-1  |     fairy = _ConnectionRecord.checkout(pool)
mealie-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
mealie-1  |     rec = pool._do_get()
mealie-1  |           ^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
mealie-1  |     raise exc.TimeoutError(
mealie-1  | sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)
mealie-1  | ERROR    2024-12-30T13:37:41 - QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)
mealie-1  | Traceback (most recent call last):
mealie-1  |   File "/app/mealie/routes/recipe/recipe_crud_routes.py", line 302, in get_one
mealie-1  |     recipe = self.service.get_one(slug)
mealie-1  |              ^^^^^^^^^^^^
mealie-1  |   File "/usr/local/lib/python3.12/functools.py", line 995, in __get__
mealie-1  |     val = self.func(instance)
mealie-1  |           ^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/app/mealie/routes/recipe/_base.py", line 53, in service
mealie-1  |     return RecipeService(self.repos, self.user, self.household, translator=self.translator)
mealie-1  |                                                 ^^^^^^^^^^^^^^
mealie-1  |   File "/app/mealie/routes/_base/base_controllers.py", line 164, in household
mealie-1  |     return self.repos.households.get_one(self.household_id)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/app/mealie/repos/repository_generic.py", line 163, in get_one
mealie-1  |     result = self.session.execute(q).unique().scalars().one_or_none()
mealie-1  |              ^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2362, in execute
mealie-1  |     return self._execute_internal(
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2237, in _execute_internal
mealie-1  |     conn = self._connection_for_bind(bind)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2106, in _connection_for_bind
mealie-1  |     return trans._connection_for_bind(engine, execution_options)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "<string>", line 2, in _connection_for_bind
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
mealie-1  |     ret_value = fn(self, *arg, **kw)
mealie-1  |                 ^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1189, in _connection_for_bind
mealie-1  |     conn = bind.connect()
mealie-1  |            ^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 3278, in connect
mealie-1  |     return self._connection_cls(self)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 146, in __init__
mealie-1  |     self._dbapi_connection = engine.raw_connection()
mealie-1  |                              ^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 3302, in raw_connection
mealie-1  |     return self.pool.connect()
mealie-1  |            ^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 449, in connect
mealie-1  |     return _ConnectionFairy._checkout(self)
mealie-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 1263, in _checkout
mealie-1  |     fairy = _ConnectionRecord.checkout(pool)
mealie-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
mealie-1  |     rec = pool._do_get()
mealie-1  |           ^^^^^^^^^^^^^^
mealie-1  |   File "/opt/pysetup/.venv/lib/python3.12/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
mealie-1  |     raise exc.TimeoutError(
mealie-1  | sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)

Mealie Version

mealie:v2.4.1

Deployment

Docker (Linux)

Additional Deployment Details

Using traefik:latest as edge router / TLS termination.
@floli floli added bug Something isn't working triage labels Dec 30, 2024
@V-ed
Copy link

V-ed commented Jan 3, 2025 •
edited
Loading

Don't like doing +1 comments, but this happens to me as well, essentially every single time that I open the timeline.

Mealie frequently stops serving requests and recovers after some minutes.

Same behavior here as well : Mealie completely stops responding and makes it seem like the website is down. I find myself relatively tech savvy and got super confused when that happened before I started investigating, and trying to make a non-techy user use mealie means I have to warn them to essentially never touch this option until this is resolved.

It still works but is quite scary 😅

@bl4ckOut
Copy link

bl4ckOut commented Jan 4, 2025

Same for me on the newest version with exactly the same behaviour.

@PancakeZik
Copy link

Me too, I was wondering if there might be a simple way to disable the timeline function (just hiding on the frontend would do)

@kmobs
Copy link

kmobs commented Jan 23, 2025 •
edited
Loading

Timeline also causes my instance to crash with the same error. Funny enough can make it happen on the demo site as well.

@PancakeZik
Copy link

For myself, I solved it by migrating to postgres.
It was quite easy to migrate and everything is much snappier.

@kmobs
Copy link

kmobs commented Jan 23, 2025

What did you use to transfer your db? Any losses in the transfer process?

@PancakeZik
Copy link

What did you use to transfer your db? Any losses in the transfer process?

In summary, created a backup inside mealie, then started it up with the postgres setup and imported the backup.

No losses noticed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@kmobs @floli @bl4ckOut @V-ed @PancakeZik