Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Invalid attempt to access database with closed connection #1651

Merged
merged 5 commits into from
Oct 24, 2023

Conversation

rapsealk
Copy link
Member

@rapsealk rapsealk commented Oct 24, 2023

This PR is a follow-up of #835

This PR moves a database accessing code to the valid context manager scope.

2023-10-24 17:51:54.064 DEBUG ai.backend.manager.plugin.error_monitor [82177] collected an error log [ERROR] "sqlalchemy.exc.ResourceClosedError: This Connection is closed" from manager
2023-10-24 17:51:54.064 ERROR ai.backend.manager.server [82177] Uncaught exception in HTTP request handlers ResourceClosedError('This Connection is closed')
Traceback (most recent call last):
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/src/ai/backend/manager/server.py", line 232, in exception_middleware
    resp = await handler(request)
    ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/src/ai/backend/manager/server.py", line 219, in api_middleware
    resp = await _handler(request)
      ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/src/ai/backend/manager/api/auth.py", line 574, in auth_middleware
    return await handler(request)
      ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/dist/export/python/virtualenvs/python-default/3.11.6/lib/python3.11/site-packages/aiotools/func.py", line 23, in wrapped
    return await coro(*args, *cargs, **kwargs, **ckwargs)
    ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/src/ai/backend/manager/api/ratelimit.py", line 73, in rlim_middleware
    response = await handler(request)
    ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/dist/export/python/virtualenvs/python-default/3.11.6/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
    ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/src/ai/backend/manager/api/auth.py", line 581, in wrapped
    return await handler(request, *args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/src/ai/backend/manager/api/manager.py", line 63, in wrapped
    return await handler(request, *args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/src/ai/backend/manager/api/vfolder.py", line 2313, in delete_by_name
    await _delete(
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/src/ai/backend/manager/api/vfolder.py", line 2227, in _delete
    await ensure_host_permission_allowed(
      ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/src/ai/backend/manager/models/vfolder.py", line 835, in ensure_host_permission_allowed
    allowed_hosts = await filter_host_allowed_permission(
      ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/src/ai/backend/manager/models/vfolder.py", line 858, in filter_host_allowed_permission
    allowed_hosts_by_user = await get_allowed_vfolder_hosts_by_user(
    ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/src/ai/backend/manager/models/vfolder.py", line 564, in get_allowed_vfolder_hosts_by_user
    if values := await conn.scalar(query):
      ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/dist/export/python/virtualenvs/python-default/3.11.6/lib/python3.11/site-packages/sqlalchemy/ext/asyncio/engine.py", line 478, in scalar
    result = await self.execute(statement, parameters, execution_options)
    ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/dist/export/python/virtualenvs/python-default/3.11.6/lib/python3.11/site-packages/sqlalchemy/ext/asyncio/engine.py", line 453, in execute
    result = await greenlet_spawn(
    ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/dist/export/python/virtualenvs/python-default/3.11.6/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 115, in greenlet_spawn
    result = context.switch(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/dist/export/python/virtualenvs/python-default/3.11.6/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
    ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/dist/export/python/virtualenvs/python-default/3.11.6/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
    ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/dist/export/python/virtualenvs/python-default/3.11.6/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
    ret = self._execute_context(
    ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/dist/export/python/virtualenvs/python-default/3.11.6/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1808, in _execute_context
    conn = self._revalidate_connection()
    ^^^^^^^^^^^^^^^^^
  File "/Users/rapsealk/Desktop/git/backend.ai-dev2/dist/export/python/virtualenvs/python-default/3.11.6/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 655, in _revalidate_connection
    raise exc.ResourceClosedError("This Connection is closed")
    ^^^^^^^^^^^^^^^^^
sqlalchemy.exc.ResourceClosedError: This Connection is closed

Checklist: (if applicable)

  • Milestone metadata specifying the target backport version
  • Mention to the original issue
  • Documentation
  • API server-client counterparts (e.g., manager API -> client SDK)
  • Test case(s) to demonstrate the difference of before/after

@rapsealk rapsealk added this to the 23.09 milestone Oct 24, 2023
@rapsealk rapsealk self-assigned this Oct 24, 2023
@github-actions github-actions bot added comp:manager Related to Manager component size:S 10~30 LoC labels Oct 24, 2023
@achimnol achimnol added type:bug Reports about that are not working urgency:5 It is imperative that action be taken right away. labels Oct 24, 2023
@achimnol achimnol added urgency:blocker IT SHOULD BE RESOLVED BEFORE NEXT RELEASE! and removed urgency:5 It is imperative that action be taken right away. labels Oct 24, 2023
@kyujin-cho kyujin-cho added this pull request to the merge queue Oct 24, 2023
Merged via the queue into main with commit d9a669c Oct 24, 2023
13 of 24 checks passed
@kyujin-cho kyujin-cho deleted the fix/invalid-attempt-to-use-closed-connection branch October 24, 2023 12:17
@kyujin-cho kyujin-cho modified the milestones: 23.09, 24.03 Oct 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:manager Related to Manager component size:S 10~30 LoC type:bug Reports about that are not working urgency:blocker IT SHOULD BE RESOLVED BEFORE NEXT RELEASE!
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants