ERROR ai.backend.manager.api.admin ADMIN.GQL Exception

[d4zzz]

What Operating System(s) are you seeing this problem on?

Linux (x86-64) Ubuntu-22.04

Backend.AI version

23.9.6

Describe the bug

When the manager service start, it produce the bug log. It seems that the GQL query param compute_session_list is None.

2024-01-02 14:23:55.583 INFO aiohttp.access [18159] 10.10.60.2 [02/Jan/2024:06:23:55 +0000] "POST /admin/graphql HTTP/1.1" 200 593 "-" "Backend.AI Web Server 23.09.6"
2024-01-02 14:23:55.597 DEBUG ai.backend.manager.api.utils [18159] stripped raw params: {'query': 'query($limit:Int!, $offset:Int!, $ak:String, $group_id:String, $status:String) {\n      compute_session_list(limit:$limit, offset:$offset, access_key:$ak, group_id:$group_id, status:$status) {\n        total_count\n      }\n    }', 'variables': {'limit': 1, 'offset': 0, 'status': 'RUNNING'}}
2024-01-02 14:23:55.617 ERROR ai.backend.manager.api.admin [18159] ADMIN.GQL Exception: {'message': "'NoneType' object has no attribute '_apply_params_to_element'", 'locations': [{'line': 2, 'column': 7}], 'path': ['compute_session_list']}
2024-01-02 14:23:55.627 DEBUG ai.backend.manager.plugin.error_monitor [18159] collected an error log [ERROR] "ai.backend.manager.api.exceptions.GraphQLError: GraphQL-generated error.
 -> extra_data: [{'message': "'NoneType' object has no attribute '_apply_params_to_element'", 'locations': [{'line': 2, 'column': 7}], 'path': ['compute_session_list']}]" from manager

the traceback info :

2024-01-05 10:39:23.244 ERROR graphql.execution.utils [15066] Traceback (most recent call last):
  File "/home/bai/manager/.venv/lib/python3.11/site-packages/promise/promise.py", line 844, in handle_future_result
    resolve(future.result())
            ^^^^^^^^^^^^^^^
  File "/home/bai/manager/.venv/lib/python3.11/site-packages/aiomonitor/monitor.py", line 487, in _coro_wrapper
    return await coro
           ^^^^^^^^^^
  File "/home/bai/manager/.venv/lib/python3.11/site-packages/ai/backend/manager/models/base.py", line 844, in wrapped
    return await resolve_func(executor, info, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bai/manager/.venv/lib/python3.11/site-packages/ai/backend/manager/models/gql.py", line 1647, in resolve_compute_session_list
    items = await ComputeSession.load_slice(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bai/manager/.venv/lib/python3.11/site-packages/ai/backend/manager/models/session.py", line 1519, in load_slice
    return [cls.from_row(ctx, r) async for r in (await db_sess.stream(query))]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bai/manager/.venv/lib/python3.11/site-packages/ai/backend/manager/models/session.py", line 1519, in <listcomp>
    return [cls.from_row(ctx, r) async for r in (await db_sess.stream(query))]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bai/manager/.venv/lib/python3.11/site-packages/sqlalchemy/ext/asyncio/result.py", line 174, in __anext__
    row = await greenlet_spawn(self._onerow_getter, self)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bai/manager/.venv/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 128, in greenlet_spawn
    result = context.switch(value)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/bai/manager/.venv/lib/python3.11/site-packages/sqlalchemy/engine/result.py", line 457, in onerow
    row = self._fetchone_impl()
          ^^^^^^^^^^^^^^^^^^^^^
  File "/home/bai/manager/.venv/lib/python3.11/site-packages/sqlalchemy/engine/result.py", line 1340, in _fetchone_impl
    return self._real_result._fetchone_impl(hard_close=hard_close)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bai/manager/.venv/lib/python3.11/site-packages/sqlalchemy/engine/result.py", line 1743, in _fetchone_impl
    row = next(self.iterator, _NO_ROW)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bai/manager/.venv/lib/python3.11/site-packages/sqlalchemy/orm/loading.py", line 158, in chunks
    post_load.invoke(context, path)
  File "/home/bai/manager/.venv/lib/python3.11/site-packages/sqlalchemy/orm/loading.py", line 1335, in invoke
    loader(context, path, states, self.load_keys, *arg, **kw)
  File "/home/bai/manager/.venv/lib/python3.11/site-packages/sqlalchemy/orm/strategies.py", line 2985, in _load_for_path
    loadopt._generate_extra_criteria(context),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bai/manager/.venv/lib/python3.11/site-packages/sqlalchemy/orm/strategy_options.py", line 154, in _generate_extra_criteria
    return k2._apply_params_to_element(k1, and_(*self._extra_criteria))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
graphql.error.located_error.GraphQLLocatedError: 'NoneType' object has no attribute '_apply_params_to_element'

2024-01-05 10:39:23.245 ERROR ai.backend.manager.api.admin [15066] ADMIN.GQL Exception: {'message': "'NoneType' object has no attribute '_apply_params_to_element'", 'locations': [{'line': 2, 'column': 7}], 'path': ['compute_session_list']}

To Reproduce

sudo systemctl restart backend.ai-manager.service

Expected Behavior

Anything else?

No response