Exception in the logs: a Mnesia table does not exist

[yellowshyang]

Describe the bug

2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> ** Reason for termination ==
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> ** {badarg,
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> [{ets,member,
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> [rabbit_queue,
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> {resource,<<"admin_vhost">>,queue,
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> <<"springCloudBus.anonymous.8-BvOUa8SuqwIan2gk9KAA">>}],
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> [{error_info,#{cause => id,module => erl_stdlib_errors}}]},
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> {rabbit_db_queue,exists_in_mnesia,1,
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> [{file,"rabbit_db_queue.erl"},{line,730}]},
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> {mnesia_to_khepri,handle_fallback,5,
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> [{file,"src/mnesia_to_khepri.erl"},{line,529}]},
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> {rabbit_mgmt_metrics_collector,aggregate_entry,4,
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> [{file,"rabbit_mgmt_metrics_collector.erl"},{line,479}]},
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> {lists,foldl,3,[{file,"lists.erl"},{line,1594}]},
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> {ets,do_foldl,4,[{file,"ets.erl"},{line,656}]},
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> {ets,foldl,3,[{file,"ets.erl"},{line,645}]},
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> {rabbit_mgmt_metrics_collector,aggregate_metrics,2,
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0> [{file,"rabbit_mgmt_metrics_collector.erl"},{line,162}]}]}
2024-07-25 15:31:46.201774+08:00 [error] <0.6839.0>
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> crasher:
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> initial call: rabbit_mgmt_metrics_collector:init/1
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> pid: <0.6839.0>
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> registered_name: queue_coarse_metrics_metrics_collector
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> exception error: bad argument
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> in function ets:member/2
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> called as ets:member(rabbit_queue,
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> {resource,<<"admin_vhost">>,queue,
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> <<"springCloudBus.anonymous.8-BvOUa8SuqwIan2gk9KAA">>})
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> *** argument 1: the table identifier does not refer to an existing ETS table
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> in call from rabbit_db_queue:exists_in_mnesia/1 (rabbit_db_queue.erl, line 730)
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> in call from mnesia_to_khepri:handle_fallback/5 (src/mnesia_to_khepri.erl, line 529)
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> in call from rabbit_mgmt_metrics_collector:aggregate_entry/4 (rabbit_mgmt_metrics_collector.erl, line 479)
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> in call from lists:foldl/3 (lists.erl, line 1594)
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> in call from ets:do_foldl/4 (ets.erl, line 656)
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> in call from ets:foldl/3 (ets.erl, line 645)
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> in call from rabbit_mgmt_metrics_collector:aggregate_metrics/2 (rabbit_mgmt_metrics_collector.erl, line 162)
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> ancestors: [rabbit_mgmt_agent_sup,rabbit_mgmt_agent_sup_sup,<0.714.0>]
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> message_queue_len: 0
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> messages: []
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> links: [<0.717.0>]
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> dictionary: []
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> trap_exit: false
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> status: running
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> heap_size: 10958
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> stack_size: 28
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> reductions: 24177
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0> neighbours:
2024-07-25 15:31:46.202256+08:00 [error] <0.6839.0>
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> supervisor: {local,rabbit_mgmt_agent_sup}
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> errorContext: child_terminated
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> reason: {badarg,
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> [{ets,member,
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> [rabbit_queue,
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> {resource,<<"admin_vhost">>,queue,
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> <<"springCloudBus.anonymous.8-BvOUa8SuqwIan2gk9KAA">>}],
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> [{error_info,
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> #{cause => id,module => erl_stdlib_errors}}]},
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> {rabbit_db_queue,exists_in_mnesia,1,
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> [{file,"rabbit_db_queue.erl"},{line,730}]},
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> {mnesia_to_khepri,handle_fallback,5,
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> [{file,"src/mnesia_to_khepri.erl"},{line,529}]},
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> {rabbit_mgmt_metrics_collector,aggregate_entry,4,
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> [{file,"rabbit_mgmt_metrics_collector.erl"},{line,479}]},
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> {lists,foldl,3,[{file,"lists.erl"},{line,1594}]},
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> {ets,do_foldl,4,[{file,"ets.erl"},{line,656}]},
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> {ets,foldl,3,[{file,"ets.erl"},{line,645}]},
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> {rabbit_mgmt_metrics_collector,aggregate_metrics,2,
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> [{file,"rabbit_mgmt_metrics_collector.erl"},
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> {line,162}]}]}
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> offender: [{pid,<0.6839.0>},
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> {id,queue_coarse_metrics_metrics_collector},
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> {mfargs,
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> {rabbit_mgmt_metrics_collector,start_link,
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> [queue_coarse_metrics]}},
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> {restart_type,permanent},
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> {significant,false},
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> {shutdown,300000},
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0> {child_type,worker}]
2024-07-25 15:31:46.202834+08:00 [error] <0.717.0>
2024-07-25 15:31:46.456803+08:00 [error] <0.6840.0> ** Generic server queue_metrics_metrics_collector terminating
2024-07-25 15:31:46.456803+08:00 [error] <0.6840.0> ** Last message in was collect_metrics
2024-07-25 15:31:46.456803+08:00 [error] <0.6840.0> ** When Server state == {state,queue_metrics,5000,
2024-07-25 15:31:46.456803+08:00 [error] <0.6840.0> {[{605,5},{3600,60}],
2024-07-25 15:31:46.456803+08:00 [error] <0.6840.0> [{605,5}],
2024-07-25 15:31:46.456803+08:00 [error] <0.6840.0> [{605,5},{3660,60},{29400,600},{86400,1800}]},
2024-07-25 15:31:46.456803+08:00 [error] <0.6840.0> basic,fun rabbit_amqqueue:exists/1,
2024-07-25 15:31:46.456803+08:00 [error] <0.6840.0> fun rabbit_exchange:exists/1,#{},undefined}

Reproduction steps

1.Restart the springboot service（ps.Connect to the RabbitMq server）

Expected behavior

none

Additional context

No response

[yellowshyang]

version info :
RabbitMQ 3.13.4Erlang 26.2.5.1

[michaelklishin]

@yellowshyang we cannot suggest anything specific with jsut this stack trace. Looks like an ETS table used by Mnesia runs into this, possibly the node hasn't finished syncing its schema tables before a client connected.

We need a detailed set of steps to reproduce. We do not guess in this community.

[michaelklishin]

@yellowshyang two specific things that we can recommend:

1. Moving to 3.13.6 (for reasons unrelated to this exception)
2. Posting full logs
3. Posting rabbitmq-diagnostics report (and editing out sensitive names)

Beyond this, we cannot suggest much with a single stack trace and a "reproduction step".

[yellowshyang]

@michaelklishin Does it matter what version I'm using?

Our project is springboot project，using spring-cloud-starter-bus-amqp（version:3.1.2）；
We using docker container to start rabbitmq server（version:3.13.4）;

[michaelklishin]

3.13.6 release notes explicitly suggest skipping 3.13.4 and upgrading.

[yellowshyang]

@michaelklishin Sorry, I'm not very good at asking questions on github

[michaelklishin]

And we are not very good at guessing. We have suggested a few things, restarting the node/pod might help get the schema database tables synced.