Quorum queue becomes unavailable, replica status is reported as noproc and there are no metrics

[cvuillemez]

Describe the bug

A quorum queue become unavailable (after it was purged manually, not able to say if it's the root cause but it's a fact).
The only possible remediation was to delete / re-create the queue.

RabbitMQ version is 3.12.13-1
Erlang is 25.3.2.11

State of the queue during issue:

sudo rabbitmq-queues --vhost vhostXX quorum_status queueXX
Status of quorum queue queueXX on node rabbit@mq02 ...
┌─────────────────────────────────────────┬────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name                               │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├─────────────────────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@mq01                             │ noproc     │           │              │                │      │                 │
├─────────────────────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@mq02                             │ noproc     │           │              │                │      │                 │
├─────────────────────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@mq03                             │ noproc     │           │              │                │      │                 │
└─────────────────────────────────────────┴────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘

See below the related logs (which is similar on 2 nodes):

Warning :

 Error purging queue: {'EXIT',{{case_clause,{error,noproc}},
       [{rabbit_channel,'-handle_method/6-fun-2-',1,
                         [{file,"rabbit_channel.erl"},{line,2624}]},
         {rabbit_misc,with_exit_handler,2,
                     [{file,"rabbit_misc.erl"},{line,511}]},
          {rabbit_channel,handle_method,6,[]}]}}
 rabbit_sysmon_handler busy_dist_port <0.22973.0> [{name,delegate_management_3},{initial_call,{delegate,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.21>,unknown}

Error :

 ** Generic server <0.26851.3021> terminating
 ** Last message in was {'$gen_cast',
                            {method,
                                {'basic.consume',0,
                                    <<"queueXX">>,
                                    <<>>,false,false,false,false,[]},
                                none,noflow}}
 ** When Server state == {ch,
                          {conf,running,rabbit_framing_amqp_0_9_1,219,
                           <0.6888.5960>,<0.17018.3016>,<0.6888.5960>,
                           <<"127.0.0.2:56276 -> 127.0.0.1:5671">>,
                           undefined,
                           {user,<<"admin">>,
                            [administrator],
                            [{rabbit_auth_backend_internal,
                              #Fun<rabbit_auth_backend_internal.3.20918515>}]},
                           <<"vhostXX">>,<<>>,<0.11978.5963>,
                           [{<<"exchange_exchange_bindings">>,bool,true},
                            {<<"connection.blocked">>,bool,true},
                            {<<"authentication_failure_close">>,bool,true},
                            {<<"basic.nack">>,bool,true},
                            {<<"publisher_confirms">>,bool,true},
                            {<<"consumer_cancel_notify">>,bool,true}],
                           none,50,134217728,1800000,#{},1000000000,false},
                          {lstate,<0.31172.3009>,false},
                          none,1,
                          {0,[],[]},
                          {state,#{},erlang},
                          #{},#{},
                          {state,fine,5000,
                           #Ref<0.1413075262.3908042754.125576>},
                          false,1,
                          {rabbit_confirms,undefined,#{}},
                          [],[],none,flow,[],
                          {rabbit_queue_type,#{}},
                          #Ref<0.1413075262.3908042754.125570>,false}
 ** Reason for termination ==                                                                                                                                                                                                          [10/2006]
 ** {{badmatch,{error,noproc}},
     [{rabbit_quorum_queue,consume,3,
                           [{file,"rabbit_quorum_queue.erl"},{line,792}]},
      {rabbit_queue_type,consume,3,[{file,"rabbit_queue_type.erl"},{line,392}]},
      {rabbit_channel,'-basic_consume/8-fun-0-',10,
                      [{file,"rabbit_channel.erl"},{line,1747}]},
      {rabbit_misc,with_exit_handler,2,[{file,"rabbit_misc.erl"},{line,511}]},
      {rabbit_channel,basic_consume,8,
                      [{file,"rabbit_channel.erl"},{line,1744}]},
      {rabbit_channel,handle_method,3,
                      [{file,"rabbit_channel.erl"},{line,1423}]},
      {rabbit_channel,handle_cast,2,[{file,"rabbit_channel.erl"},{line,633}]},
      {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1056}]}]}

   crasher:
     initial call: rabbit_channel:init/1
     pid: <0.26851.3021>
     registered_name: []
     exception exit: {{badmatch,{error,noproc}},
                      [{rabbit_quorum_queue,consume,3,
                           [{file,"rabbit_quorum_queue.erl"},{line,792}]},
                       {rabbit_queue_type,consume,3,
                           [{file,"rabbit_queue_type.erl"},{line,392}]},
                       {rabbit_channel,'-basic_consume/8-fun-0-',10,
                           [{file,"rabbit_channel.erl"},{line,1747}]},
                       {rabbit_misc,with_exit_handler,2,
                           [{file,"rabbit_misc.erl"},{line,511}]},
                       {rabbit_channel,basic_consume,8,
                           [{file,"rabbit_channel.erl"},{line,1744}]},
                       {rabbit_channel,handle_method,3,
                           [{file,"rabbit_channel.erl"},{line,1423}]},
                       {rabbit_channel,handle_cast,2,
                           [{file,"rabbit_channel.erl"},{line,633}]},
                       {gen_server2,handle_msg,2,
                           [{file,"gen_server2.erl"},{line,1056}]}]}
       in function  gen_server2:terminate/3 (gen_server2.erl, line 1172)
     ancestors: [<0.6138.3007>,<0.30273.5962>,<0.4133.5959>,<0.18031.5957>,
                   <0.16014.0>,<0.16013.0>,<0.16012.0>,<0.16010.0>,<0.16009.0>,
                   rabbit_sup,<0.831.0>]
     message_queue_len: 1
     messages: [{'$gen_cast',
                       {command,
                           {'basic.consume_ok',
                               <<"amq.ctag-oVuWAOxAiJMzPnaojKS07w">>}}}]
     links: [<0.6138.3007>]
     dictionary: [{process_name,
                       {rabbit_channel,
                           {<<"127.0.0.2:56276 -> 127.0.0.1:5671">>,
                            219}}},
                   {permission_cache,
                       [{{resource,<<"vhostXX">>,queue,
                             <<"queueXX">>},
                         #{},read}]},
                   {permission_cache_can_expire,false},
                   {rand_seed,
                       {#{jump => #Fun<rand.3.34006561>,
                          max => 288230376151711743,
                          next => #Fun<rand.5.34006561>,type => exsplus},
                        [87273550521800029|10282284222925812]}},
                   {guid_secure,
                       {{24,'rabbit@mq01',
                         #Ref<0.1413075262.3908042754.125589>},
                        0}},
                   {channel_operation_timeout,15000}]
     trap_exit: true
     status: running
     heap_size: 17731
     stack_size: 28
     reductions: 41672
   neighbours:

Reproduction steps

no steps

Expected behavior

no crash :)

Additional context

No response

[michaelklishin]

RabbitMQ 3.12.x is out of community support.

The log snippets shared simply state that a quorum queue leader process is down, and according to list_queues, all replicas are. To form a decent hypothesis, see the events that precede this exception. The exception itself tells you nothing about the root cause.

[michaelklishin]

There is a well known upgrade strategy that can lead to exactly this scenario. It is documented and explicitly recommended against. Whether something like this was used during an upgrade or because some VMs or containers that were running the nodes were replaced, I cannot know.

But you cannot replace a majority of nodes without explicitly removing replicas and adding new ones. The above doc section goes into more detail on what specifically is meant by that.

[cvuillemez]

It was not during upgrade. It is just a bug in normal condition, under normal workload. We have more than one thousand queues. And this one gets down after purging.

[kjnilsson]

There are many improvements and fixes to quorum queues in RabbitMQ v3.13.x. Please upgrade to the latest version and let us know if this occurs again during normal operation.