QQ gets into inconsistent state by add_member in pre_vote state #6511

gomoripeti · 2022-11-28T17:34:50Z

gomoripeti
Nov 28, 2022

A quorum queue crashes and gets into an inconsistent state regarding its members when a member change is attempted while there is no quorum and no elected leader.

RabbitMQ version main at ed7f5c2 with ra v2.4.1. (Was seen with 3.10.x versions as well)

Steps to reproduce: 3 node cluster with 1 quorum queue leader on node 2

stop_app on a follower 3
delete_member on leader node 2
add_member same node 2 (this hangs indefinitely waiting for a leader),
start_app on node 2 -> hanging add_member call crashes, node 2 still not member

shell log

$ sbin/rabbitmq-queues quorum_status qq4
Status of quorum queue qq4 on node rabbit@localhost...
┌───────────────────────────┬────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name                 │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ r2@localhost              │ leader     │ 2         │ 2            │ undefined      │ 1    │ 3               │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ r3@localhost              │ follower   │ 2         │ 2            │ undefined      │ 1    │ 3               │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@localhost          │ follower   │ 2         │ 2            │ undefined      │ 1    │ 3               │
└───────────────────────────┴────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘
$ sbin/rabbitmqctl --node r3 stop_app
Stopping rabbit application on node r3@localhost...
$ sbin/rabbitmq-queues quorum_status qq4
Status of quorum queue qq4 on node rabbit@localhost...
┌───────────────────────────┬────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name                 │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ r2@localhost              │ leader     │ 2         │ 2            │ undefined      │ 1    │ 3               │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ r3@localhost              │ noproc     │           │              │                │      │                 │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@localhost          │ follower   │ 2         │ 2            │ undefined      │ 1    │ 3               │
└───────────────────────────┴────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘
$ ./sbin/rabbitmq-queues delete_member qq4 r2@localhost
Removing a replica of queue qq4 on node r2@localhost...
$ sbin/rabbitmq-queues quorum_status qq4
Status of quorum queue qq4 on node rabbit@localhost...
┌───────────────────────────┬────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name                 │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ r3@localhost              │ noproc     │           │              │                │      │                 │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@localhost          │ pre_vote   │ 3         │ 2            │ undefined      │ 1    │ 3               │
└───────────────────────────┴────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘
$ ./sbin/rabbitmq-queues add_member qq4 r2@localhost
Adding a replica for queue qq4 on node r2@localhost...

<...hangs until start_app...>

Error:
{{{:badkey, :previous_cluster}, [{:erlang, :map_get, [:previous_cluster, %{aux_state:

in another shell:

$ sbin/rabbitmq-queues quorum_status qq4
Status of quorum queue qq4 on node rabbit@localhost...
┌───────────────────────────┬────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name                 │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ r3@localhost              │ noproc     │           │              │                │      │                 │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@localhost          │ pre_vote   │ 3         │ 2            │ undefined      │ 1    │ 3               │
└───────────────────────────┴────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘
$ sbin/rabbitmqctl --node r3 start_app
Starting node r3@localhost...
$ sbin/rabbitmq-queues quorum_status qq4
Status of quorum queue qq4 on node rabbit@localhost ...
┌───────────────────────────┬────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name                 │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ r3@localhost              │ leader     │ 3         │ 3            │ undefined      │ 2    │ 3               │
├───────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@localhost          │ follower   │ 3         │ 3            │ undefined      │ 2    │ 3               │
└───────────────────────────┴────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘

At this point the ra server processes are up on all 3 nodes but the queue entry in rabbit_queue table only knows about node 1 and 3.

trying add member again fails.

./sbin/rabbitmq-queues add_member qq4 r2@localhost
Adding a replica for queue qq4 on node r2@localhost...
Error:
{:shutdown, {:failed_to_start_child, :"%2F_qq4", {:already_started, #PID<12602.1338.0>}}}

node 1 log: qq_add_member_crash.txt

Answered by michaelklishin

Dec 5, 2023

main has periodic re-evaluation of QQ replicas #8218, as suggested above :)

View full answer

lukebakken · 2022-11-28T18:03:16Z

lukebakken
Nov 28, 2022
Maintainer

I'm curious if this represents a real-world issue or just some testing. It seems like a Bad Idea to stop a raft member and then do membership changes. I'm sure @kjnilsson will have more to ask/say.

1 reply

gomoripeti Nov 28, 2022
Author

I tried to come up with a reproducible list of steps. The real world issue would be a race condition between a raft membership change and a node crashing and restarting meanwhile. (If I try to check if a raft cluster/quorum queue is healthy before I do a membership change operation, a failure can still happen in between.)

The crash seems to be similar to rabbitmq/ra#33 but that should be fixed long ago.

michaelklishin · 2022-11-28T19:50:52Z

michaelklishin
Nov 28, 2022
Maintainer

If there is no quorum and elected leader you cannot perform cluster membership changes since they go through Raft log.

Ra intentionally limits cluster membership changes to one at a time (so do some other Raft implementations).

So this is at least in part by design.

3 replies

gomoripeti Nov 28, 2022
Author

sorry I was not clear about the expected behaviour

It is totally fine and expected that when there is no quorum/no leader cluster membership changes are either rejected or pending until later (the later is observed in this case)

However I would expect that when quorum is restored, the pending change is either rejected cleanly or applied. In this case it causes a crash and although it looks like one node is not a member, it is not possible to add it.

michaelklishin Nov 29, 2022
Maintainer

Accumulating membership change operations until after the quorum is restored violates two ideas in Ra:

that clusters without a quorum won't support any write operations, including membership changes
that only one change is applied at a time

The only exception here, which at some point will be addressed, is a forced boot of a single node so that at a cluster can be "restored" by replacing all nodes but one from the original cluster.

gomoripeti Nov 29, 2022
Author

I can fully accept those main ideas but in that case operations that violate those principles should be rejected. It would be very helpful if the rabbitmq-queues commands (add_member, delete_member, grow, shrink) would return an error immediately. (I intentionally mention the rabbitmq cli, and not lower levels of abstraction like the Ra library, as I am not familiar with the implementation. But as an operator who manually executes those commands or from a script an error return would be very helpful)

kjnilsson · 2022-11-29T12:09:46Z

kjnilsson
Nov 29, 2022
Maintainer

I think the issue we see here is one of unclear sources of truth. One may think that mnesia is the source of truth in RabbitMQ and for classic queues that is the case mostly but for quorum queues and streams mnesia is more of a view of the source of truth. In this case the source of truth of the actual QQ membership is the quorum queue itself, not mnesia and in this instance, because the membership change did not complete in the QQ the two have diverged.

that is why we have functions like this one where we periodically check if the actual leader has diverged from what is in mnesia and then updates mnesia as necessary. I think we need to also check that the configured nodes are in sync in order to repair a diverged situation.

3 replies

gomoripeti Nov 29, 2022
Author

absolutely.

I also see repair_amqqueue_nodes, but I don't see it called from anywhere (still handy for debugging)

I'd like to mention that we've also seen discrepancies between membership in the quorum queue and on which nodes the ra_server processes are actually running. ( between ra:start_server and ra:add_member). For example a server was running but not a member, resulting in a {:failed_to_start_child, :..., {:already_started, #PID<12354.17656.0>}}} error.

First I thought that a fix would prevent such inconsistencies from happening, but maybe a more fool-proof approach is to make rabbit_quorum_queue:add_member/delete_member more idempotent (after the source of truth is clarified)

kjnilsson Nov 29, 2022
Maintainer

There may well be stuff we can do to make member changes better in Ra itself but I think from a RabbitMQ's POV it would be valuable to include nodes repair in the regular repair that is done.

kjnilsson Nov 29, 2022
Maintainer

the obvious change in ra woudl be to change the add_member command to await_consensus rather than replying as soon as the entry as been written to the log.

tvhong-amazon · 2023-03-09T00:57:48Z

tvhong-amazon
Mar 9, 2023

I also saw a similar issue in 3.10.10 while testing. Here are some observations:

mnesia shows 3 members:

$ rabbitmqctl eval "{ok, Q} = rabbit_amqqueue:lookup(rabbit_misc:r(<<\"/\">>, queue,
> <<\"test-queue-1\">>)), Q."
{amqqueue,{resource,<<"/">>,queue,<<"test-queue-1">>},
          true,false,none,
          [{<<"x-queue-type">>,longstr,<<"quorum">>}],
          {'%2F_test-queue-1','[email protected]'},
          [],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,
          #{user => <<"admin">>},
          rabbit_quorum_queue,
          #{nodes =>
                ['[email protected]',
                 '[email protected]',
                 '[email protected]']}}

But ra shows only 2 members

$ rabbitmqctl eval "QName = rabbit_misc:r(<<\"/\">>, queue, <<\"test-queue-1\">>), {ok, Q0} = rabbit_amqqueue:lookup(QName), Leader = amqqueue:get_pid(Q0), {ok, Members, _} =ra:members(Leader), Members."

[{'%2F_test-queue-1','[email protected]'},
 {'%2F_test-queue-1','[email protected]'}]

rabbitmq-queues quorum_status output:

┌──────────────────────────────────────────────────┬────────────┬───────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name                                        │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├──────────────────────────────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ [email protected] │ follower   │ 978432    │ 28917882     │ undefined      │ 6    │ 2               │
├──────────────────────────────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ [email protected]  │ follower   │ 28917883  │ 28917883     │ undefined      │ 6    │ 2               │
├──────────────────────────────────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ [email protected]   │ leader     │ 28917883  │ 28917883     │ undefined      │ 6    │ 2               │
└──────────────────────────────────────────────────┴────────────┴───────────┴──────────────┴────────────────┴──────┴─────────────────┘

rabbitmq-queues delete_queue output:

rabbitmq-queues delete_member test-queue-1 [email protected]
Removing a replica of queue test-queue-1 on node [email protected]...
Error:
{:no_more_servers_to_try, [error: :not_member, error: :not_member, error: :not_member]}

I'm still trying to get minimal reproduction steps, but IIRC, what I did was:

Load a queue with ~30M messages (then stop the clients)
Delete a quorum member (i.e. [email protected])
Re-add the member
Delete the member again while it's catching up

I repeated steps 2-4 a few times.

1 reply

kjnilsson Mar 9, 2023
Maintainer

yes we need to do periodic repair of the members stored in mnesia.

lukebakken · 2023-03-23T01:32:34Z

lukebakken
Mar 23, 2023
Maintainer

@michaelklishin @kjnilsson I've had this discussion in my TODO list for a while. Should we promote it to a real issue in rabbitmq-server and/or ra?

0 replies

illotum · 2023-12-04T17:28:44Z

illotum
Dec 4, 2023

FWIW couldn't reproduce this issue on the current main, 4ec0ca9.

1 reply

michaelklishin Dec 5, 2023
Maintainer

main has periodic re-evaluation of QQ replicas #8218, as suggested above :)

Answer selected by michaelklishin

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

QQ gets into inconsistent state by add_member in pre_vote state #6511

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments 9 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

QQ gets into inconsistent state by add_member in pre_vote state #6511

gomoripeti Nov 28, 2022

Replies: 6 comments · 9 replies

lukebakken Nov 28, 2022 Maintainer

gomoripeti Nov 28, 2022 Author

michaelklishin Nov 28, 2022 Maintainer

gomoripeti Nov 28, 2022 Author

michaelklishin Nov 29, 2022 Maintainer

gomoripeti Nov 29, 2022 Author

kjnilsson Nov 29, 2022 Maintainer

gomoripeti Nov 29, 2022 Author

kjnilsson Nov 29, 2022 Maintainer

kjnilsson Nov 29, 2022 Maintainer

tvhong-amazon Mar 9, 2023

kjnilsson Mar 9, 2023 Maintainer

lukebakken Mar 23, 2023 Maintainer

illotum Dec 4, 2023

michaelklishin Dec 5, 2023 Maintainer

gomoripeti
Nov 28, 2022

Replies: 6 comments 9 replies

lukebakken
Nov 28, 2022
Maintainer

gomoripeti Nov 28, 2022
Author

michaelklishin
Nov 28, 2022
Maintainer

gomoripeti Nov 28, 2022
Author

michaelklishin Nov 29, 2022
Maintainer

gomoripeti Nov 29, 2022
Author

kjnilsson
Nov 29, 2022
Maintainer

gomoripeti Nov 29, 2022
Author

kjnilsson Nov 29, 2022
Maintainer

kjnilsson Nov 29, 2022
Maintainer

tvhong-amazon
Mar 9, 2023

kjnilsson Mar 9, 2023
Maintainer

lukebakken
Mar 23, 2023
Maintainer

illotum
Dec 4, 2023

michaelklishin Dec 5, 2023
Maintainer