Exchange can't route messages after node restarts

[Regmir]

Hi, I have 3 node RabbitMQ cluster (version 3.9.15) running on Kubernetes.
I have topic exchange with 6 classic transient queues binded to it (every queue has 1 routing key,which is equal to queue name). When I restart the node on which some of these queues are located, I recreate them with the same names and same routing keys on another node. Sometimes after restart when I try to publish messages to my exchange I got "Message published, but not routed.", but when I try to publish message using default exchange, everything ok and message is published successfully. This problem can be fixed with queue recreation or with rebinding queue to the exchange.

[michaelklishin]

All bindings to transient queues will be deleted when the queues itself are after a node restart.
If you want your topology to survive a node restart, every single element has to be durable.

[Regmir]

but I recreate all queues and bindings on different node before my previous node starts again, and exchange can not route message with existing binding

[mkuratczyk]

Ignoring the actual problem for a moment, can you explain why you decided to design it like this?

1. If you want your queues to remain available when a node goes down, why not a quorum queue instead? You'd avoid this problem altogether
2. If your binding keys are equal to queue names, why do you use a topic exchange? I'm not sure whether this is important for the issue, but a direct exchange is a much simpler mechanism, offering higher performance (although, probably not noticeable if the system is not demanding)

[Regmir]

1. I am using this configuration with olso messaging library for python, in older versions there is no quorum queue support and I faced this problem, after migration to qourum queue problem was solved.
2. I use this exchange when there are more than one service in one vhost. Yes when I have only one service I can just use default exchange and all works fine

[michaelklishin]

I'm afraid "I use oslo.messaging" is not specific enough. We generally recommend avoiding that library and Kombu, as they have proven to have made multiple very unfortunate decisions in the past. Kombu in particular was a very problematic client between 2015-2018 and I somewhat doubt things have dramatically changed.

[Regmir]

I faced same problem as #4237. Without node restarts, one of my durable queue in topic exchange stops receiving messages, but on management plugin I can see binding exists, also other queue on different node in same exchange works fine with same routing key (in this case routing key different from queue name). Queue still gets messages from default exchange. Problem solved with unbinding and binding again or with recreating queue. Any ideas what can cause this problem?

[MaxBreida]

We are facing the same problems as mentioned in #5076 (comment), has anyone found a fix/workaround yet?

RabbitMQ version: 3.9.16

[mkuratczyk]

Can you provide an executable test case we can use?

Also, keep in mind that ephemeral queues and exchanges are deprecated and will be removed in 4.0.

[michaelklishin]

Ephemeral exchanges or queues also mean their bindings will be ephemeral, which is often the root cause of such questions.

[dormanze]

I have encountered this problem many times, especially when the exchange type is topic. I tried to analyze the source code and found that this problem was caused by data inconsistency between the two tables. The rabbit_route table is queried on the admin page or by running the ctl command. The rabbit_topic_trie_binding table is queried during message routing. When a route is added, the rabbit_route table is inserted first and then the rabbit_topic_trie_binding table is inserted. This problem is triggered in most scenarios during rolling upgrade. At the same time, I suggest whether the two tables should be transactional. If the two tables are successfully inserted at the same time, the route is successfully added.

[zaeemarshad]

Adding my own experience, I have seen this happen only twice in the last two years. The latest one was was with 3.12.4 on a 3 node cluster running a combination of quorum and classic mirrored queues. 2 nodes were rebooted which meant the leadership for the queues was concentrated on a single node. Publishers connected to this node experienced 312 NO_ROUTE for two bindings only. When the publisher was connected to another node, it just worked.