Queue creation by queue definitions could by interupted during cluster formation

[motmot80]

Describe the bug

Creating a queue (external client) during the cluster formation is cancelling the queue creation process from a cluster definition.

To Reproduce

Steps to reproduce the behavior:

1. Create a AMQPv1 consumer against Queue "A" and reconnecting/reattaching on failure
2. Start the AMQPv1 consumer against a non existing RMQ cluster service endpoint
3. Deploy a RMQ cluster (> 1 Node) with a queue definition of queue "B"
4. As soon as the first node is ready clients could connect and create the queue "A"
5. If a queue is created before the queue definitions are applied then the queue definitions is not applied.

Expected behavior

• Queue "B" ist created by the definition
• Queue "A" is created by the AMQPv1 client consumer

Actual behavior

• Only queue "A" is created as soon as one node is reachable
• No queue from the defintion yaml is created
• Exchanges, bindings and policies are not affected. Queue bindings are related to none existing queues.

Workaround
Actively suppress incoming connections as long as the cluster is not fully started and queue defintions are not applied successfully.

Analysis
If the queue creation process is intererupted by someone creating an "unexpected" queue the queue definition execution seems to be canceled.
The definitions are executed after the cluster has formed. Which it totally fine and expected.
But as soon as ONE of the RMQs nodes confirms it is ready the k8s network service lets clients connect which therefore could create local queues during cluster formation which therefore interupts the queue creation process.

Version and environment information

• RabbitMQ: 3.11.21
• RabbitMQ Cluster Operator: 0.32.0
• Kubernetes: 1.26.3
• Cloud provider or hardware configuration: Microsoft AKS

Additional context

In BCDR contexts it should be possible to recreate a cluster without any extra steps.
Maybe RMQ clusters should not be connectable before the are fully configured to reduce risks of inconsistent states.

[michaelklishin]

@motmot80 we have seen a good dozen of variations of this, and tried multiple things over the years.
We already start client listeners as late as possible in the boot process, and it is never enough in a distributed and highly concurrent system.

I only have two ideas left in mind:

• Introduce a way to inject a randomized or fixed delay into the boot process right before we start client listeners
• Introduce a way to opt-in to starting listeners only after all cluster nodes are online. This may seem like a great idea but what happens when your last node fails to boot: should your entire cluster become unavailable?

I don't see how definition import can interfere with queue declaration but a booting cluster itself
can result in rejected or "semi-complete" queue declaration, for example, a QQ having fewer replicas than expected.

https://github.com/rabbitmq/ra/pulls?q=is%3Apr+is%3Aclosed is good example of how having N replicas can be see as "not good enough" early in the cluster formation process: a catching up replica is technically online but it is not functional replica.

[michaelklishin]

Another idea suggested on the core team is: define a custom startupProbe in the K8S Operator that would likely need a new health check or two, which would reflect the state of boot time definition import (or more).