The problem is updating to version 3.12.10

[perrfect]

Describe the bug

Hello.
I have a rabbitmq cluster from 3 nodes.
On all nodes the version rabbitmq-server and erlang are:

rabbitmq-server-3.10.23-1.el8.noarch
erlang-25.3.2.7-1.el8.x86_64

I want to update all my nodes to rebbitmq-server version 3.12.10.
But finally I can't start rabbitmq-server.service on the version 3.12.10 and on logs I see:

[error] <0.231.0> Feature flags: `direct_exchange_routing_v2`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ.                                                                                                                                             
[error] <0.231.0> Failed to initialize feature flags registry: {disabled_required_feature_flag, direct_exchange_routing_v2}   
BOOT FAILED 

Reproduction steps

1. I switch the first node to drain mode on rabbitmq-server version 3.10.23:
   rabbitmq-upgrade drain
2. Then enable all existing feature flags:
   rabbitmqctl enable_feature_flag all
3. Update to version 3.11.18:
   yum update rabbitmq-server-3.11.18
4. Check that service is running:

systemctl status rabbitmq-server.service
● rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/rabbitmq-server.service.d
           └─limits.conf
   Active: active (running) since Thu 2023-12-21 10:21:42 CET; 10min ago
 Main PID: 749519 (beam.smp)

5. Check feature flags on version 3.11.18:
   rabbitmqctl list_feature_flags

Listing feature flags ...
name    state
classic_mirrored_queue_version  enabled
classic_queue_type_delivery_support     enabled
direct_exchange_routing_v2      unavailable
drop_unroutable_metric  enabled
empty_basic_get_metric  enabled
feature_flags_v2        unavailable
implicit_default_bindings       enabled
listener_records_in_ets unavailable
maintenance_mode_status enabled
quorum_queue    enabled
stream_queue    enabled
stream_sac_coordinator_unblock_group    unavailable
stream_single_active_consumer   unavailable
tracking_records_in_ets unavailable
user_limits     enabled
virtual_host_metadata   enabled

On this step I see unavailable feature flags and I can't enable it. Tried, but got an error:
rabbitmqctl enable_feature_flag all

Enabling all feature flags ...
Error:
This feature flag is not supported by node

rabbitmqctl enable_feature_flag direct_exchange_routing_v2

Enabling feature flag "direct_exchange_routing_v2" ...
Error:
This feature flag is not supported by node

6. Try to update from 3.11.18 to 3.12.10. The process completes successfully, but after that the service rabbitmq-server doesn't start.

Expected behavior

Successfully upgraded from version 3.11.18 to version 3.12.10

Additional context

No response

[mkuratczyk]

Have you upgraded the whole cluster to 3.11.18 or just one node? If just one node, then you can't enable the feature flags, since the other two nodes are still running 3.10 and don't support these flags.

[perrfect]

Have you upgraded the whole cluster to 3.11.18 or just one node? If just one node, then you can't enable the feature flags, since the other two nodes are still running 3.10 and don't support these flags.

Hello. It helped. I updated all nodes to 3.11 and then to 3.12.
Thank you.

[michaelklishin]

This is mentioned right in the section on feature flag fundamentals in the docs.