Node upgraded to 4.0 without mandatory feature flags enabled: how to make it start?

[specialorange]

Describe the bug

In the docs here it states "Moreover, feature flags have to be enabled before the upgrade. The upgrade will fail if you miss this step." it does not explain what to do if your upgrade did not perform this.

Reproduction steps

Looks like RMQ updated last week. According to their docs "Moreover, feature flags have to be enabled before the upgrade. The upgrade will fail if you miss this step." but it does not have any info that I can find or open tickets on Github regarding what to do if you don't enable the feature flags prior to updating. .

How do you turn on the flags on a non running node to get it to start?

Notes:

I on an AWS ubuntu 22.04.5 LTS. I use 'supervisor' to run rabbit and other related services. Occassionally it suggests to run security updates via sudo apt update && sudo apt upgrade -y. I have never had any issues with this. My erlang is at 26.2.5.4

Logs:

tail -n 355 /var/log/rabbitmq/startup_err

BOOT FAILED
===========
Error during startup: {error,failed_to_initialize_feature_flags_registry}

Runtime terminating during boot (terminating)

Crash dump is being written to: erl_crash.dump...done
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
[os_mon] memory supervisor port (memsup): Erlang has closed

tail -n 355 /var/log/rabbitmq/startup_log

=INFO REPORT==== 18-Oct-2024::13:57:09.478721 ===
    alarm_handler: {set,{system_memory_high_watermark,[]}}
D/T [error]<0.254.0> Feature flags: `stream_filtering`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ.
D/T [error]<0.254.0> Failed to initialize feature flags registry: {disabled_required_feature_flag,
D/T [error]<0.254.0>                                               stream_filtering}
D/T [error]<0.254.0> 
D/T [error]<0.254.0> BOOT FAILED
D/T [error]<0.254.0> ===========
D/T [error]<0.254.0> Error during startup: {error,failed_to_initialize_feature_flags_registry}
D/T [error]<0.254.0> 
D/T [error]<0.253.0>   crasher:
D/T [error]<0.253.0>     initial call: application_master:init/4
D/T [error]<0.253.0>     pid: <0.253.0>
D/T [error]<0.253.0>     registered_name: []
D/T [error]<0.253.0>     exception exit: {failed_to_initialize_feature_flags_registry,
D/T [error]<0.253.0>                         {rabbit,start,[normal,[]]}}
D/T [error]<0.253.0>       in function  application_master:init/4 (application_master.erl, line 142)
D/T [error]<0.253.0>     ancestors: [<0.252.0>]
D/T [error]<0.253.0>     message_queue_len: 1
D/T [error]<0.253.0>     messages: [{'EXIT',<0.254.0>,normal}]
D/T [error]<0.253.0>     links: [<0.252.0>,<0.44.0>]
D/T [error]<0.253.0>     dictionary: []
D/T [error]<0.253.0>     trap_exit: true
D/T [error]<0.253.0>     status: running
D/T [error]<0.253.0>     heap_size: 376
D/T [error]<0.253.0>     stack_size: 28
D/T [error]<0.253.0>     reductions: 170
D/T [error]<0.253.0>   neighbours:
D/T [error]<0.253.0> 

sudo service rabbitmq-server start

Job for rabbitmq-server.service failed because the control process exited with error code.
See "systemctl status rabbitmq-server.service" and "journalctl -xeu rabbitmq-server.service" for details.

systemctl status rabbitmq-server.service

● rabbitmq-server.service - RabbitMQ broker
     Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
     Active: activating (start) since Fri 2024-10-18 23:42:16 UTC; 2s ago
   Main PID: 164057 (beam.smp)
     Status: "Standing by"
      Tasks: 25 (limit: 2338)
     Memory: 64.6M
        CPU: 1.495s
     CGroup: /system.slice/rabbitmq-server.service
             ├─164057 /usr/lib/erlang/erts-14.2.5.4/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs >
             ├─164067 erl_child_setup 32768
             ├─164084 sh -s disksup
             ├─164086 /usr/lib/erlang/lib/os_mon-2.9.1/priv/bin/memsup
             ├─164087 /usr/lib/erlang/lib/os_mon-2.9.1/priv/bin/cpu_sup
             ├─164088 /usr/lib/erlang/erts-14.2.5.4/bin/inet_gethost 4
             ├─164089 /usr/lib/erlang/erts-14.2.5.4/bin/inet_gethost 4
             └─164099 /usr/lib/erlang/erts-14.2.5.4/bin/epmd -daemon

D/T ip-187 rabbitmq-server[164057]: 2024-10-18 23:42:18.585056+00:00 [error] <0.255.0>
D/T ip-187 rabbitmq-server[164057]: Runtime terminating during boot (terminating)
D/T ip-187 rabbitmq-server[164057]: 
D/T ip-187 rabbitmq-server[164057]: Crash dump is being written to: erl_crash.dump...done
D/T ip-187 rabbitmq-server[164087]: [os_mon] cpu supervisor port (cpu_sup): Erlang has closed
D/T ip-187 rabbitmq-server[164086]: [os_mon] memory supervisor port (memsup): Erlang has closed
D/T ip-187 systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=1/FAILURE
D/T ip-187 systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
D/T ip-187 systemd[1]: Failed to start RabbitMQ broker.
D/T ip-187 systemd[1]: rabbitmq-server.service: Consumed 1.545s CPU time.

Expected behavior

prevent security upgrades from doing major version upgrades without applying the CLI flags for feature flags to be turned on.

Additional context

Stackoverflow question with additional details

https://stackoverflow.com/questions/79108255/how-do-you-turn-on-feature-flags-on-a-non-running-node-to-get-it-to-start

[michaelklishin]

@specialorange using issues for questions in 2024 in repositories that have discussions enabled is just rude.

RabbitMQ 4.0

RabbitMQ 4.0.1 and 4.0.2 shipped about a month ago.

How to Override the List of Feature Flags to Enable on Initial Startup

How to Override the List of Feature Flags to Enable on Initial Startup.

You will have to list all flags, not just the ones you want to be enabled. Since feature flags
are cluster-wide state, in a cluster of N nodes you will likely have to do a full stop and restart if you ended up upgrading to 4.0 without enabling all feature flags.

Stated End Goal

prevent security upgrades from doing major version upgrades without applying the CLI flags for feature flags to be turned on

I'm afraid I don't understand the end goal. Security updates in theory can introduce a feature flag necessary to change a problematic behavior that affects the inter-node communication protocol but we have never had an actual example of such a patch release.

The Root Cause

Upgrading to 4.0 is not a "security update". A few more fundamental problems you are facing @specialorange are

1. You provision RabbitMQ with apt without version pinning so a major release can be installed automatically. Team RabbitMQ has documented version pinning but it's on you to use it if you do not control when upgrades are performed
2. Upgrades in your environment can happen at any moment. Is this really a good approach for distributed data services? My answer is "no, clearly not without version pinning"
3. Follow the release notes every so often. Not only 4.0 had multiple preview releases with release notes over a period of several months, the 4.0.1 GA release notes dedicate a good page to upgrades, obvious or potentially breaking behavior changes, and so on, including a very visible note on feature flags

[michaelklishin]

Here is a RABBITMQ_FEATURE_FLAGS value that will enable all 4.0 feature flags except for khepri_db which is opt-in and not required:

RABBITMQ_FEATURE_FLAGS="delete_ra_cluster_mqtt_node,virtual_host_metadata,stream_single_active_consumer,quorum_queue,classic_mirrored_queue_version,rabbit_mqtt_qos0_queue,implicit_default_bindings,empty_basic_get_metric,'rabbitmq_4.0.0',message_containers,user_limits,queue_master_locator,detailed_queues_endpoint,stream_sac_coordinator_unblock_group,stream_update_config_command,stream_queue,stream_filtering,rabbit_exchange_type_local_random,quorum_queue_non_voters,tracking_records_in_ets,direct_exchange_routing_v2,amqp_address_v1,transient_nonexcl_queues,message_containers_deaths_v2,classic_queue_mirroring,management_metrics_collection,maintenance_mode_status,listener_records_in_ets,feature_flags_v2,global_qos,classic_queue_type_delivery_support,mqtt_v5,ram_node_type,drop_unroutable_metric,restart_streams"

The same list using the advanced.config option:

%% ...
{rabbit, [
  %% ...
  {forced_feature_flags_on_init, [
    maintenance_mode_status,
    direct_exchange_routing_v2,
    user_limits,
    transient_nonexcl_queues,
    amqp_address_v1,stream_filtering,
    implicit_default_bindings,
    quorum_queue_non_voters,
    'rabbitmq_4.0.0',
    tracking_records_in_ets,
    delete_ra_cluster_mqtt_node,
    classic_queue_type_delivery_support,
    restart_streams,
    message_containers_deaths_v2,
    feature_flags_v2,empty_basic_get_metric,
    classic_queue_mirroring,
    rabbit_exchange_type_local_random,
    detailed_queues_endpoint,
    stream_queue,
    classic_mirrored_queue_version,
    quorum_queue,
    management_metrics_collection,
    message_containers,
    ram_node_type,
    stream_sac_coordinator_unblock_group,
    drop_unroutable_metric,
    stream_single_active_consumer,
    virtual_host_metadata,
    listener_records_in_ets,
    stream_update_config_command,
    global_qos,
    queue_master_locator,
    rabbit_mqtt_qos0_queue,mqtt_v5
   ]}
  %% ...
]},
%% ...

I will add both to the doc guides.

[michaelklishin]

rabbitmq/rabbitmq-website@38d0f68