RabbitMQ connection closing unexpectedly causes the service to crash

[odesenfans]

Logs:

P2P service:

[2023-05-15T16:53:43Z INFO actix_web::middleware::logger] 172.22.0.8 "GET /api/p2p/identify HTTP/1.1" 200 76 "-" "Python/3.11 aiohttp/3.8.3" 0.000180
[2023-05-15T16:54:35Z WARN libp2p_gossipsub::handler] Inbound stream error: Io(Kind(UnexpectedEof))
[2023-05-15T16:54:35Z WARN libp2p_gossipsub::handler] Inbound substream error while closing: Io(Custom { kind: UnexpectedEof, error: "unexpected end of file" })
[2023-05-15T16:54:35Z WARN libp2p_gossipsub::handler] Inbound stream error: Io(Kind(UnexpectedEof))
[2023-05-15T16:54:35Z WARN libp2p_gossipsub::handler] Inbound substream error while closing: Io(Custom { kind: UnexpectedEof, error: "unexpected end of file" })
[2023-05-15T16:55:19Z WARN libp2p_gossipsub::handler] Inbound stream error: Io(Kind(UnexpectedEof))
[2023-05-15T16:55:19Z WARN libp2p_gossipsub::handler] Inbound substream error while closing: Io(Custom { kind: UnexpectedEof, error: "unexpected end of file" })
[2023-05-15T16:55:40Z ERROR lapin::channel] Connection closed channel=0 method=Close { reply_code: 320, reply_text: ShortString("CONNECTION_FORCED - broker forced connection closure with reason 'shutdow}
[2023-05-15T16:55:41Z ERROR lapin::channels] Connection error error=protocol error: AMQP hard error: CONNECTION-FORCED: CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
[2023-05-15T16:55:41Z ERROR lapin::io_loop] error doing IO error=IOError(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })
thread 'main' panicked at 'called Result::unwrap() on an Err value: ProtocolError(AMQPError { kind: Hard(CONNECTIONFORCED), message: ShortString("CONNECTION_FORCED - broker forced connection closure6
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
thread 'main' panicked at 'receiver should not be dropped: SendError { kind: Disconnected }', src/p2p/network.rs:281:30
thread 'actix-rt|system:0|arbiter:0' panicked at 'Sender should not be dropped: Canceled', src/p2p/network.rs:165:24
thread 'actix-rt|system:0|arbiter:1' panicked at 'Command receiver should not to be dropped: SendError { kind: Disconnected }', src/p2p/network.rs:164:41

RabbitMQ:

2023-05-15 16:55:57.187715+00:00 [notice] <0.44.0> Application syslog exited with reason: stopped
2023-05-15 16:55:57.194412+00:00 [notice] <0.230.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
2023-05-15 16:55:57.211616+00:00 [notice] <0.230.0> Logging: configured log handlers are now ACTIVE
2023-05-15 16:55:57.451300+00:00 [info] <0.230.0> ra: starting system quorum_queues
2023-05-15 16:55:57.451445+00:00 [info] <0.230.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@1e8d1e887977/quorum/rabbit@1e8d1e887977
2023-05-15 16:55:57.529147+00:00 [info] <0.266.0> ra system 'quorum_queues' running pre init for 0 registered servers
2023-05-15 16:55:57.538614+00:00 [info] <0.267.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
2023-05-15 16:55:57.552300+00:00 [notice] <0.272.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables
2023-05-15 16:55:57.583423+00:00 [info] <0.230.0> ra: starting system coordination
2023-05-15 16:55:57.583524+00:00 [info] <0.230.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@1e8d1e887977/coordination/rabbit@1e8d1e887977
2023-05-15 16:55:57.584728+00:00 [info] <0.279.0> ra system 'coordination' running pre init for 0 registered servers
2023-05-15 16:55:57.585381+00:00 [info] <0.280.0> ra: meta data store initialised for system coordination. 0 record(s) recovered
2023-05-15 16:55:57.585595+00:00 [notice] <0.285.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables
2023-05-15 16:55:57.588354+00:00 [info] <0.230.0>
2023-05-15 16:55:57.588354+00:00 [info] <0.230.0> Starting RabbitMQ 3.11.15 on Erlang 25.3.2 [jit]
2023-05-15 16:55:57.588354+00:00 [info] <0.230.0> Copyright (c) 2007-2023 VMware, Inc. or its affiliates.
2023-05-15 16:55:57.588354+00:00 [info] <0.230.0> Licensed under the MPL 2.0. Website: https://rabbitmq.com

## RabbitMQ 3.11.15

########## Copyright (c) 2007-2023 VMware, Inc. or its affiliates.

########## Licensed under the MPL 2.0. Website: https://rabbitmq.com

Erlang: 25.3.2 [jit]
TLS Library: OpenSSL - OpenSSL 3.0.8 7 Feb 2023
Release series support status: supported

Doc guides: https://rabbitmq.com/documentation.html
Support: https://rabbitmq.com/contact.html
Tutorials: https://rabbitmq.com/getstarted.html
Monitoring: https://rabbitmq.com/monitoring.html

Logs: /var/log/rabbitmq/rabbit@1e8d1e887977_upgrade.log

Config file(s): /etc/rabbitmq/conf.d/10-defaults.conf

Starting broker...2023-05-15 16:55:57.589431+00:00 [info] <0.230.0>
2023-05-15 16:55:57.589431+00:00 [info] <0.230.0> node : rabbit@1e8d1e887977
2023-05-15 16:55:57.589431+00:00 [info] <0.230.0> home dir : /var/lib/rabbitmq
2023-05-15 16:55:57.589431+00:00 [info] <0.230.0> config file(s) : /etc/rabbitmq/conf.d/10-defaults.conf
2023-05-15 16:55:57.589431+00:00 [info] <0.230.0> cookie hash : Ce5LxEayUvTR5XVsAUReLA==
2023-05-15 16:55:57.589431+00:00 [info] <0.230.0> log(s) : /var/log/rabbitmq/rabbit@1e8d1e887977_upgrade.log
2023-05-15 16:55:57.589431+00:00 [info] <0.230.0> :
2023-05-15 16:55:57.589431+00:00 [info] <0.230.0> database dir : /var/lib/rabbitmq/mnesia/rabbit@1e8d1e887977
2023-05-15 16:56:01.992301+00:00 [info] <0.230.0> Running boot step pre_boot defined by app rabbit
2023-05-15 16:56:01.992496+00:00 [info] <0.230.0> Running boot step rabbit_global_counters defined by app rabbit
2023-05-15 16:56:01.993251+00:00 [info] <0.230.0> Running boot step rabbit_osiris_metrics defined by app rabbit
2023-05-15 16:56:01.993604+00:00 [info] <0.230.0> Running boot step rabbit_core_metrics defined by app rabbit
2023-05-15 16:56:01.995114+00:00 [info] <0.230.0> Running boot step rabbit_alarm defined by app rabbit
2023-05-15 16:56:01.999614+00:00 [info] <0.300.0> Memory high watermark set to 38604 MiB (40480042188 bytes) of 96511 MiB (101200105472 bytes) total
2023-05-15 16:56:02.002079+00:00 [info] <0.302.0> Enabling free disk space monitoring (disk free space: 5395905806336, total memory: 101200105472)
2023-05-15 16:56:02.002146+00:00 [info] <0.302.0> Disk free limit set to 50MB
2023-05-15 16:56:02.003725+00:00 [info] <0.230.0> Running boot step code_server_cache defined by app rabbit
2023-05-15 16:56:02.003875+00:00 [info] <0.230.0> Running boot step file_handle_cache defined by app rabbit
2023-05-15 16:56:02.004151+00:00 [info] <0.305.0> Limiting to approx 1048479 file handles (943629 sockets)
2023-05-15 16:56:02.004311+00:00 [info] <0.306.0> FHC read buffering: OFF
2023-05-15 16:56:02.004388+00:00 [info] <0.306.0> FHC write buffering: ON
2023-05-15 16:56:02.004799+00:00 [info] <0.230.0> Running boot step worker_pool defined by app rabbit
2023-05-15 16:56:02.004893+00:00 [info] <0.287.0> Will use 8 processes for default worker pool
2023-05-15 16:56:02.004961+00:00 [info] <0.287.0> Starting worker pool 'worker_pool' with 8 processes in it
2023-05-15 16:56:02.005610+00:00 [info] <0.230.0> Running boot step database defined by app rabbit
2023-05-15 16:56:02.005880+00:00 [info] <0.230.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@1e8d1e887977 is empty. Assuming we need to join an existing cluster or initialise from scratc.
2023-05-15 16:56:02.005945+00:00 [info] <0.230.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2023-05-15 16:56:02.005991+00:00 [info] <0.230.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2023-05-15 16:56:02.006153+00:00 [info] <0.230.0> All discovered existing cluster peers:
2023-05-15 16:56:02.006196+00:00 [info] <0.230.0> Discovered no peer nodes to cluster with. Some discovery backends can filter nodes out based on a readiness criteria. Enabling debug logging might help .
2023-05-15 16:56:02.008250+00:00 [notice] <0.44.0> Application mnesia exited with reason: stopped
2023-05-15 16:56:02.116423+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2023-05-15 16:56:02.116633+00:00 [info] <0.230.0> Successfully synced tables from a peer
2023-05-15 16:56:02.156192+00:00 [info] <0.230.0> Feature flags: feature_flags_v2: supported, attempt to enable...
2023-05-15 16:56:02.191443+00:00 [notice] <0.288.0> Feature flags: attempt to enable classic_mirrored_queue_version...
2023-05-15 16:56:02.226295+00:00 [notice] <0.288.0> Feature flags: classic_mirrored_queue_version enabled
2023-05-15 16:56:02.227025+00:00 [notice] <0.288.0> Feature flags: attempt to enable classic_queue_type_delivery_support...
2023-05-15 16:56:02.242638+00:00 [notice] <0.288.0> Feature flags: attempt to enable stream_queue...
2023-05-15 16:56:02.274992+00:00 [notice] <0.288.0> Feature flags: stream_queue enabled
2023-05-15 16:56:02.291627+00:00 [notice] <0.288.0> Feature flags: classic_queue_type_delivery_support enabled
2023-05-15 16:56:02.292221+00:00 [notice] <0.288.0> Feature flags: attempt to enable direct_exchange_routing_v2...
2023-05-15 16:56:02.308624+00:00 [info] <0.503.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2023-05-15 16:56:02.308884+00:00 [info] <0.503.0> Successfully synced tables from a peer
2023-05-15 16:56:02.335486+00:00 [notice] <0.288.0> Feature flags: direct_exchange_routing_v2 enabled
2023-05-15 16:56:02.336152+00:00 [notice] <0.288.0> Feature flags: attempt to enable drop_unroutable_metric...
2023-05-15 16:56:02.376124+00:00 [notice] <0.288.0> Feature flags: drop_unroutable_metric enabled
2023-05-15 16:56:02.376999+00:00 [notice] <0.288.0> Feature flags: attempt to enable empty_basic_get_metric...
2023-05-15 16:56:02.415193+00:00 [notice] <0.288.0> Feature flags: empty_basic_get_metric enabled
2023-05-15 16:56:02.416101+00:00 [notice] <0.288.0> Feature flags: attempt to enable listener_records_in_ets...
2023-05-15 16:56:02.459885+00:00 [notice] <0.288.0> Feature flags: listener_records_in_ets enabled
2023-05-15 16:56:02.461692+00:00 [notice] <0.288.0> Feature flags: attempt to enable stream_sac_coordinator_unblock_group...
2023-05-15 16:56:02.478514+00:00 [notice] <0.288.0> Feature flags: attempt to enable stream_single_active_consumer...
2023-05-15 16:56:02.521069+00:00 [notice] <0.288.0> Feature flags: stream_single_active_consumer enabled
2023-05-15 16:56:02.537064+00:00 [notice] <0.288.0> Feature flags: stream_sac_coordinator_unblock_group enabled
2023-05-15 16:56:02.537845+00:00 [notice] <0.288.0> Feature flags: attempt to enable tracking_records_in_ets...
2023-05-15 16:56:02.580119+00:00 [notice] <0.288.0> Feature flags: tracking_records_in_ets enabled
2023-05-15 16:56:02.581080+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2023-05-15 16:56:02.581206+00:00 [info] <0.230.0> Successfully synced tables from a peer
2023-05-15 16:56:02.591543+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2023-05-15 16:56:02.591698+00:00 [info] <0.230.0> Successfully synced tables from a peer
2023-05-15 16:56:02.591744+00:00 [info] <0.230.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2023-05-15 16:56:02.591788+00:00 [info] <0.230.0> Will try to unlock with peer discovery backend rabbit_peer_discovery_classic_config
2023-05-15 16:56:02.591874+00:00 [info] <0.230.0> Running boot step tracking_metadata_store defined by app rabbit
2023-05-15 16:56:02.591980+00:00 [info] <0.636.0> Setting up a table for connection tracking on this node: tracked_connection
2023-05-15 16:56:02.592043+00:00 [info] <0.636.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost
2023-05-15 16:56:02.592103+00:00 [info] <0.636.0> Setting up a table for per-user connection counting on this node: tracked_connection_per_user
2023-05-15 16:56:02.592157+00:00 [info] <0.636.0> Setting up a table for channel tracking on this node: tracked_channel
2023-05-15 16:56:02.592225+00:00 [info] <0.636.0> Setting up a table for channel tracking on this node: tracked_channel_per_user
2023-05-15 16:56:02.592352+00:00 [info] <0.230.0> Running boot step networking_metadata_store defined by app rabbit
2023-05-15 16:56:02.592468+00:00 [info] <0.230.0> Running boot step database_sync defined by app rabbit
2023-05-15 16:56:02.592583+00:00 [info] <0.230.0> Running boot step feature_flags defined by app rabbit
2023-05-15 16:56:02.592747+00:00 [info] <0.230.0> Running boot step codec_correctness_check defined by app rabbit
2023-05-15 16:56:02.592811+00:00 [info] <0.230.0> Running boot step external_infrastructure defined by app rabbit
2023-05-15 16:56:02.592869+00:00 [info] <0.230.0> Running boot step rabbit_event defined by app rabbit
2023-05-15 16:56:02.593039+00:00 [info] <0.230.0> Running boot step rabbit_registry defined by app rabbit
2023-05-15 16:56:02.593180+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2023-05-15 16:56:02.593286+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2023-05-15 16:56:02.593408+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2023-05-15 16:56:02.593532+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2023-05-15 16:56:02.593637+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2023-05-15 16:56:02.593770+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2023-05-15 16:56:02.593908+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2023-05-15 16:56:02.594008+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2023-05-15 16:56:02.594107+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2023-05-15 16:56:02.594206+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2023-05-15 16:56:02.594316+00:00 [info] <0.230.0> Running boot step rabbit_priority_queue defined by app rabbit
2023-05-15 16:56:02.594382+00:00 [info] <0.230.0> Priority queues enabled, real BQ is rabbit_variable_queue
2023-05-15 16:56:02.594530+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2023-05-15 16:56:02.594658+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2023-05-15 16:56:02.594758+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_random defined by app rabbit
2023-05-15 16:56:02.594869+00:00 [info] <0.230.0> Running boot step kernel_ready defined by app rabbit
2023-05-15 16:56:02.594916+00:00 [info] <0.230.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2023-05-15 16:56:02.595052+00:00 [info] <0.230.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2023-05-15 16:56:02.595752+00:00 [info] <0.645.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2023-05-15 16:56:02.595872+00:00 [info] <0.230.0> Running boot step guid_generator defined by app rabbit
2023-05-15 16:56:02.597562+00:00 [info] <0.230.0> Running boot step rabbit_node_monitor defined by app rabbit
2023-05-15 16:56:02.597783+00:00 [info] <0.649.0> Starting rabbit_node_monitor
2023-05-15 16:56:02.597912+00:00 [info] <0.230.0> Running boot step delegate_sup defined by app rabbit
2023-05-15 16:56:02.598562+00:00 [info] <0.230.0> Running boot step rabbit_memory_monitor defined by app rabbit
2023-05-15 16:56:02.598783+00:00 [info] <0.230.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit
2023-05-15 16:56:02.598888+00:00 [info] <0.230.0> Running boot step core_initialized defined by app rabbit
2023-05-15 16:56:02.598951+00:00 [info] <0.230.0> Running boot step upgrade_queues defined by app rabbit
2023-05-15 16:56:02.608249+00:00 [info] <0.230.0> message_store upgrades: 1 to apply
2023-05-15 16:56:02.608410+00:00 [info] <0.230.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2023-05-15 16:56:02.608611+00:00 [info] <0.230.0> message_store upgrades: No durable queues found. Skipping message store migration
2023-05-15 16:56:02.608673+00:00 [info] <0.230.0> message_store upgrades: Removing the old message store data
2023-05-15 16:56:02.609067+00:00 [info] <0.230.0> message_store upgrades: All upgrades applied successfully
2023-05-15 16:56:02.642613+00:00 [info] <0.230.0> Running boot step channel_tracking defined by app rabbit
2023-05-15 16:56:02.642749+00:00 [info] <0.230.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
2023-05-15 16:56:02.642876+00:00 [info] <0.230.0> Running boot step connection_tracking defined by app rabbit
2023-05-15 16:56:02.642940+00:00 [info] <0.230.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2023-05-15 16:56:02.643044+00:00 [info] <0.230.0> Running boot step rabbit_definitions_hashing defined by app rabbit
2023-05-15 16:56:02.643189+00:00 [info] <0.230.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2023-05-15 16:56:02.643328+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2023-05-15 16:56:02.643756+00:00 [info] <0.230.0> Running boot step rabbit_policies defined by app rabbit
2023-05-15 16:56:02.644254+00:00 [info] <0.230.0> Running boot step rabbit_policy defined by app rabbit
2023-05-15 16:56:02.644356+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2023-05-15 16:56:02.644529+00:00 [info] <0.230.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2023-05-15 16:56:02.644679+00:00 [info] <0.230.0> Running boot step rabbit_stream_coordinator defined by app rabbit
2023-05-15 16:56:02.644853+00:00 [info] <0.230.0> Running boot step rabbit_vhost_limit defined by app rabbit
2023-05-15 16:56:02.644983+00:00 [info] <0.230.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2023-05-15 16:56:02.645080+00:00 [info] <0.230.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2023-05-15 16:56:02.645161+00:00 [info] <0.230.0> Management plugin: using rates mode 'basic'
2023-05-15 16:56:02.645738+00:00 [info] <0.230.0> Running boot step recovery defined by app rabbit
2023-05-15 16:56:02.646573+00:00 [info] <0.230.0> Running boot step empty_db_check defined by app rabbit
2023-05-15 16:56:02.646652+00:00 [info] <0.230.0> Will seed default virtual host and user...
2023-05-15 16:56:02.646774+00:00 [info] <0.230.0> Adding vhost '/' (description: 'Default virtual host', tags: [])
2023-05-15 16:56:02.662623+00:00 [info] <0.691.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@1e8d1e887977/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2023-05-15 16:56:02.663582+00:00 [info] <0.691.0> Setting segment_entry_count for vhost '/' with 0 queues to '2048'
2023-05-15 16:56:02.665887+00:00 [info] <0.691.0> Starting message stores for vhost '/'
2023-05-15 16:56:02.666123+00:00 [info] <0.696.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2023-05-15 16:56:02.667154+00:00 [info] <0.691.0> Started message store of type transient for vhost '/'
2023-05-15 16:56:02.667375+00:00 [info] <0.700.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2023-05-15 16:56:02.668031+00:00 [warning] <0.700.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2023-05-15 16:56:02.668829+00:00 [info] <0.691.0> Started message store of type persistent for vhost '/'
2023-05-15 16:56:02.669042+00:00 [info] <0.691.0> Recovering 0 queues of type rabbit_classic_queue took 5ms
2023-05-15 16:56:02.669103+00:00 [info] <0.691.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
2023-05-15 16:56:02.669159+00:00 [info] <0.691.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
2023-05-15 16:56:02.673202+00:00 [info] <0.230.0> Created user 'aleph-p2p'
2023-05-15 16:56:02.688691+00:00 [info] <0.230.0> Successfully set user tags for user 'aleph-p2p' to [administrator]
2023-05-15 16:56:02.690575+00:00 [info] <0.230.0> Successfully set permissions for 'aleph-p2p' in virtual host '/' to '.', '.', '.*'
2023-05-15 16:56:02.690676+00:00 [info] <0.230.0> Running boot step rabbit_observer_cli defined by app rabbit
2023-05-15 16:56:02.690779+00:00 [info] <0.230.0> Running boot step rabbit_looking_glass defined by app rabbit
2023-05-15 16:56:02.690852+00:00 [info] <0.230.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2023-05-15 16:56:02.691072+00:00 [info] <0.230.0> Running boot step background_gc defined by app rabbit
2023-05-15 16:56:02.691360+00:00 [info] <0.230.0> Running boot step routing_ready defined by app rabbit
2023-05-15 16:56:02.691429+00:00 [info] <0.230.0> Running boot step pre_flight defined by app rabbit
2023-05-15 16:56:02.691496+00:00 [info] <0.230.0> Running boot step notify_cluster defined by app rabbit
2023-05-15 16:56:02.691564+00:00 [info] <0.230.0> Running boot step networking defined by app rabbit
2023-05-15 16:56:02.691631+00:00 [info] <0.230.0> Running boot step definition_import_worker_pool defined by app rabbit
2023-05-15 16:56:02.691747+00:00 [info] <0.287.0> Starting worker pool 'definition_import_pool' with 8 processes in it
2023-05-15 16:56:02.692530+00:00 [info] <0.230.0> Running boot step cluster_name defined by app rabbit
2023-05-15 16:56:02.692644+00:00 [info] <0.230.0> Initialising internal cluster ID to 'rabbitmq-cluster-id-qjzSiLuVlRuUABP3dETcTw'
2023-05-15 16:56:02.694373+00:00 [info] <0.230.0> Running boot step direct_client defined by app rabbit
2023-05-15 16:56:02.694536+00:00 [info] <0.230.0> Running boot step rabbit_maintenance_mode_state defined by app rabbit
2023-05-15 16:56:02.694617+00:00 [info] <0.230.0> Creating table rabbit_node_maintenance_states for maintenance mode status
2023-05-15 16:56:02.698077+00:00 [info] <0.230.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
2023-05-15 16:56:02.698280+00:00 [info] <0.741.0> Resetting node maintenance status
2023-05-15 16:56:02.712338+00:00 [info] <0.800.0> Management plugin: HTTP (non-TLS) listener started on port 15672
2023-05-15 16:56:02.712517+00:00 [info] <0.828.0> Statistics database started.
2023-05-15 16:56:02.712618+00:00 [info] <0.827.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2023-05-15 16:56:02.722748+00:00 [info] <0.842.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
2023-05-15 16:56:02.722919+00:00 [info] <0.741.0> Ready to start client connection listeners
2023-05-15 16:56:02.724796+00:00 [info] <0.886.0> started TCP listener on [::]:5672
completed with 4 plugins.
2023-05-15 16:56:02.978067+00:00 [info] <0.741.0> Server startup complete; 4 plugins started.
2023-05-15 16:56:02.978067+00:00 [info] <0.741.0> * rabbitmq_prometheus
2023-05-15 16:56:02.978067+00:00 [info] <0.741.0> * rabbitmq_management
2023-05-15 16:56:02.978067+00:00 [info] <0.741.0> * rabbitmq_web_dispatch
2023-05-15 16:56:02.978067+00:00 [info] <0.741.0> * rabbitmq_management_agent
2023-05-15 16:56:05.065247+00:00 [info] <0.889.0> accepting AMQP connection <0.889.0> (172.22.0.9:35298 -> 172.22.0.3:5672)
2023-05-15 16:56:05.066148+00:00 [info] <0.892.0> accepting AMQP connection <0.892.0> (172.22.0.9:35310 -> 172.22.0.3:5672)
2023-05-15 16:56:05.067432+00:00 [info] <0.895.0> accepting AMQP connection <0.895.0> (172.22.0.9:35318 -> 172.22.0.3:5672)
2023-05-15 16:56:05.069258+00:00 [info] <0.898.0> accepting AMQP connection <0.898.0> (172.22.0.9:35320 -> 172.22.0.3:5672)
2023-05-15 16:56:05.069998+00:00 [info] <0.901.0> accepting AMQP connection <0.901.0> (172.22.0.9:35332 -> 172.22.0.3:5672)
2023-05-15 16:56:05.070258+00:00 [info] <0.904.0> accepting AMQP connection <0.904.0> (172.22.0.9:35342 -> 172.22.0.3:5672)
2023-05-15 16:56:05.071782+00:00 [info] <0.889.0> connection <0.889.0> (172.22.0.9:35298 -> 172.22.0.3:5672): user 'aleph-p2p' authenticated and granted access to vhost '/'
2023-05-15 16:56:05.072201+00:00 [info] <0.895.0> connection <0.895.0> (172.22.0.9:35318 -> 172.22.0.3:5672): user 'aleph-p2p' authenticated and granted access to vhost '/'
2023-05-15 16:56:05.073374+00:00 [info] <0.892.0> connection <0.892.0> (172.22.0.9:35310 -> 172.22.0.3:5672): user 'aleph-p2p' authenticated and granted access to vhost '/'
2023-05-15 16:56:05.074962+00:00 [info] <0.942.0> accepting AMQP connection <0.942.0> (172.22.0.9:35344 -> 172.22.0.3:5672)
2023-05-15 16:56:05.075145+00:00 [info] <0.898.0> connection <0.898.0> (172.22.0.9:35320 -> 172.22.0.3:5672): user 'aleph-p2p' authenticated and granted access to vhost '/'
2023-05-15 16:56:05.075681+00:00 [info] <0.901.0> connection <0.901.0> (172.22.0.9:35332 -> 172.22.0.3:5672): user 'aleph-p2p' authenticated and granted access to vhost '/'
2023-05-15 16:56:05.075417+00:00 [info] <0.904.0> connection <0.904.0> (172.22.0.9:35342 -> 172.22.0.3:5672): user 'aleph-p2p' authenticated and granted access to vhost '/'
2023-05-15 16:56:05.079157+00:00 [info] <0.971.0> accepting AMQP connection <0.971.0> (172.22.0.9:35354 -> 172.22.0.3:5672)
2023-05-15 16:56:05.081360+00:00 [info] <0.942.0> connection <0.942.0> (172.22.0.9:35344 -> 172.22.0.3:5672): user 'aleph-p2p' authenticated and granted access to vhost '/'
2023-05-15 16:56:05.083036+00:00 [info] <0.971.0> connection <0.971.0> (172.22.0.9:35354 -> 172.22.0.3:5672): user 'aleph-p2p' authenticated and granted access to vhost '/'
2023-05-15 16:56:09.185963+00:00 [warning] <0.901.0> closing AMQP connection <0.901.0> (172.22.0.9:35332 -> 172.22.0.3:5672, vhost: '/', user: 'aleph-p2p'):
2023-05-15 16:56:09.185963+00:00 [warning] <0.901.0> client unexpectedly closed TCP connection
2023-05-15 16:56:09.189866+00:00 [warning] <0.942.0> closing AMQP connection <0.942.0> (172.22.0.9:35344 -> 172.22.0.3:5672, vhost: '/', user: 'aleph-p2p'):
2023-05-15 16:56:09.189866+00:00 [warning] <0.942.0> client unexpectedly closed TCP connection
2023-05-15 16:56:09.279338+00:00 [warning] <0.892.0> closing AMQP connection <0.892.0> (172.22.0.9:35310 -> 172.22.0.3:5672, vhost: '/', user: 'aleph-p2p'):
2023-05-15 16:56:09.279338+00:00 [warning] <0.892.0> client unexpectedly closed TCP connection
2023-05-15 16:56:09.356685+00:00 [warning] <0.889.0> closing AMQP connection <0.889.0> (172.22.0.9:35298 -> 172.22.0.3:5672, vhost: '/', user: 'aleph-p2p'):
2023-05-15 16:56:09.356685+00:00 [warning] <0.889.0> client unexpectedly closed TCP connection
2023-05-15 16:56:09.470903+00:00 [warning] <0.898.0> closing AMQP connection <0.898.0> (172.22.0.9:35320 -> 172.22.0.3:5672, vhost: '/', user: 'aleph-p2p'):
2023-05-15 16:56:09.470903+00:00 [warning] <0.898.0> client unexpectedly closed TCP connection
2023-05-15 16:56:09.928003+00:00 [warning] <0.904.0> closing AMQP connection <0.904.0> (172.22.0.9:35342 -> 172.22.0.3:5672, vhost: '/', user: 'aleph-p2p'):
2023-05-15 16:56:09.928003+00:00 [warning] <0.904.0> client unexpectedly closed TCP connection
2023-05-15 16:56:09.970666+00:00 [warning] <0.971.0> closing AMQP connection <0.971.0> (172.22.0.9:35354 -> 172.22.0.3:5672, vhost: '/', user: 'aleph-p2p'):
2023-05-15 16:56:09.970666+00:00 [warning] <0.971.0> client unexpectedly closed TCP connection
2023-05-15 16:56:10.029611+00:00 [warning] <0.895.0> closing AMQP connection <0.895.0> (172.22.0.9:35318 -> 172.22.0.3:5672, vhost: '/', user: 'aleph-p2p'):
2023-05-15 16:56:10.029611+00:00 [warning] <0.895.0> client unexpectedly closed TCP connection
2023-05-15 17:03:02.672790+00:00 [info] <0.1061.0> accepting AMQP connection <0.1061.0> (172.22.0.8:46784 -> 172.22.0.3:5672)
2023-05-15 17:03:02.675700+00:00 [info] <0.1061.0> connection <0.1061.0> (172.22.0.8:46784 -> 172.22.0.3:5672): user 'aleph-p2p' authenticated and granted access to vhost '/'