Connection and Queue tab details are missing when rabbitmq is deployment in kubernetes environment

[pachalk]

We have two environments where Rabbitmq servers are deployed one is docker-compose and other is kubernetes clusters, we have same configurations for both the environments but we are not getting details when clicking on "Connection" and "Queue" tab in kubernetes environment but its working in docker-compose.
In kubernetes we are using rabbitmq:3.8.5-alpine, can anyone help me to resolve this issue

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

There isn't a whole lot of information to work with. You haven't shared any details on how you provisioned any of the clusters,
what effective configuration they run, whether rabbitmqadmin against the same API endpoints demonstrates the same difference, no server logs. We do not guess in this community.

I'd start with exploring the responses, perhaps they are not JSON but some kind of "access refused" HTML page served in your environment. Browser developer tools is the easiest option.

[pachalk]

In docker-compose deployed as a container with below configuration -
Dockerfile -
FROM rabbitmq:3.8.5-management-alpine

COPY rabbitmq.conf /etc/rabbitmq/
COPY definitions.json /etc/rabbitmq/

rabbitmq.conf -
default_user = guest
default_pass = guest
management.load_definitions = /etc/rabbitmq/definitions.json
loopback_users.guest = false
listeners.tcp.default = 5672
management.tcp.port = 15672
heartbeat = 5

defnitions.json -
{
"global_parameters": [],
"users": [
{
"name": "containername",
"password": "passcode",
"tags": "administrator"
}
],
"vhosts": [
{
"name": "/"
}
],
"permissions": [
{
"user": "containername",
"vhost": "/",
"configure": ".",
"read": ".",
"write": ".*"
}

In kubernetes deployed as a statefulset with rabbitmq:3.8.5-alpine and remaining application pods will be connected using URL - "amqp://podname:password@{{ .fullnameOverride }}:{{ .rabbitmqNodePort }}"
rabbitmq.conf:
default_user = guest
default_pass = guest
loopback_users.guest = false
management.load_definitions = /etc/definitions/definitions.json

kubectl logs - we are getting only these logs
rabbitmqctl node_health_check and its HTTP API counterpart are DEPRECATED. See https://www.rabbitmq.com/monitoring.html#health-checks for replacement options.

[michaelklishin]

Can you please paste your actual deployment YAML and logs, and not a single line definition?

Can you please inspect the responses to GET /api/queues/{host} and similar endpoints called by the management UI?

We won't be able to help you without this information.

[michaelklishin]

This does a whole bunch of things that we generally do not recommend (but they are unlikely to affect HTTP API responses):

• It enables remote access for the default administrative user with known credentials. This is a terrible idea. You are asking for hackers to exploit you.
• It uses a really low heartbeat timeout
• It is 11 patch releases behind
• and it does not use our Kubernetes Operator so we can only guess what the deployment may involve (e.g. what kind of load balancers with what kind of settings)

[pachalk]

YAML file

apiVersion: v1
kind: Service
metadata:
name: message-queue
spec:
selector:
app: message-queue
ports:
- name: http
protocol: TCP
port: 15672
- name: amqp
protocol: TCP
port: 5672
targetPort: amqp

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: message-queue
labels:
app: message-queue
{{- if .Values.gitlabIntegration }}
annotations:
app.gitlab.com/app: {{ .Values.gitlabIntegration.app | quote }}
app.gitlab.com/env: {{ .Values.gitlabIntegration.env | quote }}
{{- end }}
spec:
podManagementPolicy: OrderedReady
serviceName: message-queue
replicas: 1
updateStrategy:
type: OnDelete
selector:
matchLabels:
app: message-queue
template:
metadata:
labels:
app: message-queue
spec:
{{- if .Values.imagepullsecretsWeb.enabled }}
imagePullSecrets:
- name: {{ .Values.imagepullsecretsWeb.imagePullSecret.secretName }}
{{- end }}
terminationGracePeriodSeconds: 10
securityContext:
fsGroup: 101
runAsGroup: 101
runAsNonRoot: true
runAsUser: 100
initContainers:
- name: bootstrap
image: {{ .Values.defaultContainers.busybox }}
imagePullPolicy: IfNotPresent
securityContext:
runAsGroup: 0
runAsNonRoot: false
runAsUser: 0
command: ['sh']
args:
- "-c"
- |
set -ex
cp /configmap/* /etc/rabbitmq
echo "${RABBITMQ_ERLANG_COOKIE}" > /var/lib/rabbitmq/.erlang.cookie
chown -R 100:101 /var/lib/rabbitmq/
chown -R 100:101 /etc/rabbitmq
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: RABBITMQ_ERLANG_COOKIE
valueFrom:
secretKeyRef:
name: message-queue
key: rabbitmq-erlang-cookie
resources:
{}
volumeMounts:
- name: configmap
mountPath: /configmap
- name: config
mountPath: /etc/rabbitmq
- name: rabbitmqdata
mountPath: /var/lib/rabbitmq
containers:
- name: message-queue
image: rabbitmq:3.8.5-alpine
imagePullPolicy: IfNotPresent
ports:
- name: epmd
protocol: TCP
containerPort: 4369
- name: amqp
protocol: TCP
containerPort: 5672
- name: http
protocol: TCP
containerPort: 15672
livenessProbe:
exec:
command:
- /bin/sh
- -c
- 'timeout 5 wget -O - -q --header "Authorization: Basic echo -n \"$RABBIT_MANAGEMENT_USER:$RABBIT_MANAGEMENT_PASSWORD\" | base64" http://localhost:15672/api/healthchecks/node | grep -qF "{"status":"ok"}"'
failureThreshold: 6
initialDelaySeconds: 120
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
exec:
command:
- /bin/sh
- -c
- 'timeout 3 wget -O - -q --header "Authorization: Basic echo -n \"$RABBIT_MANAGEMENT_USER:$RABBIT_MANAGEMENT_PASSWORD\" | base64" http://localhost:15672/api/healthchecks/node | grep -qF "{"status":"ok"}"'
failureThreshold: 6
initialDelaySeconds: 20
periodSeconds: 5
timeoutSeconds: 3
resources:
requests:
memory: "2Gi"
cpu: "1000m"
limits:
memory: "4Gi"
cpu: "2000m"
env:
- name: MY_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: RABBITMQ_USE_LONGNAME
value: "true"
- name: RABBITMQ_NODENAME
value: rabbit@$(MY_POD_NAME).message-queue.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}
- name: K8S_HOSTNAME_SUFFIX
value: message-queue.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}
- name: K8S_SERVICE_NAME
value: message-queue
- name: RABBITMQ_ERLANG_COOKIE
valueFrom:
secretKeyRef:
name: message-queue
key: rabbitmq-erlang-cookie
- name: RABBIT_MANAGEMENT_USER
valueFrom:
secretKeyRef:
name: message-queue
key: rabbitmq-management-username
- name: RABBIT_MANAGEMENT_PASSWORD
valueFrom:
secretKeyRef:
name: message-queue
key: rabbitmq-management-password
volumeMounts:
- name: rabbitmqdata
mountPath: /var/lib/rabbitmq
- name: config
mountPath: /etc/rabbitmq
- name: definitions
mountPath: /etc/definitions
readOnly: true
volumes:
- name: config
emptyDir: {}
- name: configmap
configMap:
name: message-queue
- name: definitions
secret:
secretName: message-queue
items:
- key: definitions.json
path: definitions.json
- name: rabbitmqdata
emptyDir: {}

apiVersion: v1
kind: Secret
metadata:
name: message-queue
labels:
app: message-queue
type: Opaque
data:
{{- $password := .Values.messagequeue.rabbitmqPassword | default (randAlphaNum 24 | nospace) -}}
{{- $_ := set .Values "messagequeue.rabbitmqPassword" $password }}
{{- $managementPassword := .Values.managementPassword | default (randAlphaNum 24 | nospace) -}}
{{- $_ := set .Values "messagequeue.managementPassword" $managementPassword }}
rabbitmq-username: {{ .Values.messagequeue.rabbitmqUsername | b64enc | quote }}
rabbitmq-password: {{ .Values.messagequeue.rabbitmqPassword | b64enc | quote }}
rabbitmq-management-username: {{ .Values.messagequeue.managementUsername | b64enc | quote }}
rabbitmq-management-password: {{ .Values.messagequeue.managementPassword | b64enc | quote }}
rabbitmq-erlang-cookie: {{ .Values.messagequeue.rabbitmqErlangCookie | default (randAlphaNum 32) | b64enc | quote }}
definitions.json: "passwords for services decoded to base64"

apiVersion: v1
kind: ConfigMap
metadata:
name: message-queue
labels:
app: message-queue
data:
enabled_plugins: |
[
rabbitmq_shovel,
rabbitmq_shovel_management,
rabbitmq_federation,
rabbitmq_federation_management,
rabbitmq_consistent_hash_exchange,
rabbitmq_management,
rabbitmq_peer_discovery_k8s
].
rabbitmq.conf: |
## RabbitMQ configuration
## Ref: https://github.com/rabbitmq/rabbitmq-server/blob/master/docs/rabbitmq.conf.example

## Authentification

## Clustering
#cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
#cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
#cluster_formation.k8s.address_type = hostname
#cluster_formation.node_cleanup.interval = 10
# Set to false if automatic cleanup of absent nodes is desired.
# This can be dangerous, see http://www.rabbitmq.com/cluster-formation.html#node-health-checks-and-cleanup.
#cluster_formation.node_cleanup.only_log_warning = true
#cluster_partition_handling = autoheal
## The default "guest" user is only permitted to access the server
# Default user
default_user = guest
default_pass = guest
## via a loopback interface (e.g. localhost)
loopback_users.guest = false

management.load_definitions = /etc/definitions/definitions.json

## Memory-based Flow Control threshold
vm_memory_high_watermark.absolute = 1GB

## Auth HTTP Backend Plugin

## LDAP Plugin

## MQTT Plugin

## Web MQTT Plugin

## STOMP Plugin

## Web STOMP Plugin

## Prometheus Plugin

## AMQPS support

heartbeat = 20

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-message-queue
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.org/mergeable-ingress-type: "minion"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:

• host: {{ .Values.ingress.host | quote }}
  http:
  paths:
  • path: /rabbitmq(/|$)(.*)
    backend:
    serviceName: message-queue
    servicePort: 15672

rabbitmq container logs
[debug] <0.283.0> Lager installed handler error_logger_lager_h into error_logger
[debug] <0.286.0> Lager installed handler lager_forwarder_backend into error_logger_lager_event
[debug] <0.289.0> Lager installed handler lager_forwarder_backend into rabbit_log_lager_event
[debug] <0.292.0> Lager installed handler lager_forwarder_backend into rabbit_log_channel_lager_event
[debug] <0.295.0> Lager installed handler lager_forwarder_backend into rabbit_log_connection_lager_event
debug] <0.298.0> Lager installed handler lager_forwarder_backend into rabbit_log_feature_flags_lager_event
[debug] <0.301.0> Lager installed handler lager_forwarder_backend into rabbit_log_federation_lager_event
[debug] <0.304.0> Lager installed handler lager_forwarder_backend into rabbit_log_ldap_lager_event
[debug] <0.307.0> Lager installed handler lager_forwarder_backend into rabbit_log_mirroring_lager_event
[debug] <0.322.0> Lager installed handler lager_forwarder_backend into rabbit_log_upgrade_lager_event
[debug] <0.310.0> Lager installed handler lager_forwarder_backend into rabbit_log_prelaunch_lager_event
[debug] <0.313.0> Lager installed handler lager_forwarder_backend into rabbit_log_queue_lager_event
[debug] <0.316.0> Lager installed handler lager_forwarder_backend into rabbit_log_ra_lager_event
[debug] <0.319.0> Lager installed handler lager_forwarder_backend into rabbit_log_shovel_lager_event
[info] <0.268.0> HiPE disabled: no modules were natively recompiled.
[info] <0.268.0>
Starting RabbitMQ 3.8.5 on Erlang 23.0.3
Copyright (c) 2007-2020 VMware, Inc. or its affiliates.
Licensed under the MPL 1.1. Website: https://rabbitmq.com

## RabbitMQ 3.8.5

########## Copyright (c) 2007-2020 VMware, Inc. or its affiliates.

########## Licensed under the MPL 1.1. Website: https://rabbitmq.com

Doc guides: https://rabbitmq.com/documentation.html
Support: https://rabbitmq.com/contact.html
Tutorials: https://rabbitmq.com/getstarted.html
Monitoring: https://rabbitmq.com/monitoring.html

Logs:

Config file(s): /etc/rabbitmq/rabbitmq.conf

Starting broker...2021-05-23 01:12:20.774 [info] <0.268.0>
node : [email protected]
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.conf
cookie hash : tbxSkl
log(s) :
database dir : /var/lib/rabbitmq/mnesia/[email protected]
[debug] <0.279.0> Lager installed handler lager_backend_throttle into lager_event
[info] <0.268.0> Running boot step pre_boot defined by app rabbit
[info] <0.268.0> Running boot step rabbit_core_metrics defined by app rabbit
[info] <0.268.0> Running boot step rabbit_alarm defined by app rabbit
[info] <0.350.0> Memory high watermark set to 953 MiB (1000000000 bytes) of 385449 MiB (404173434880 bytes) total
[info] <0.352.0> Enabling free disk space monitoring
[info] <0.352.0> Disk free limit set to 50MB
[info] <0.268.0> Running boot step code_server_cache defined by app rabbit
[info] <0.268.0> Running boot step file_handle_cache defined by app rabbit
[info] <0.355.0> Limiting to approx 1048479 file handles (943629 sockets)
[info] <0.356.0> FHC read buffering: OFF
[info] <0.356.0> FHC write buffering: ON
[info] <0.268.0> Running boot step worker_pool defined by app rabbit
[info] <0.342.0> Will use 88 processes for default worker pool
[info] <0.342.0> Starting worker pool 'worker_pool' with 88 processes in it
[info] <0.268.0> Running boot step database defined by app rabbit
[info] <0.268.0> Node database directory at /var/lib/rabbitmq/mnesia/[email protected] is empty. Assuming we need to join an existing cluster or initialise from scratch...
[info] <0.268.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
[info] <0.268.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
[info] <0.268.0> Peer discovery backend does not support locking, falling back to randomized delay
[info] <0.268.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping randomized startup delay.
[info] <0.268.0> All discovered existing cluster peers:
[info] <0.268.0> Discovered no peer nodes to cluster with. Some discovery backends can filter nodes out based on a readiness criteria. Enabling debug logging might help troubleshoot.
[info] <0.44.0> Application mnesia exited with reason: stopped
[info] <0.268.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
[info] <0.268.0> Successfully synced tables from a peer
[info] <0.268.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
[info] <0.268.0> Successfully synced tables from a peer
[info] <0.268.0> Feature flag implicit_default_bindings: supported, attempt to enable...
[info] <0.268.0> Feature flag implicit_default_bindings: mark as enabled=state_changing
[info] <0.268.0> Feature flags: list of feature flags found:
2021-05-23 01:12:22.333 [info] <0.268.0> Feature flags: [] implicit_default_bindings
2021-05-23 01:12:22.333 [info] <0.268.0> Feature flags: [ ] quorum_queue
2021-05-23 01:12:22.333 [info] <0.268.0> Feature flags: [ ] virtual_host_metadata
2021-05-23 01:12:22.333 [info] <0.268.0> Feature flags: feature flag states written to disk: yes
2021-05-23 01:12:22.344 [info] <0.268.0> Waiting for Mnesia tables for 30000 ms, 0 retries left
2021-05-23 01:12:22.345 [info] <0.268.0> Successfully synced tables from a peer
2021-05-23 01:12:22.345 [info] <0.268.0> Feature flag implicit_default_bindings: mark as enabled=true
2021-05-23 01:12:22.353 [info] <0.268.0> Feature flags: list of feature flags found:
2021-05-23 01:12:22.353 [info] <0.268.0> Feature flags: [x] implicit_default_bindings
2021-05-23 01:12:22.353 [info] <0.268.0> Feature flags: [ ] quorum_queue
2021-05-23 01:12:22.354 [info] <0.268.0> Feature flags: [ ] virtual_host_metadata
2021-05-23 01:12:22.354 [info] <0.268.0> Feature flags: feature flag states written to disk: yes
2021-05-23 01:12:22.373 [info] <0.268.0> Feature flag quorum_queue: supported, attempt to enable...
2021-05-23 01:12:22.373 [info] <0.268.0> Feature flag quorum_queue: mark as enabled=state_changing
2021-05-23 01:12:22.381 [info] <0.268.0> Feature flags: list of feature flags found:
2021-05-23 01:12:22.381 [info] <0.268.0> Feature flags: [x] implicit_default_bindings
2021-05-23 01:12:22.381 [info] <0.268.0> Feature flags: [] quorum_queue
2021-05-23 01:12:22.382 [info] <0.268.0> Feature flags: [ ] virtual_host_metadata
2021-05-23 01:12:22.382 [info] <0.268.0> Feature flags: feature flag states written to disk: yes
2021-05-23 01:12:22.391 [info] <0.268.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-05-23 01:12:22.392 [info] <0.268.0> Successfully synced tables from a peer
2021-05-23 01:12:22.392 [info] <0.268.0> Feature flag quorum_queue: migrating Mnesia table rabbit_queue...
2021-05-23 01:12:22.413 [info] <0.268.0> Feature flag quorum_queue: migrating Mnesia table rabbit_durable_queue...
2021-05-23 01:12:22.448 [info] <0.268.0> Feature flag quorum_queue: Mnesia tables migration done
2021-05-23 01:12:22.448 [info] <0.268.0> Feature flag quorum_queue: mark as enabled=true
2021-05-23 01:12:22.456 [info] <0.268.0> Feature flags: list of feature flags found:
2021-05-23 01:12:22.456 [info] <0.268.0> Feature flags: [x] implicit_default_bindings
2021-05-23 01:12:22.456 [info] <0.268.0> Feature flags: [x] quorum_queue
2021-05-23 01:12:22.456 [info] <0.268.0> Feature flags: [ ] virtual_host_metadata
2021-05-23 01:12:22.456 [info] <0.268.0> Feature flags: feature flag states written to disk: yes
2021-05-23 01:12:22.467 [info] <0.268.0> Feature flag virtual_host_metadata: supported, attempt to enable...
2021-05-23 01:12:22.467 [info] <0.268.0> Feature flag virtual_host_metadata: mark as enabled=state_changing
2021-05-23 01:12:22.474 [info] <0.268.0> Feature flags: list of feature flags found:
2021-05-23 01:12:22.474 [info] <0.268.0> Feature flags: [x] implicit_default_bindings
2021-05-23 01:12:22.474 [info] <0.268.0> Feature flags: [x] quorum_queue
2021-05-23 01:12:22.474 [info] <0.268.0> Feature flags: [~] virtual_host_metadata
2021-05-23 01:12:22.474 [info] <0.268.0> Feature flags: feature flag states written to disk: yes
2021-05-23 01:12:22.494 [info] <0.268.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-05-23 01:12:22.494 [info] <0.268.0> Successfully synced tables from a peer
2021-05-23 01:12:22.525 [info] <0.268.0> Feature flag virtual_host_metadata: mark as enabled=true
2021-05-23 01:12:22.542 [info] <0.268.0> Feature flags: list of feature flags found:
2021-05-23 01:12:22.542 [info] <0.268.0> Feature flags: [x] implicit_default_bindings
2021-05-23 01:12:22.542 [info] <0.268.0> Feature flags: [x] quorum_queue
2021-05-23 01:12:22.542 [info] <0.268.0> Feature flags: [x] virtual_host_metadata
2021-05-23 01:12:22.542 [info] <0.268.0> Feature flags: feature flag states written to disk: yes
2021-05-23 01:12:22.555 [info] <0.268.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-05-23 01:12:22.555 [info] <0.268.0> Successfully synced tables from a peer
2021-05-23 01:12:22.584 [info] <0.268.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-05-23 01:12:22.585 [info] <0.268.0> Successfully synced tables from a peer
2021-05-23 01:12:22.585 [info] <0.268.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2021-05-23 01:12:22.585 [info] <0.268.0> Running boot step database_sync defined by app rabbit
2021-05-23 01:12:22.585 [info] <0.268.0> Running boot step feature_flags defined by app rabbit
2021-05-23 01:12:22.585 [info] <0.268.0> Running boot step codec_correctness_check defined by app rabbit
2021-05-23 01:12:22.585 [info] <0.268.0> Running boot step external_infrastructure defined by app rabbit
2021-05-23 01:12:22.585 [info] <0.268.0> Running boot step rabbit_registry defined by app rabbit
2021-05-23 01:12:22.585 [info] <0.268.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2021-05-23 01:12:22.585 [info] <0.268.0> Running boot step rabbit_queue_location_random defined by app rabbit
2021-05-23 01:12:22.585 [info] <0.268.0> Running boot step rabbit_event defined by app rabbit
2021-05-23 01:12:22.585 [info] <0.268.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2021-05-23 01:12:22.586 [info] <0.268.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2021-05-23 01:12:22.586 [info] <0.268.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2021-05-23 01:12:22.586 [info] <0.268.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2021-05-23 01:12:22.586 [info] <0.268.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2021-05-23 01:12:22.586 [info] <0.268.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2021-05-23 01:12:22.587 [info] <0.268.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2021-05-23 01:12:22.587 [info] <0.268.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2021-05-23 01:12:22.587 [info] <0.268.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2021-05-23 01:12:22.587 [info] <0.268.0> Running boot step rabbit_priority_queue defined by app rabbit
2021-05-23 01:12:22.587 [info] <0.268.0> Priority queues enabled, real BQ is rabbit_variable_queue
2021-05-23 01:12:22.588 [info] <0.268.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2021-05-23 01:12:22.588 [info] <0.268.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2021-05-23 01:12:22.588 [info] <0.268.0> Running boot step kernel_ready defined by app rabbit
2021-05-23 01:12:22.588 [info] <0.268.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2021-05-23 01:12:22.588 [info] <0.268.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2021-05-23 01:12:22.589 [info] <0.665.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2021-05-23 01:12:22.589 [info] <0.268.0> Running boot step guid_generator defined by app rabbit
2021-05-23 01:12:22.590 [info] <0.268.0> Running boot step rabbit_node_monitor defined by app rabbit
2021-05-23 01:12:22.590 [info] <0.669.0> Starting rabbit_node_monitor
2021-05-23 01:12:22.590 [info] <0.268.0> Running boot step delegate_sup defined by app rabbit
2021-05-23 01:12:22.591 [info] <0.268.0> Running boot step rabbit_memory_monitor defined by app rabbit
2021-05-23 01:12:22.591 [info] <0.268.0> Running boot step core_initialized defined by app rabbit
2021-05-23 01:12:22.591 [info] <0.268.0> Running boot step upgrade_queues defined by app rabbit
2021-05-23 01:12:22.606 [info] <0.268.0> message_store upgrades: 1 to apply
2021-05-23 01:12:22.606 [info] <0.268.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2021-05-23 01:12:22.606 [info] <0.268.0> message_store upgrades: No durable queues found. Skipping message store migration
2021-05-23 01:12:22.606 [info] <0.268.0> message_store upgrades: Removing the old message store data
2021-05-23 01:12:22.606 [info] <0.268.0> message_store upgrades: All upgrades applied successfully
2021-05-23 01:12:22.621 [info] <0.268.0> Running boot step rabbit_connection_tracking defined by app rabbit
2021-05-23 01:12:22.621 [info] <0.268.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2021-05-23 01:12:22.621 [info] <0.268.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2021-05-23 01:12:22.621 [info] <0.268.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2021-05-23 01:12:22.622 [info] <0.268.0> Running boot step rabbit_policies defined by app rabbit
2021-05-23 01:12:22.622 [info] <0.268.0> Running boot step rabbit_policy defined by app rabbit
2021-05-23 01:12:22.623 [info] <0.268.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2021-05-23 01:12:22.623 [info] <0.268.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2021-05-23 01:12:22.623 [info] <0.268.0> Running boot step rabbit_vhost_limit defined by app rabbit
2021-05-23 01:12:22.623 [info] <0.268.0> Running boot step recovery defined by app rabbit
2021-05-23 01:12:22.623 [info] <0.268.0> Running boot step definition_import_worker_pool defined by app rabbit
2021-05-23 01:12:22.623 [info] <0.342.0> Starting worker pool 'definition_import_pool' with 88 processes in it
2021-05-23 01:12:22.626 [info] <0.268.0> Running boot step load_core_definitions defined by app rabbit
2021-05-23 01:12:22.626 [info] <0.268.0> Running boot step empty_db_check defined by app rabbit
2021-05-23 01:12:22.626 [info] <0.268.0> Adding vhost '/' (description: 'Default virtual host')
2021-05-23 01:12:22.685 [info] <0.796.0> Making sure data directory '/var/lib/rabbitmq/mnesia/[email protected]/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2021-05-23 01:12:22.704 [info] <0.796.0> Starting message stores for vhost '/'
2021-05-23 01:12:22.704 [info] <0.800.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2021-05-23 01:12:22.706 [info] <0.796.0> Started message store of type transient for vhost '/'
2021-05-23 01:12:22.707 [info] <0.804.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2021-05-23 01:12:22.708 [warning] <0.804.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2021-05-23 01:12:22.710 [info] <0.796.0> Started message store of type persistent for vhost '/'
2021-05-23 01:12:22.731 [info] <0.268.0> Created user 'guest'
2021-05-23 01:12:22.748 [info] <0.268.0> Successfully set user tags for user 'guest' to [administrator]
2021-05-23 01:12:22.756 [info] <0.268.0> Successfully set permissions for 'guest' in virtual host '/' to '.', '.', '.'
2021-05-23 01:12:22.756 [info] <0.268.0> Running boot step rabbit_looking_glass defined by app rabbit
2021-05-23 01:12:22.756 [info] <0.268.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2021-05-23 01:12:22.756 [info] <0.268.0> Running boot step background_gc defined by app rabbit
2021-05-23 01:12:22.757 [info] <0.268.0> Running boot step connection_tracking defined by app rabbit
2021-05-23 01:12:22.765 [info] <0.268.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@message-queue-0.message-queue.namespace.svc.cluster.local'
2021-05-23 01:12:22.774 [info] <0.268.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@message-queue-0.message-queue.namespace.svc.cluster.local'
2021-05-23 01:12:22.775 [info] <0.268.0> Running boot step routing_ready defined by app rabbit
2021-05-23 01:12:22.775 [info] <0.268.0> Running boot step pre_flight defined by app rabbit
2021-05-23 01:12:22.775 [info] <0.268.0> Running boot step notify_cluster defined by app rabbit
2021-05-23 01:12:22.775 [info] <0.268.0> Running boot step networking defined by app rabbit
2021-05-23 01:12:22.781 [info] <0.851.0> started TCP listener on [::]:5672
2021-05-23 01:12:22.782 [info] <0.268.0> Running boot step cluster_name defined by app rabbit
2021-05-23 01:12:22.782 [info] <0.268.0> Initialising internal cluster ID to 'rabbitmq-cluster-id-__TgJmJI'
2021-05-23 01:12:22.784 [info] <0.268.0> Running boot step direct_client defined by app rabbit
2021-05-23 01:12:23.347 [info] <0.854.0> Feature flags: list of feature flags found:
2021-05-23 01:12:23.347 [info] <0.854.0> Feature flags: [ ] drop_unroutable_metric
2021-05-23 01:12:23.347 [info] <0.854.0> Feature flags: [ ] empty_basic_get_metric
2021-05-23 01:12:23.347 [info] <0.854.0> Feature flags: [x] implicit_default_bindings
2021-05-23 01:12:23.347 [info] <0.854.0> Feature flags: [x] quorum_queue
2021-05-23 01:12:23.347 [info] <0.854.0> Feature flags: [x] virtual_host_metadata
2021-05-23 01:12:23.347 [info] <0.854.0> Feature flags: feature flag states written to disk: yes
2021-05-23 01:12:23.481 [info] <0.854.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2021-05-23 01:12:23.481 [info] <0.854.0> Management plugin: using rates mode 'basic'
2021-05-23 01:12:23.637 [info] <0.854.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2021-05-23 01:12:23.637 [info] <0.854.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
2021-05-23 01:12:23.637 [info] <0.854.0> Applying definitions from /etc/definitions/definitions.json
2021-05-23 01:12:23.637 [info] <0.854.0> Asked to import definitions. Acting user: rmq-internal
2021-05-23 01:12:23.637 [info] <0.854.0> Importing concurrently 8 users...
2021-05-23 01:12:23.639 [info] <0.705.0> Created user 'user1'
2021-05-23 01:12:23.639 [info] <0.703.0> Created user 'user2'
2021-05-23 01:12:23.639 [info] <0.699.0> Created user 'user3'
2021-05-23 01:12:23.639 [info] <0.700.0> Created user 'user4'
2021-05-23 01:12:23.648 [info] <0.701.0> Created user 'user5'
2021-05-23 01:12:23.648 [info] <0.704.0> Created user 'user6'
2021-05-23 01:12:23.648 [info] <0.698.0> Created user 'management'
2021-05-23 01:12:23.649 [info] <0.702.0> Created user 'user7'
2021-05-23 01:12:23.657 [info] <0.699.0> Successfully set user tags for user 'user1' to [administrator]
2021-05-23 01:12:23.657 [info] <0.703.0> Successfully set user tags for user 'user2' to [administrator]
2021-05-23 01:12:23.657 [info] <0.700.0> Successfully set user tags for user 'user3' to [administrator]
2021-05-23 01:12:23.665 [info] <0.705.0> Successfully set user tags for user 'user4' to [administrator]
2021-05-23 01:12:23.665 [info] <0.698.0> Successfully set user tags for user 'management' to [management]
2021-05-23 01:12:23.665 [info] <0.704.0> Successfully set user tags for user 'user5' to [administrator]
2021-05-23 01:12:23.665 [info] <0.701.0> Successfully set user tags for user 'user6' to [administrator]
2021-05-23 01:12:23.665 [info] <0.702.0> Successfully set user tags for user 'user7' to [administrator]
2021-05-23 01:12:23.666 [info] <0.854.0> Importing concurrently 1 vhosts...
2021-05-23 01:12:23.666 [info] <0.854.0> Importing concurrently 7 permissions...
2021-05-23 01:12:23.673 [info] <0.698.0> Successfully set permissions for 'user1' in virtual host '/' to '.', '.', '.'
2021-05-23 01:12:23.682 [info] <0.700.0> Successfully set permissions for 'user2' in virtual host '/' to '.', '.', '.'
2021-05-23 01:12:23.682 [info] <0.701.0> Successfully set permissions for 'user3' in virtual host '/' to '.', '.', '.'
2021-05-23 01:12:23.682 [info] <0.699.0> Successfully set permissions for 'user4' in virtual host '/' to '.', '.', '.'
2021-05-23 01:12:23.682 [info] <0.703.0> Successfully set permissions for 'user5' in virtual host '/' to '.', '.', '.'
2021-05-23 01:12:23.682 [info] <0.704.0> Successfully set permissions for 'user6' in virtual host '/' to '.', '.', '.'
2021-05-23 01:12:23.682 [info] <0.702.0> Successfully set permissions for 'user7' in virtual host '/' to '.', '.', '.'
2021-05-23 01:12:23.735 [info] <0.927.0> Management plugin: HTTP (non-TLS) listener started on port 15672
2021-05-23 01:12:23.736 [info] <0.1033.0> Statistics database started.
2021-05-23 01:12:23.736 [info] <0.1032.0> Starting worker pool 'management_worker_pool' with 3 processes in it
[info] <0.854.0> Running boot step rabbit_exchange_type_consistent_hash_mnesia defined by app rabbitmq_consistent_hash_exchange
[info] <0.854.0> Running boot step rabbit_exchange_type_consistent_hash_registry defined by app rabbitmq_consistent_hash_exchange
[info] <0.854.0> Running boot step rabbit_federation_supervisor defined by app rabbitmq_federation
[info] <0.854.0> Running boot step rabbit_federation_exchange defined by app rabbitmq_federation
[info] <0.854.0> Running boot step rabbit_federation_parameters defined by app rabbitmq_federation
[info] <0.854.0> Running boot step rabbit_federation_queue defined by app rabbitmq_federation
[info] <0.854.0> Running boot step rabbit_federation_upstream_exchange defined by app rabbitmq_federation
[info] <0.854.0> Running boot step rabbit_shovel_parameters defined by app rabbitmq_shovel
[info] <0.1077.0> Peer discovery: node cleanup is disabled
[info] <0.854.0> Server startup complete; 10 plugins started.

• rabbitmq_peer_discovery_k8s
• rabbitmq_federation_management
• rabbitmq_peer_discovery_common
• rabbitmq_shovel_management
• rabbitmq_shovel
• rabbitmq_federation
• rabbitmq_consistent_hash_exchange
• rabbitmq_management
• rabbitmq_web_dispatch
• rabbitmq_management_agent
  completed with 10 plugins.
  [warning] <0.1106.0> rabbitmqctl node_health_check and its HTTP API counterpart are DEPRECATED. See https://www.rabbitmq.com/monitoring.html#health-checks for replacement options.
  [warning] <0.1113.0> rabbitmqctl node_health_check and its HTTP API counterpart are DEPRECATED. See https

Each user is connected with mq_uri

"amqp://user1:password@message-queue:5672"

i run rabbitmq commands in rabbitmq container like
rabbitmqctl list_vhosts
Listing vhosts ...
name
/

rabbitmqctl list_users
Listing users ...
user tags
user1 [administrator]
management [management]
user2 [administrator]
user3 [administrator]
user4 [administrator]
user5 [administrator]
user6 [administrator]
user6 [administrator]
user7 [administrator]

rabbitmqctl cluster_status
Cluster status of node [email protected] ...
Basics

Cluster name: [email protected]

Disk Nodes

[email protected]

Running Nodes

[email protected]

Versions

[email protected]: RabbitMQ 3.8.5 on Erlang 23.0.3

Alarms

(none)

Network Partitions

(none)

Listeners

Node: [email protected], interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: [email protected], interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: [email protected], interface: [::], port: 15672, protocol: http, purpose: HTTP API

Feature flags

Flag: drop_unroutable_metric, state: disabled
Flag: empty_basic_get_metric, state: disabled
Flag: implicit_default_bindings, state: enabled
Flag: quorum_queue, state: enabled
Flag: virtual_host_metadata, state: enabled

GET /api/queues/host - Result
curl -u guest:guest http://serviceIP:15672/api/queues/
[{"arguments":{},"auto_delete":false,"backing_queue_status":{"avg_ack_egress_rate":0.0,"avg_ack_ingress_rate":0.0,"avg_egress_rate":0.0,"avg_ingress_rate":0.0,"delta":["delta","undefined",0,0,"undefined"],"len":0,"mode":"default","next_seq_id":0,"q1":0,"q2":0,"q3":0,"q4":0,"target_ram_count":"infinity"},"consumer_utilisation":1.0,"consumers":1,"durable":true,"effective_policy_definition":{},"exclusive":false,"exclusive_consumer_tag":null,"garbage_collection":{"fullsweep_after":65535,"max_heap_size":0,"min_bin_vheap_size":46422,"min_heap_size":233,"minor_gcs":476},"head_message_timestamp":null,"memory":143196,"message_bytes":0,"message_bytes_paged_out":0,"message_bytes_persistent":0,"message_bytes_ram":0,"message_bytes_ready":0,"message_bytes_unacknowledged":0,"messages":0,"messages_details":{"rate":0.0},"messages_paged_out":0,"messages_persistent":0,"messages_ram":0,"messages_ready":0,"messages_ready_details":{"rate":0.0},"messages_ready_ram":0,"messages_unacknowledged":0,"messages_unacknowledged_details":{"rate":0.0},"messages_unacknowledged_ram":0,"name":"user4_jobs","node":"[email protected]","operator_policy":null,"policy":null,"recoverable_slaves":null,"reductions":22152744,"reductions_details":{"rate":227.4},"single_active_consumer_tag":null,"state":"running","type":"classic","vhost":"/"},{"arguments":{},"auto_delete":false,"backing_queue_status":{"avg_ack_egress_rate":0.0,"avg_ack_ingress_rate":0.0,"avg_egress_rate":0.0,"avg_ingress_rate":0.0,"delta":["delta","undefined",0,0,"undefined"],"len":0,"mode":"default","next_seq_id":0,"q1":0,"q2":0,"q3":0,"q4":0,"target_ram_count":"infinity"},"consumer_utilisation":1.0,"consumers":1,"durable":true,"effective_policy_definition":{},"exclusive":false,"exclusive_consumer_tag":null,"garbage_collection":{"fullsweep_after":65535,"max_heap_size":0,"min_bin_vheap_size":46422,"min_heap_size":233,"minor_gcs":1651},"head_message_timestamp":null,"memory":143196,"message_bytes":0,"message_bytes_paged_out":0,"message_bytes_persistent":0,"message_bytes_ram":0,"message_bytes_ready":0,"message_bytes_unacknowledged":0,"messages":0,"messages_details":{"rate":0.0},"messages_paged_out":0,"messages_persistent":0,"messages_ram":0,"messages_ready":0,"messages_ready_details":{"rate":0.0},"messages_ready_ram":0,"messages_unacknowledged":0,"messages_unacknowledged_details":{"rate":0.0},"messages_unacknowledged_ram":0,"name":"user3_jobs","node":"[email protected]","operator_policy":null,"policy":null,"recoverable_slaves":null,"reductions":22144455,"reductions_details":{"rate":227.4},"single_active_consumer_tag":null,"state":"running","type":"classic","vhost":"/"},{"arguments":{},"auto_delete":false,"backing_queue_status":{"avg_ack_egress_rate":0.0,"avg_ack_ingress_rate":0.0,"avg_egress_rate":0.0,"avg_ingress_rate":0.0,"delta":["delta","undefined",0,0,"undefined"],"len":0,"mode":"default","next_seq_id":0,"q1":0,"q2":0,"q3":0,"q4":0,"target_ram_count":"infinity"},"consumer_utilisation":1.0,"consumers":1,"durable":true,"effective_policy_definition":{},"exclusive":false,"exclusive_consumer_tag":null,"garbage_collection":{"fullsweep_after":65535,"max_heap_size":0,"min_bin_vheap_size":46422,"min_heap_size":233,"minor_gcs":5113},"head_message_timestamp":null,"memory":197332,"message_bytes":0,"message_bytes_paged_out":0,"message_bytes_persistent":0,"message_bytes_ram":0,"message_bytes_ready":0,"message_bytes_unacknowledged":0,"messages":0,"messages_details":{"rate":0.0},"messages_paged_out":0,"messages_persistent":0,"messages_ram":0,"messages_ready":0,"messages_ready_details":{"rate":0.0},"messages_ready_ram":0,"messages_unacknowledged":0,"messages_unacknowledged_details":{"rate":0.0},"messages_unacknowledged_ram":0,"name":"user2_jobs","node":"[email protected]","operator_policy":null,"policy":null,"recoverable_slaves":null,"reductions":22135048,"reductions_details":{"rate":227.4},"single_active_consumer_tag":null,"state":"running","type":"classic","vhost":"/"},{"arguments":{},"auto_delete":false,"backing_queue_status":{"avg_ack_egress_rate":0.0,"avg_ack_ingress_rate":0.0,"avg_egress_rate":0.0,"avg_ingress_rate":0.0,"delta":["delta","undefined",0,0,"undefined"],"len":0,"mode":"default","next_seq_id":0,"q1":0,"q2":0,"q3":0,"q4":0,"target_ram_count":"infinity"},"consumer_utilisation":1.0,"consumers":2,"durable":true,"effective_policy_definition":{},"exclusive":false,"exclusive_consumer_tag":null,"garbage_collection":{"fullsweep_after":65535,"max_heap_size":0,"min_bin_vheap_size":46422,"min_heap_size":233,"minor_gcs":5063},"head_message_timestamp":null,"memory":197500,"message_bytes":0,"message_bytes_paged_out":0,"message_bytes_persistent":0,"message_bytes_ram":0,"message_bytes_ready":0,"message_bytes_unacknowledged":0,"messages":0,"messages_details":{"rate":0.0},"messages_paged_out":0,"messages_persistent":0,"messages_ram":0,"messages_ready":0,"messages_ready_details":{"rate":0.0},"messages_ready_ram":0,"messages_unacknowledged":0,"messages_unacknowledged_details":{"rate":0.0},"messages_unacknowledged_ram":0,"name":"user1_jobs","node":"[email protected]","operator_policy":null,"policy":null,"recoverable_slaves":null,"reductions":22578441,"reductions_details":{"rate":232.2},"single_active_consumer_tag":null,"state":"running","type":"classic","vhost":"/"},{"arguments":{},"auto_delete":false,"backing_queue_status":{"avg_ack_egress_rate":0.0,"avg_ack_ingress_rate":0.0,"avg_egress_rate":0.0,"avg_ingress_rate":0.0,"delta":["delta","undefined",0,0,"undefined"],"len":0,"mode":"default","next_seq_id":0,"q1":0,"q2":0,"q3":0,"q4":0,"target_ram_count":"infinity"},"consumer_utilisation":1.0,"consumers":1,"durable":true,"effective_policy_definition":{},"exclusive":false,"exclusive_consumer_tag":null,"garbage_collection":{"fullsweep_after":65535,"max_heap_size":0,"min_bin_vheap_size":46422,"min_heap_size":233,"minor_gcs":535},"head_message_timestamp":null,"memory":143244,"message_bytes":0,"message_bytes_paged_out":0,"message_bytes_persistent":0,"message_bytes_ram":0,"message_bytes_ready":0,"message_bytes_unacknowledged":0,"messages":0,"messages_details":{"rate":0.0},"messages_paged_out":0,"messages_persistent":0,"messages_ram":0,"messages_ready":0,"messages_ready_details":{"rate":0.0},"messages_ready_ram":0,"messages_unacknowledged":0,"messages_unacknowledged_details":{"rate":0.0},"messages_unacknowledged_ram":0,"name":"user5","node":"[email protected]","operator_policy":null,"policy":null,"recoverable_slaves":null,"reductions":22141735,"reductions_details":{"rate":227.4},"single_active_consumer_tag":null,"state":"running","type":"classic","vhost":"/"},{"arguments":{},"auto_delete":false,"backing_queue_status":{"avg_ack_egress_rate":0.0,"avg_ack_ingress_rate":0.0,"avg_egress_rate":0.0,"avg_ingress_rate":0.0,"delta":["delta","undefined",0,0,"undefined"],"len":0,"mode":"default","next_seq_id":0,"q1":0,"q2":0,"q3":0,"q4":0,"target_ram_count":"infinity"},"consumer_utilisation":1.0,"consumers":1,"durable":true,"effective_policy_definition":{},"exclusive":false,"exclusive_consumer_tag":null,"garbage_collection":{"fullsweep_after":65535,"max_heap_size":0,"min_bin_vheap_size":46422,"min_heap_size":233,"minor_gcs":3385},"head_message_timestamp":null,"memory":176732,"message_bytes":0,"message_bytes_paged_out":0,"message_bytes_persistent":0,"message_bytes_ram":0,"message_bytes_ready":0,"message_bytes_unacknowledged":0,"messages":0,"messages_details":{"rate":0.0},"messages_paged_out":0,"messages_persistent":0,"messages_ram":0,"messages_ready":0,"messages_ready_details":{"rate":0.0},"messages_ready_ram":0,"messages_unacknowledged":0,"messages_unacknowledged_details":{"rate":0.0},"messages_unacknowledged_ram":0,"name":"user7","node":"[email protected]","operator_policy":null,"policy":null,"recoverable_slaves":null,"reductions":22143810,"reductions_details":{"rate":229.8},"single_active_consumer_tag":null,"state":"running","type":"classic","vhost":"/"}]

[mkuratczyk]

You perform URL rewriting on your Ingress so I'd expect that to be the culprit. Check https://www.rabbitmq.com/management.html#path-prefix

[pachalk]

Can you tell me how can we fix it?

[mkuratczyk]

I'd try this:

management.path_prefix = /rabbitmq

since you are rewriting the URLs to /rabbitmq. If that doesn't help then you need to dig in but given you have HTTP-level issues we've never seen before and you are rewriting the URLs then most likely the problem is with how you rewrite them.

[pachalk]

I tried by adding management.path_prefix = /rabbitmq its failing at readiness check
Error found in pod events - but logs didn't show any error
Readiness probe failed: wget: server returned error: HTTP/1.1 404 Not Found

[mkuratczyk]

You configured your readinessProbe as http://localhost:15672/api/healthchecks/node. You now need to adjust it to the new path. I'd also suggest just using the RabbitMQ Operator instead of developing your own YAML (eg. the Operator doesn't use the HTTP health check as a readiness probe).

Can I ask you to test whether the Management UI works correctly when you connect directly to the RabbitMQ node? If yes - there is no issue with RabbitMQ but purely with how you forward/rewrite traffic to it.

[pachalk]

Yes its resolved, I changed url of api health check to http://localhost:15672/rabbitmq/api/healthchecks/node because we are adding this management.path_prefix = /rabbitmq and then fixed the ingress by changing path to /rabbitmq. Here the issue is the url path we provided to ingress its not allowing us to check the connection details.

Can I ask you to test whether the Management UI works correctly when you connect directly to the RabbitMQ node? If yes - there is no issue with RabbitMQ but purely with how you forward/rewrite traffic to it -
Tested - Thats true when i changed to NodePort Rabbitmq Management UI works perfectly, so the culprit is ingress traffic forward issue.

Thank you so much for help me and I really appreciate your response and quick resolution.