ERROR: epmd error for host rabbitmq-test-server-0.rabbitmq-test-nodes.rabbitmq-test: timeout (timed out)

[bkelava]

Describe the bug

I'm trying to deploy a rabbitmq in kubernetes. I am using a cluster-operator and CRD RabbitmqCluster.

Reproduction steps

1. I've applied official cluster-operator.yaml
2. I've created a configuration file as follows

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq-test
spec:
  replicas: 1
  image: **<repo>**/rabbitmq/rabbitmq:3.13.4-management
  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      cpu: 1
      memory: 2Gi
  persistence:
    storageClassName: nfs-rabbitmq-test-storage
    storage: "10Gi"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - servisi-0023
            - servisi-0024
  rabbitmq:
    additionalConfig: |
      management.load_definitions = /etc/rabbitmq/definitions.json

definitions.json is baked into docker image
...

Expected behavior

Error from my pod

Defaulted container "rabbitmq" out of: rabbitmq, setup-container (init)

BOOT FAILED
2024-07-23 08:16:02.762547+00:00 [error] <0.156.0>
2024-07-23 08:16:02.762547+00:00 [error] <0.156.0> BOOT FAILED
2024-07-23 08:16:02.762547+00:00 [error] <0.156.0> ===========
2024-07-23 08:16:02.762547+00:00 [error] <0.156.0> ERROR: epmd error for host rabbitmq-test-server-0.rabbitmq-test-nodes.rabbitmq-test: timeout (timed out)
2024-07-23 08:16:02.762547+00:00 [error] <0.156.0>
===========
ERROR: epmd error for host rabbitmq-test-server-0.rabbitmq-test-nodes.rabbitmq-test: timeout (timed out)

2024-07-23 08:16:03.772020+00:00 [error] <0.156.0>     supervisor: {local,rabbit_prelaunch_sup}
2024-07-23 08:16:03.772020+00:00 [error] <0.156.0>     errorContext: start_error
2024-07-23 08:16:03.772020+00:00 [error] <0.156.0>     reason: {epmd_error,"rabbitmq-test-server-0.rabbitmq-test-nodes.rabbitmq-test",
2024-07-23 08:16:03.772020+00:00 [error] <0.156.0>                         timeout}
2024-07-23 08:16:03.772020+00:00 [error] <0.156.0>     offender: [{pid,undefined},
2024-07-23 08:16:03.772020+00:00 [error] <0.156.0>                {id,prelaunch},
2024-07-23 08:16:03.772020+00:00 [error] <0.156.0>                {mfargs,{rabbit_prelaunch,run_prelaunch_first_phase,[]}},
2024-07-23 08:16:03.772020+00:00 [error] <0.156.0>                {restart_type,transient},
2024-07-23 08:16:03.772020+00:00 [error] <0.156.0>                {significant,false},
2024-07-23 08:16:03.772020+00:00 [error] <0.156.0>                {shutdown,5000},
2024-07-23 08:16:03.772020+00:00 [error] <0.156.0>                {child_type,worker}]
2024-07-23 08:16:03.772020+00:00 [error] <0.156.0>
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>   crasher:
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>     initial call: application_master:init/4
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>     pid: <0.154.0>
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>     registered_name: []
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>     exception exit: {{shutdown,
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>                          {failed_to_start_child,prelaunch,
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>                              {epmd_error,
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>                                  "rabbitmq-test-server-0.rabbitmq-test-nodes.rabbitmq-test",
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>                                  timeout}}},
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>                      {rabbit_prelaunch_app,start,[normal,[]]}}
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>       in function  application_master:init/4 (application_master.erl, line 142)
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>     ancestors: [<0.153.0>]
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>     message_queue_len: 1
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>     messages: [{'EXIT',<0.155.0>,normal}]
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>     links: [<0.153.0>,<0.44.0>]
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>     dictionary: []
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>     trap_exit: true
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>     status: running
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>     heap_size: 610
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>     stack_size: 28
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>     reductions: 199
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>   neighbours:
2024-07-23 08:16:03.772579+00:00 [error] <0.154.0>
Runtime terminating during boot (terminating)

Crash dump is being written to: erl_crash.dump...[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
[os_mon] memory supervisor port (memsup): Erlang has closed

Describe from my pod

Name:             rabbitmq-test-server-0
Namespace:        rabbitmq-test
Priority:         0
Service Account:  rabbitmq-test-server
Node:             servisi-0023/172.17.12.16
Start Time:       Tue, 23 Jul 2024 09:50:14 +0200
Labels:           app.kubernetes.io/component=rabbitmq
                  app.kubernetes.io/name=rabbitmq-test
                  app.kubernetes.io/part-of=rabbitmq
                  apps.kubernetes.io/pod-index=0
                  controller-revision-hash=rabbitmq-test-server-bcbfb69f8
                  statefulset.kubernetes.io/pod-name=rabbitmq-test-server-0
Annotations:      <none>
Status:           Running
IP:               10.33.128.2
IPs:
  IP:           10.33.128.2
Controlled By:  StatefulSet/rabbitmq-test-server
Init Containers:
  setup-container:
    Container ID:  containerd://9f3a9d53b755e258b68cfe29364b8960c8bd6f6ee8f2ea7ec99dc8cefa3e6a0f
    Image:         172.17.12.132:9110/rabbitmq/rabbitmq:3.13.4-management
    Image ID:      172.17.12.132:9110/rabbitmq/rabbitmq@sha256:87ba9ab0f6222e3e6e0615db337960f8fe6b4f787bd5a66a58a1ba7b8a51214f
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      cp /tmp/erlang-cookie-secret/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie && chmod 600 /var/lib/rabbitmq/.erlang.cookie ; cp /tmp/rabbitmq-plugins/enabled_plugins /operator/enabled_plugins ; echo '[default]' > /var/lib/rabbitmq/.rabbitmqadmin.conf && sed -e 's/default_user/username/' -e 's/default_pass/password/' /tmp/default_user.conf >> /var/lib/rabbitmq/.rabbitmqadmin.conf && chmod 600 /var/lib/rabbitmq/.rabbitmqadmin.conf ; sleep 30
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 23 Jul 2024 09:50:15 +0200
      Finished:     Tue, 23 Jul 2024 09:50:45 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  500Mi
    Requests:
      cpu:        100m
      memory:     500Mi
    Environment:  <none>
    Mounts:
      /operator from rabbitmq-plugins (rw)
      /tmp/default_user.conf from rabbitmq-confd (rw,path="default_user.conf")
      /tmp/erlang-cookie-secret/ from erlang-cookie-secret (rw)
      /tmp/rabbitmq-plugins/ from plugins-conf (rw)
      /var/lib/rabbitmq/ from rabbitmq-erlang-cookie (rw)
      /var/lib/rabbitmq/mnesia/ from persistence (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dfzf2 (ro)
Containers:
  rabbitmq:
    Container ID:   containerd://cac585ce913ee1c2b4e8330edc126b0e63cfb98276d2bef2f66d83bac3af454c
    Image:          172.17.12.132:9110/rabbitmq/rabbitmq:3.13.4-management
    Image ID:       172.17.12.132:9110/rabbitmq/rabbitmq@sha256:87ba9ab0f6222e3e6e0615db337960f8fe6b4f787bd5a66a58a1ba7b8a51214f
    Ports:          4369/TCP, 5672/TCP, 15672/TCP, 15692/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 23 Jul 2024 10:15:00 +0200
      Finished:     Tue, 23 Jul 2024 10:16:03 +0200
    Ready:          False
    Restart Count:  8
    Limits:
      cpu:     1
      memory:  2Gi
    Requests:
      cpu:      500m
      memory:   1Gi
    Readiness:  tcp-socket :amqp delay=10s timeout=5s period=10s #success=1 #failure=3
    Environment:
      MY_POD_NAME:                    rabbitmq-test-server-0 (v1:metadata.name)
      MY_POD_NAMESPACE:               rabbitmq-test (v1:metadata.namespace)
      K8S_SERVICE_NAME:               rabbitmq-test-nodes
      RABBITMQ_ENABLED_PLUGINS_FILE:  /operator/enabled_plugins
      RABBITMQ_USE_LONGNAME:          true
      RABBITMQ_NODENAME:              rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE)
      K8S_HOSTNAME_SUFFIX:            .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE)
    Mounts:
      /etc/pod-info/ from pod-info (rw)
      /etc/rabbitmq/conf.d/10-operatorDefaults.conf from rabbitmq-confd (rw,path="operatorDefaults.conf")
      /etc/rabbitmq/conf.d/11-default_user.conf from rabbitmq-confd (rw,path="default_user.conf")
      /etc/rabbitmq/conf.d/90-userDefinedConfiguration.conf from rabbitmq-confd (rw,path="userDefinedConfiguration.conf")
      /operator from rabbitmq-plugins (rw)
      /var/lib/rabbitmq/ from rabbitmq-erlang-cookie (rw)
      /var/lib/rabbitmq/mnesia/ from persistence (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dfzf2 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  persistence:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  persistence-rabbitmq-test-server-0
    ReadOnly:   false
  plugins-conf:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rabbitmq-test-plugins-conf
    Optional:  false
  rabbitmq-confd:
    Type:                Projected (a volume that contains injected data from multiple sources)
    ConfigMapName:       rabbitmq-test-server-conf
    ConfigMapOptional:   <nil>
    SecretName:          rabbitmq-test-default-user
    SecretOptionalName:  <nil>
  rabbitmq-erlang-cookie:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  erlang-cookie-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rabbitmq-test-erlang-cookie
    Optional:    false
  rabbitmq-plugins:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  pod-info:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels['skipPreStopChecks'] -> skipPreStopChecks
  kube-api-access-dfzf2:
    Type:                     Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:   3607
    ConfigMapName:            kube-root-ca.crt
    ConfigMapOptional:        <nil>
    DownwardAPI:              true
QoS Class:                    Burstable
Node-Selectors:               <none>
Tolerations:                  node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints:  topology.kubernetes.io/zone:ScheduleAnyway when max skew 1 is exceeded for selector app.kubernetes.io/name=rabbitmq-test
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  28m                  default-scheduler  Successfully assigned rabbitmq-test/rabbitmq-test-server-0 to servisi-0023
  Normal   Pulled     28m                  kubelet            Container image "172.17.12.132:9110/rabbitmq/rabbitmq:3.13.4-management" already present on machine
  Normal   Created    28m                  kubelet            Created container setup-container
  Normal   Started    28m                  kubelet            Started container setup-container
  Normal   Pulled     25m (x3 over 27m)    kubelet            Container image "172.17.12.132:9110/rabbitmq/rabbitmq:3.13.4-management" already present on machine
  Normal   Created    25m (x3 over 27m)    kubelet            Created container rabbitmq
  Normal   Started    25m (x3 over 27m)    kubelet            Started container rabbitmq
  Warning  BackOff    8m3s (x52 over 25m)  kubelet            Back-off restarting failed container rabbitmq in pod rabbitmq-test-server-0_rabbitmq-test(b0d2ec84-4307-4d95-ae29-840c997f8f9f)
  Warning  Unhealthy  3m4s (x46 over 27m)  kubelet            Readiness probe failed: dial tcp 10.33.128.2:5672: connect: connection refused

Additional context

No response

[michaelklishin]

@bkelava I'm not sure why you've decided to file this as an issue, and in particular in the documentation repo.

The node tried to contact epmd and that operation has timed out.

My best guess is that access to the epmd port is restricted one way or another or, like in #6439, the node is configured to use short node names but they do not actually resolve.