Docker swarm on freshly installed 3 vm's initialised properly but 2377 does not open => workers or managers cant join the cluster

[cazacubogdan]

Description

architecture:
3 hosts running proxmox. - simple networking - not even vlans.
3 vm's running ubuntu 22.04.2 (one for each proxmox hosts)
any of the 3 vm's if i init a docker swarm, it initialised properly but no port is opened.

kuber@kuber-pl-01:~$ netstat -tulpn
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:61209 0.0.0.0:* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
udp 0 0 127.0.0.53:53 0.0.0.0:* -
udp 0 0 192.168.0.30:68 0.0.0.0:* -

kuber@kuber-pl-02:/etc/systemd/system$ nc -vz 192.168.0.30 2377
nc: connect to 192.168.0.30 port 2377 (tcp) failed: Connection refused

kuber@kuber-pl-01:~$ docker swarm init --listen-addr=192.168.0.30:2377
Error response from daemon: manager stopped: failed to listen on remote API address: listen tcp 192.168.0.30:2377: bind: cannot assign requested address

but! if i try with:
kuber@kuber-pl-01:~$ docker swarm init --advertise-addr=192.168.0.30:2377
Swarm initialized: current node (s9s7qjsddyr8015ionvdvzzdt) is now a manager.

To add a worker to this swarm, run the following command:

docker swarm join --token SWMTKN-1-496gjmns84wrz9mz7zw7qwoygrk7isw1am3xipj6oe0lvamtqy-d605pswgvnox1bzx48cahapxq 192.168.0.30:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

results on any of the vm's
kuber@kuber-pl-03:~$ docker swarm join --token SWMTKN-1-496gjmns84wrz9mz7zw7qwoygrk7isw1am3xipj6oe0lvamtqy-b4dxahq23o422zzlzlqw6g5qz 192.168.0.30:2377
Error response from daemon: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 192.168.0.30:2377: connect: connection refused"

kuber@kuber-pl-01:~$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
s9s7qjsddyr8015ionvdvzzdt * kuber-pl-01 Ready Active Leader 24.0.7

• same version on all vm's

Reproduce

see above

Expected behavior

completion of adding a manager to the swarm

docker version

kuber@kuber-pl-01:~$ docker version
Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:07:41 2023
 OS/Arch:           linux/amd64
 Context:           rootless

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:07:41 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          1.1.1
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       builtin
  StateDir:         /tmp/rootlesskit1430401958
 slirp4netns:
  Version:          1.0.1
  GitCommit:        6a7b16babc95b6a3056b33fb45b74a6f62262dd4

docker info

kuber@kuber-pl-01:~$ docker info
Client: Docker Engine - Community
 Version:    24.0.7
 Context:    rootless
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: s9s7qjsddyr8015ionvdvzzdt
  Is Manager: true
  ClusterID: fy0um5tvoqif1j81e4p8ind2e
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.0.30
  Manager Addresses:
   192.168.0.30:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 5.15.0-88-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 4.394GiB
 Name: kuber-pl-01
 ID: 4c21e420-eb02-4288-8347-1afc5fe6663e
 Docker Root Dir: /home/kuber/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support

Diagnostics ID

lll

Additional Info

kuber@kuber-pl-01:/opt/containerd$ sudo service ufw status
○ ufw.service - Uncomplicated firewall
Loaded: loaded (/lib/systemd/system/ufw.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Wed 2023-11-08 18:23:13 UTC; 27min ago
Docs: man:ufw(8)
Main PID: 621 (code=exited, status=0/SUCCESS)
CPU: 2ms

Nov 08 18:16:29 kuber-pl-01 systemd[1]: Starting Uncomplicated firewall...
Nov 08 18:16:29 kuber-pl-01 systemd[1]: Finished Uncomplicated firewall.
Nov 08 18:23:13 kuber-pl-01 systemd[1]: Stopping Uncomplicated firewall...
Nov 08 18:23:13 kuber-pl-01 ufw-init[2570]: Skip stopping firewall: ufw (not enabled)
Nov 08 18:23:13 kuber-pl-01 systemd[1]: ufw.service: Deactivated successfully.
Nov 08 18:23:13 kuber-pl-01 systemd[1]: Stopped Uncomplicated firewall.

so its not the firewall...

ps: havent found a "for linux" repo to open the issue in...that's why i opened it here

[bsousaa]

This should likely go to the swarmkit repo https://github.com/moby/swarmkit

[cazacubogdan]

moved to moby/swarmkit#3157