Docker swarm on freshly installed 3 vm's initialised properly but 2377 does not open => workers or managers cant join the cluster

[cazacubogdan]

Description
architecture:
3 hosts running proxmox. - simple networking - not even vlans.
3 vm's running ubuntu 22.04.2 (one for each proxmox hosts)
any of the 3 vm's if i initialise a docker swarm, it initialised properly but no 2377 port is opened.

LE: no firewalls, no proxies, no ids/ips/snort or sniffers, no wifi, just simple LAN with all hosts in one network.
LE2: python web server starts on 2377 even if docker is still running.

kuber@kuber-pl-01:~$ sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:61209 0.0.0.0:* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
udp 0 0 127.0.0.53:53 0.0.0.0:* -
udp 0 0 192.168.0.30:68 0.0.0.0:* -

kuber@kuber-pl-02:/etc/systemd/system$ nc -vz 192.168.0.30 2377
nc: connect to 192.168.0.30 port 2377 (tcp) failed: Connection refused

kuber@kuber-pl-01:~$ docker swarm init --listen-addr=192.168.0.30:2377
Error response from daemon: manager stopped: failed to listen on remote API address: listen tcp 192.168.0.30:2377: bind: cannot assign requested address

but! if i try with:
kuber@kuber-pl-01:~$ docker swarm init --advertise-addr=192.168.0.30:2377
Swarm initialized: current node (s9s7qjsddyr8015ionvdvzzdt) is now a manager.

To add a worker to this swarm, run the following command:

docker swarm join --token SWMTKN-1-496gjmns84wrz9mz7zw7qwoygrk7isw1am3xipj6oe0lvamtqy-d605pswgvnox1bzx48cahapxq 192.168.0.30:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

results on any of the vm's
kuber@kuber-pl-03:~$ docker swarm join --token SWMTKN-1-496gjmns84wrz9mz7zw7qwoygrk7isw1am3xipj6oe0lvamtqy-b4dxahq23o422zzlzlqw6g5qz 192.168.0.30:2377
Error response from daemon: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 192.168.0.30:2377: connect: connection refused"

kuber@kuber-pl-01:~$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
s9s7qjsddyr8015ionvdvzzdt * kuber-pl-01 Ready Active Leader 24.0.7

same version on all vm's
Reproduce
see above

Expected behavior
completion of adding a manager to the swarm

docker version
kuber@kuber-pl-01:~$ docker version
Client: Docker Engine - Community
Version: 24.0.7
API version: 1.43
Go version: go1.20.10
Git commit: afdd53b
Built: Thu Oct 26 09:07:41 2023
OS/Arch: linux/amd64
Context: rootless

Server: Docker Engine - Community
Engine:
Version: 24.0.7
API version: 1.43 (minimum version 1.12)
Go version: go1.20.10
Git commit: 311b9ff
Built: Thu Oct 26 09:07:41 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.24
GitCommit: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
runc:
Version: 1.1.9
GitCommit: v1.1.9-0-gccaecfc
docker-init:
Version: 0.19.0
GitCommit: de40ad0
rootlesskit:
Version: 1.1.1
ApiVersion: 1.1.1
NetworkDriver: slirp4netns
PortDriver: builtin
StateDir: /tmp/rootlesskit1430401958
slirp4netns:
Version: 1.0.1
GitCommit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
docker info
kuber@kuber-pl-01:~$ docker info
Client: Docker Engine - Community
Version: 24.0.7
Context: rootless
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.21.0
Path: /usr/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 24.0.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: false
userxattr: true
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: s9s7qjsddyr8015ionvdvzzdt
Is Manager: true
ClusterID: fy0um5tvoqif1j81e4p8ind2e
Managers: 1
Nodes: 1
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 192.168.0.30
Manager Addresses:
192.168.0.30:2377
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
runc version: v1.1.9-0-gccaecfc
init version: de40ad0
Security Options:
seccomp
Profile: builtin
rootless
cgroupns
Kernel Version: 5.15.0-88-generic
Operating System: Ubuntu 22.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 4.394GiB
Name: kuber-pl-01
ID: 4c21e420-eb02-4288-8347-1afc5fe6663e
Docker Root Dir: /home/kuber/.local/share/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support
Diagnostics ID
lll

Additional Info
kuber@kuber-pl-01:/opt/containerd$ sudo service ufw status
○ ufw.service - Uncomplicated firewall
Loaded: loaded (/lib/systemd/system/ufw.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Wed 2023-11-08 18:23:13 UTC; 27min ago
Docs: man:ufw(8)
Main PID: 621 (code=exited, status=0/SUCCESS)
CPU: 2ms

Nov 08 18:16:29 kuber-pl-01 systemd[1]: Starting Uncomplicated firewall...
Nov 08 18:16:29 kuber-pl-01 systemd[1]: Finished Uncomplicated firewall.
Nov 08 18:23:13 kuber-pl-01 systemd[1]: Stopping Uncomplicated firewall...
Nov 08 18:23:13 kuber-pl-01 ufw-init[2570]: Skip stopping firewall: ufw (not enabled)
Nov 08 18:23:13 kuber-pl-01 systemd[1]: ufw.service: Deactivated successfully.
Nov 08 18:23:13 kuber-pl-01 systemd[1]: Stopped Uncomplicated firewall.

so its not the firewall either...

[cazacubogdan]

update: issue present only in ubuntu 22.04.2. 22.04 and 20.04 don't have this issue

[s4ke]

Which Base Image are you using? I think I saw recently some people having issues with a KVM specific image on Proxmox with Swarm.

[cazacubogdan]

i'm using the server install img from ubuntu...