proxy.go: Can't lock /run/xtables. lock: Resource temporarily unavailable Another app currently holding the xtables lock.

[liyimeng]

Environmental Info:
K3s Version:

1.28.9

Node(s) CPU architecture, OS, and Version:

amd64 ubuntu 22.04, kernel 6.22

Cluster Configuration:

1 server

Describe the bug:

When creating a replicaset with replicas number set to 100 pod, k3s keep restarting and error log like below keep showing up.

Steps To Reproduce:

• Installed K3s:
• Create a deployment with 100 replicas
• k3s keep restarting
• see the log
  Expected behavior:

k3s server should keep healthy and pods should be created

Actual behavior:

k3s keep restarting, pod creation is painful slow.

Additional context / logs:

E1012 13:47:04.628717268531 proxier.go:1447] "Failed to execute iptables-restore" err=‹exit status 4: Ignoring deprecated--wait-interval option.
Can't lock /run/xtables. lock: Resource temporarily unavailable  Another app currently holding the xtables lock.
Stopped waiting after 5s.

> rules="*nat\n: KUBE-SERVICES - 10:01\n: KUBE-POSTROUTING - [0:01\n-A KUBE-SERVICES -S 127.0.0.0/8 - j RETURN\n-A KUBE-POSTROUTING -m comment
BACK dst, dst, src - j MASQUERADE\n-A KUBE-NODE-PORT -p tcp
-m set --match-set KUBE-LOOP-srcomn MAsaatADencta kuptno0t-Pant ptent,, concent feonent intpbernetes ndeuort nop ortndc"-ast erateumose\" -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQNn-A KUBE-SERVICES -m comment service cluster ip + port for masquerade purpose\"
-m set --match-set KUBE-CLUSTER-IP dst,dst ! -s 10.42.0.0/16 - j KUBE-MARK-MASO\n-A KUBE -SERVICES -m comment -comment \"Kubernetes service external ip + port for masquerade and filter purposel" -m set --match-set KU BE-EXTERNAL- IP dst, dst -j KUBE-MARK-MASQn-A KUBE-SERVICES -m comment --comment \'Kubernetes service external ip + port for masq
uerade and filter
SEE4SACCEPRRUBESERU1CS-coment=ComentTubernetesseruiceexternatipporttornasqueradeand1
--src-type LO
--comment \"Kubernetes service external ip + port for masquerade and filter purpose!"
—m set
--match-set KUBE-EXTERNAL-IP dst,dst -m addrtype
--dst-type LOCAL -j ACCEPTAn-A KUBE-SERVICES -m addrtype
--dst-type LOC AL - j KUBE-NODE-PORTAn-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ n-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst, dst - j AC CEPT\N-A KUBE-POSTROUTING -m mark ! --mark 0x00004000/0x00004000 - j RETURN\n-A KUBE-POSTROUTING - j MARK --xor-mark 0x00004000\n -A KUBE-POSTROUTING -m comment --comment \"kubernetes service traffic requiring SNAT\"
-j MASQUERADE --random-fully\n-A KUBE-MARK -MASQ -j MARK --or-mark 0x00004000\COMMIT\n*filter\n-A KUBE-SOURCE-RANGES-FIREWALL -j DROP\n-A KUBE-FORWARD -m comment
t \"kubernetes forwarding rules)"
-m mark --mark 0x00004000/0x00004000
-j ACCEPTAn-A KUBE-FORWARD
--commen
—m
comment --comment \"kuberne
tes forwarding
contrack rule\" -m conntrack
--ctstate RELATED, ESTABLISHED -j ACCEPTAn-A KUBE-NODE-PORT -m comment
--comment \"K
ubernetes health check node ports" -m set -match-set KUBE-HEALTH-CHECK-NODE-PORT dst -j ACCEPTAn-A KUBE-IPUS-FILTER -m set --ma tch-set KUBE-LOAD-BALANCER dst,dst -j RETURN\n-A KUBE-IPUS-FILTER -m set --match-set KUBE-CLUSTER-IP dst, dst -j RETURN\n-A KUBE-
IPUS-FILTER -m set --match-set KUBE-EXTERNAL-IP dst,dst -i RETURN\n-A KUBE-IPUS-FILTER -m set
--match-set KUBE-EXTERNAL- IP-LOCAL dst,dst -j RETURN\n-A KUBE-IPUS-FILTER -m set --match-set KUBE-HEALTH-CHECK-NODE-PORT dst - j RETURN\n-A KUBE-IPUS-FILTER -m con track --ctstate NEW -m set --match-set KUBE-IPUS-IPS dst -j REJECT\ncommit\n"

[brandond]

This isn't a fatal error, if k3s is exiting then something else is going wrong. It does suggest that you have multiple things trying to manage iptables rules, or that updating the iptables rules is taking so long that things are holding the lock for excessively long. Are you sure that this node is actually capable of supporting the workload you're trying to place on it? The iptables locks are not USUALLy where we see a lot of problems, so either this is a VERY underpowered node, or you have something else going on.

[liyimeng]

@brandond Thanks! it seems to be resource constraint.