[BUG] kube-multus-ds crashloopbackoff - text file busy

[reski-rukmantiyo]

Kube-OVN Version

v1.12.12

Kubernetes Version

v1.28.10

Operation-system/Kernel Version

Ubuntu 22.04.4 LTS

Description

Pod multus condition keep crashing

NAME                                      READY   STATUS                  RESTARTS       AGE
coredns-6799fbcd5-x6wc5                   1/1     Running                 0              69d
kube-multus-ds-25vh7                      1/1     Running                 0              3m49s
kube-multus-ds-ctk6l                      0/1     Init:CrashLoopBackOff   5 (62s ago)    3m49s
kube-multus-ds-rc8j5                      1/1     Running                 0              3m49s
kube-ovn-cni-d292p                        1/1     Running                 3 (159m ago)   26d
kube-ovn-cni-ghvw5                        1/1     Running                 1 (68m ago)    26d
kube-ovn-cni-kxm7d                        1/1     Running                 2 (20d ago)    26d
kube-ovn-controller-7bbbdb7cdf-kgpx5      1/1     Running                 1 (83d ago)    83d
kube-ovn-monitor-cb8df866b-g9dqf          1/1     Running                 0              83d
kube-ovn-pinger-qdfjt                     1/1     Running                 0              36d
kube-ovn-pinger-sfz77                     1/1     Running                 0              83d
kube-ovn-pinger-tgkpp                     1/1     Running                 1 (69d ago)    83d
local-path-provisioner-84db5d44d9-gszp6   1/1     Running                 0              69d
ovn-central-5d94c84b7d-6j58w              1/1     Running                 0              83d
ovs-ovn-h8hfq                             1/1     Running                 1 (20d ago)    20d
ovs-ovn-k4xmc                             1/1     Running                 1              20d
ovs-ovn-zzkb8                             1/1     Running                 0              20d

Apparently it's due to

cp: cannot create regular file '/host/opt/cni/bin/multus-shim': Text file busy

in PodInitializing - install-multus-binary stage - therefore it wont allow kube-multus on running

Steps To Reproduce

In the production, it appears that if I remove kube-multus pod either gracefully or by force, it will create this situation.

Current Behavior

Following are the description of the current behaviour

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  2m8s                 default-scheduler  Successfully assigned kube-system/kube-multus-ds-ctk6l to tbspxdgpuwkr004
  Normal   Pulled     43s (x5 over 2m7s)   kubelet            Container image "ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot-thick" already present on machine
  Normal   Created    43s (x5 over 2m7s)   kubelet            Created container install-multus-binary
  Normal   Started    42s (x5 over 2m6s)   kubelet            Started container install-multus-binary
  Warning  BackOff    19s (x10 over 2m5s)  kubelet            Back-off restarting failed container install-multus-binary in pod kube-multus-ds-ctk6l_kube-system(695d7264-ede6-4927-aa3e-d76d9988df21)

Defaulted container "kube-multus" out of: kube-multus, install-multus-binary (init)
Error from server (BadRequest): container "kube-multus" in pod "kube-multus-ds-ctk6l" is waiting to start: PodInitializing

Expected Behavior

Kube-multus can detect the problem, therefore kube-multus can run.
Options

1. can release the process of multus-shim
2. can choose to ignore, then process the next

[zhangzujian]

Please report this problem to the multus community, unless you are sure it's caused by Kube-OVN.

[reski-rukmantiyo]

Hi @zhangzujian noted, let me cross with them as well. Let's close this.

[reski-rukmantiyo]

hi all, i think can use this code as unofficial patch : k8snetworkplumbingwg/multus-cni#1221