Removes lingering kubelet plugins on snap removal

[claudiubelu]

At the moment, the kubelet plugins in /var/lib/kubelet/plugins/ are not removed, even though the Pods associated with those plugins have been removed.

This means that on reinstallation, kubelet will try to access those inexistent plugins through the *.sock files, including for plugins we don't want anymore, and delaying registration for the ones we do want.

Sample logs:

Oct 10 12:57:13 ubuntu k8s.kubelet[47511]: W1010 12:57:13.690383   47511 logging.go:55] [core] [Channel #75 SubChannel #77]grpc: addrConn.createTransport failed to connect to {Addr: "/var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock", ServerName: "%2Fvar%2Flib%2Fkubelet%2Fplugins_registry%2Fdriver.longhorn.io-reg.sock", }. Err: connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock: connect: connection refused"
Oct 10 12:57:13 ubuntu k8s.kubelet[47511]: W1010 12:57:13.690383   47511 logging.go:55] [core] [Channel #74 SubChannel #76]grpc: addrConn.createTransport failed to connect to {Addr: "/var/lib/kubelet/plugins_registry/nfs.csi.k8s.io-reg.sock", ServerName: "%2Fvar%2Flib%2Fkubelet%2Fplugins_registry%2Fnfs.csi.k8s.io-reg.sock", }. Err: connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins_registry/nfs.csi.k8s.io-reg.sock: connect: connection refused"
Oct 10 12:57:14 ubuntu k8s.kubelet[47511]: W1010 12:57:14.690864   47511 logging.go:55] [core] [Channel #74 SubChannel #76]grpc: addrConn.createTransport failed to connect to {Addr: "/var/lib/kubelet/plugins_registry/nfs.csi.k8s.io-reg.sock", ServerName: "%2Fvar%2Flib%2Fkubelet%2Fplugins_registry%2Fnfs.csi.k8s.io-reg.sock", }. Err: connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins_registry/nfs.csi.k8s.io-reg.sock: connect: connection refused"
Oct 10 12:57:14 ubuntu k8s.kubelet[47511]: W1010 12:57:14.690871   47511 logging.go:55] [core] [Channel #75 SubChannel #77]grpc: addrConn.createTransport failed to connect to {Addr: "/var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock", ServerName: "%2Fvar%2Flib%2Fkubelet%2Fplugins_registry%2Fdriver.longhorn.io-reg.sock", }. Err: connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock: connect: connection refused"
Oct 10 12:57:16 ubuntu k8s.kubelet[47511]: W1010 12:57:16.248852   47511 logging.go:55] [core] [Channel #75 SubChannel #77]grpc: addrConn.createTransport failed to connect to {Addr: "/var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock", ServerName: "%2Fvar%2Flib%2Fkubelet%2Fplugins_registry%2Fdriver.longhorn.io-reg.sock", }. Err: connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock: connect: connection refused"
Oct 10 12:57:16 ubuntu k8s.kubelet[47511]: W1010 12:57:16.318454   47511 logging.go:55] [core] [Channel #74 SubChannel #76]grpc: addrConn.createTransport failed to connect to {Addr: "/var/lib/kubelet/plugins_registry/nfs.csi.k8s.io-reg.sock", ServerName: "%2Fvar%2Flib%2Fkubelet%2Fplugins_registry%2Fnfs.csi.k8s.io-reg.sock", }. Err: connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins_registry/nfs.csi.k8s.io-reg.sock: connect: connection refused"
Oct 10 12:57:19 ubuntu k8s.kubelet[47511]: W1010 12:57:19.003182   47511 logging.go:55] [core] [Channel #74 SubChannel #76]grpc: addrConn.createTransport failed to connect to {Addr: "/var/lib/kubelet/plugins_registry/nfs.csi.k8s.io-reg.sock", ServerName: "%2Fvar%2Flib%2Fkubelet%2Fplugins_registry%2Fnfs.csi.k8s.io-reg.sock", }. Err: connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins_registry/nfs.csi.k8s.io-reg.sock: connect: connection refused"
Oct 10 12:57:19 ubuntu k8s.kubelet[47511]: W1010 12:57:19.054009   47511 logging.go:55] [core] [Channel #75 SubChannel #77]grpc: addrConn.createTransport failed to connect to {Addr: "/var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock", ServerName: "%2Fvar%2Flib%2Fkubelet%2Fplugins_registry%2Fdriver.longhorn.io-reg.sock", }. Err: connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock: connect: connection refused"
Oct 10 12:57:22 ubuntu k8s.kubelet[47511]: W1010 12:57:22.827338   47511 logging.go:55] [core] [Channel #74 SubChannel #76]grpc: addrConn.createTransport failed to connect to {Addr: "/var/lib/kubelet/plugins_registry/nfs.csi.k8s.io-reg.sock", ServerName: "%2Fvar%2Flib%2Fkubelet%2Fplugins_registry%2Fnfs.csi.k8s.io-reg.sock", }. Err: connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins_registry/nfs.csi.k8s.io-reg.sock: connect: connection refused"
Oct 10 12:57:23 ubuntu k8s.kubelet[47511]: E1010 12:57:23.690986   47511 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/nfs.csi.k8s.io-reg.sock" failed. No retries permitted until 2024-10-10 12:59:25.690966001 +0000 UTC m=+473.184762730 (durationBeforeRetry 2m2s). Error: RegisterPlugin error -- dial failed at socket /var/lib/kubelet/plugins_registry/nfs.csi.k8s.io-reg.sock, err: failed to dial socket /var/lib/kubelet/plugins_registry/nfs.csi.k8s.io-reg.sock, err: context deadline exceeded
Oct 10 12:57:23 ubuntu k8s.kubelet[47511]: E1010 12:57:23.691070   47511 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock" failed. No retries permitted until 2024-10-10 12:59:25.691044 +0000 UTC m=+473.184840729 (durationBeforeRetry 2m2s). Error: RegisterPlugin error -- dial failed at socket /var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock, err: failed to dial socket /var/lib/kubelet/plugins_registry/driver.longhorn.io-reg.sock, err: context deadline exceeded

[addyess]

@claudiubelu
how will this be combined with #726? If both are merged?

[claudiubelu]

@claudiubelu how will this be combined with #726? If both are merged?

Will conflict, will have to rebase one of them after one of them merges. But other than that, they're independent. This only refers to the kubelet plugins *.sock files, which are used as a communication channel between kubelet and the plugin itself. This does not refer to any volumes.

[bschimke95]

LGTM