Flannel is broken in Clearlinux due to CNI directory setup in crio.conf

[mcastelino]

Core DNS Pod logs

  Warning  FailedCreatePodSandBox  98s                    kubelet, clr-02    Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fb8b8dccf-9fpfn_kube-system_50f10a4d-947c-11e9-9e0f-52540041e026_0(fd2c062948fa94deb8214b9c088acbe3a6f69dd2606f8261f56aae85209d876b): failed to find plugin "loopback" in path [/opt/cni/bin/ /opt/cni/bin/]

Clearlinux version

sudo swupd info
Installed version: 30030

CRIO Version

sudo crictl version
Version:  0.1.0
RuntimeName:  cri-o
RuntimeVersion:  1.14.4
RuntimeApiVersion:  v1alpha1

Root cause

# Paths to directories where CNI plugin binaries are located.
plugin_dirs = [
        "/usr/libexec/cni",
        "/opt/cni/bin/",
]

/usr/share/defaults/crio/crio.conf has the proper values, but /etc/crio/crio.conf is incorrect

It looks like based on which version of crio you were on. When clearlinux is updated from a version where crio did not support plugin_dirs list to one that did, the exiting /etc/crio/crio.conf will have the older value.

Hence on a crio update the user unfortunately needs to delete /etc/crio/crio.conf

Hence we should delete crio.conf and restart crio which will automatically setup crio based on latest values.

[mcastelino]

/cc @bryteise @hnanni

[mythi]

@mcastelino

Hence we should delete crio.conf and restart crio which will automatically setup crio based on latest values.

I'd propose we first check can we get rid of kata-runtime adding a copy of crio.conf in /etc/crio to get to proper stateless. kata-runtime is now installed part of cloud-native-basic so also default crio.conf could assume kata is present. #89 is also relevant in that case.

[mcastelino]

I'd propose we first check can we get rid of kata-runtime adding a copy of crio.conf in /etc/crio to get to proper stateless. kata-runtime is now installed part of cloud-native-basic so also default crio.conf could assume kata is present. #89 is also relevant in that case.

@mythi I agree. It does not hurt to have kata enabled in crio even if the system cannot launch kata (i.e. if VT-x is not enabled or not available). What makes the problem worse was the crio did not seem to output any failure logs. The kata binaries are present and the user may choose to enable VT-x at a later state. Also nothing prevents the user from disabling VT-x. So having kata presetup in the stateless crio.conf makes sense to me.

/cc @krsna1729

[ahsan518]

Tested this with Flannel and Crio, and this works fine. Testing Flannel with containerd, and seems like coredns service isn't starting, debugging it. Shall we add flannel to the CNI as well ? so the user have the option to use Flannel when desired.

[ahsan518]

Closing this as Flannel is merged