Resolve tier 1 image CVEs that are identified by security team

[caroline-suse-rancher]

Refer to Jira ticket for more detail: https://jira.suse.com/browse/SURE-6236

[matttrach]

from the JIRA issue:
Resolve all the applicable CVEs flagged by image-scanning in tier 1 images that belong to team/rke2-k3s:

rancher/k3s-upgrade
rancher/rke2-cloud-provider
rancher/rke2-runtime
rancher/rke2-upgrade
rancher/system-agent-installer-k3s
rancher/system-agent-installer-rke2
rancher/system-upgrade-controller
The tier 1 images can be checked in image-scanning's dashboard - https://github.com/rancherlabs/image-scanning/issues/2062. Based on the image name, the issues can be filtered in https://github.com/rancherlabs/image-scanning/issues?q=is%3Aopen by adding the labels "team/rke2-k3s" and "cve/release/v2.7-head".

We are not listing the exact issues here, because new versions of the above images might be flagged by image-scanning in the meantime.

Upstream CVEs are outside the scope of this work, giving that we can't control when they are patched.

Please consult with the Security team in case of questions.

[matttrach]

Our Team CVEs

[matttrach]

Just RKE2 CVEs

[matttrach]

Not Mirrored: 82
Only Mirrors: 91

[matttrach]

I have noticed some duplication of information across the issues I have worked, so in order to work more efficiently, I have decided to compile a list of all the images, and the binaries that are in them. Once this is done, I can take that a level further and find all of the dependencies for the binaries that have CVEs. In the end I hope to have a list of the CVEs for dependencies of binaries that we build, and what images those binaries exist in. This should give us an understanding of the scope of a CVE based on number of images it affects. We can then choose to resolve/ignore a CVE and understand what affect that has on the system as a whole.

[matttrach]

Image Xref Part 1

rancher/rke2-runtime

• is built by rancher/rke2 repo
• contains these 9 binaries
  • runc
    • is copied from image rancher/hardened-runc
    • is built by rancher/image-build-runc
    • according to github.com/opencontainers/runc
  • crictl
    • is copied from image rancher/hardened-crictl
    • is built by rancher/image-build-crictl
    • according to github.com/kubernetes-sigs/cri-tools
  • containerd
    • is copied from image rancher/hardened-containerd
    • is built by rancher/image-build-containerd
    • according to k3s-io/containerd
  • containerd-shim
    • is copied from image rancher/hardened-containerd
    • is built by rancher/image-build-containerd
    • according to github.com/k3s-io/containerd
  • containerd-shim-runc-v1
    • is copied from image rancher/hardened-containerd
    • is built by rancher/image-build-containerd
    • according to github.com/k3s-io/containerd
  • containerd-shim-runc-v2
    • is copied from image rancher/hardened-containerd
    • is built by rancher/image-build-containerd
    • according to github.com/k3s-io/containerd
  • ctr
    • is copied from image rancher/hardened-containerd
    • is built by rancher/image-build-containerd
    • according to github.com/k3s-io/containerd
  • kubectl
    • is copied from image rancher/hardened-kubernetes
    • is built by rancher/image-build-kubernetes
    • according to github.com/kubernetes/kubernetes
  • kubelet
    • is copied from image rancher/hardened-kubernetes
    • is built by rancher/image-build-kubernetes
    • according to github.com/kubernetes/kubernetes

rancher/hardened-runc

• is built by rancher/image-build-runc
• has 1 binary
  • runc
    • is built by rancher/image-build-runc
    • using make
    • according to github.com/opencontainers/runc

rancher/hardened-crictl

• is built by rancher/image-build-crictl
• has 1 binary
  • crictl
    • is built by rancher/image-build-crictl
    • using go-build-static.sh
    • according to github.com/kubernetes-sigs/cri-tools

rancher/hardened-containerd

• is built by rancher/image-build-containerd
• has 6 binaries
  • ctr
    • is built by rancher/image-build-containerd
    • using go-build-static.sh
    • according to github.com/k3s-io/containerd
  • containerd
    • is built by rancher/image-build-containerd
    • using go-build-static.sh
    • according to github.com/k3s-io/containerd
  • containerd-stress
    • is built by rancher/image-build-containerd
    • using go-build-static.sh
    • according to github.com/k3s-io/containerd
  • containerd-shim
    • is built by rancher/image-build-containerd
    • using go-build-static.sh
    • according to github.com/k3s-io/containerd
  • containerd-shim-runc-v1
    • is built by rancher/image-build-containerd
    • using go-build-static.sh
    • according to github.com/k3s-io/containerd
  • containerd-shim-runc-v2
    • is built by rancher/image-build-containerd
    • using go-build-static.sh
    • according to github.com/k3s-io/containerd

rancher/hardened-kubernetes

• is built by rancher/image-build-kubernetes
• has 14 binaries
  • 8 binaries from k3s-root (and 2 shell scripts, not listed)
    • we download configs from buildroot
      • we use the default buildroot version
    • the buildroot configs say how to download package source and build it
    • the k3s-io/root ci does build packages from source
      • using the buildroot tool
• ipset
  • is copied from k3s-io/k3s-root/releases/download
  • is built by k3s-io/root
  • according to buildroot ipset config
• ebtables-legacy
  • is copied from k3s-io/k3s-root/releases/download
  • is built by k3s-io/root
  • according to buildroot ebtables config
• ebtablesd
  • is copied from k3s-io/k3s-root/releases/download
  • is built by k3s-io/root
  • according to buildroot ebtables config
• ebtablesu
  • is copied from k3s-io/k3s-root/releases/download
  • is built by k3s-io/root
  • according to buildroot ebtables config
• iptables-apply
  • is copied from k3s-io/k3s-root/releases/download
  • is built by k3s-io/root
  • according to buildroot iptables config
• xtables-legacy-multi
  • is copied from k3s-io/k3s-root/releases/download
  • is built by k3s-io/root
  • according to buildroot xtables-addons config
• xtables-nft-multi
  • is copied from k3s-io/k3s-root/releases/download
  • is built by k3s-io/root
  • according to buildroot xtables-addons config
• kube-apiserver
  • is built in rancher/image-build-kubernetes
  • using go-build-static.sh
  • according to github.com/kubernetes/kubernetes
• kube-controller-manager
  • is built in rancher/image-build-kubernetes
  • using go-build-static.sh
  • according to github.com/kubernetes/kubernetes
• kube-scheduler
  • is built in rancher/image-build-kubernetes
  • using go-build-static.sh
  • according to github.com/kubernetes/kubernetes
• kube-proxy
  • is built in rancher/image-build-kubernetes
  • using go-build-static.sh
  • according to github.com/kubernetes/kubernetes
• kubeadm
  • is built in rancher/image-build-kubernetes
  • using go-build-static.sh
  • according to github.com/kubernetes/kubernetes
• kubectl
  • is built in rancher/image-build-kubernetes
  • using go-build-static.sh
  • according to github.com/kubernetes/kubernetes
• kubelet
  • is built in rancher/image-build-kubernetes
  • using go-build-static.sh
  • according to github.com/kubernetes/kubernetes

rancher/rke2-upgrade

• is built by drone docker plugin
• has 2 binaries
• rke2
  • is copied from rancher/rke2/releases/download
  • is built by rancher/rke2
  • using rancher/rke2/scripts/build-binary
  • according to rancher/rke2
• kubectl
  • is copied from storage.googleapis.com/kubernetes-release/release
  • is built by kubernets/kubernetes
  • according to kubernetes/kubernetes

[matttrach]

The most prevalent problem is the duplicate/overlapping CVEs

new plan!

1. build a list of binaries and what images they are on
2. build a list of CVE dependencies and what binaries they are on
3. create a cross reference list of dependencies with CVEs to what images they affect

[matttrach]

Image Xref part 2

• rancher/hardened-build-base
  • built by rancher/image-build-base repo
  • based on golang alpine
  • contains 0 binaries that we build
• rancher/hardened-calico
  • is built by rancher/image-build-calico repo
  • [based](https://github.com/rancher/image-build-calico/blob/master/Dockerfile#L204 on bci
  • contains 19 binaries that we build
    • runit
      • this is built by rancher/image-build-calico
      • using make
        • the git repo is not easily available
        • we download the sourcecode from http://smarden.org/runit/runit-2.1.2.tar.gz
        • we unpack it and use the src/install script it ships with to compile it
        • the program is written in c and we compile it with gcc and make
        • the makefile ships with the code
    • ebtables-legacy
      • is copied from k3s-io/k3s-root/releases/download
      • is built by k3s-io/root
      • according to buildroot ebtables config
    • ebtablesd
      • is copied from k3s-io/k3s-root/releases/download
      • is built by k3s-io/root
      • according to buildroot ebtables config
    • ebtablesu
      • is copied from k3s-io/k3s-root/releases/download
      • is built by k3s-io/root
      • according to buildroot ebtables config
    • iptables-apply
      • is copied from k3s-io/k3s-root/releases/download
      • is built by k3s-io/root
      • according to buildroot iptables config
    • xtables-legacy-multi
      • is copied from k3s-io/k3s-root/releases/download
      • is built by k3s-io/root
      • according to buildroot xtables-addons config
    • xtables-nft-multi
      • is copied from k3s-io/k3s-root/releases/download
      • is built by k3s-io/root
      • according to buildroot xtables-addons config
    • bird
      • is copied from calico/bird
      • is built by make using gcc
        • we do not build this
        • I am not able to find any code which automatically builds or pushes the image
      • according to create_binaries.sh
    • bird6
      • is copied from calico/bird
      • is built by make using gcc
        • we do not build this
        • I am not able to find any code which automatically builds or pushes the image
      • according to create_binaries.sh
    • birdcl
      • is copied from calico/bird
      • is built by make using gcc
        • we do not build this
        • I am not able to find any code which automatically builds or pushes the image
      • according to create_binaries.sh
    • birdcl6
      • is [copied](https://github.com/rancher/image-build-calico/blob/master/Dockerfile#L164 from calico/bird
      • is built by make using gcc
        • we do not build this
        • I am not able to find any code which automatically builds or pushes the image
      • according to create_binaries.sh
    • calicoctl
      • is built by rancher/image-build-calico repo
      • using go-build-static.sh
      • according to projectcalico/calico go.mod
    • calico
      • is built by rancher/image-build-calico
      • using go-build-static.sh
      • according to projectcalico/calico
    • calico-ipam
      • is built by rancher/image-build-calico
      • using go-build-static.sh
      • according to projectcalico/calico
    • install (calico cni plugin)
      • is built by rancher/image-build-calico
      • using go-build-static.sh
      • according to projectcalico/calico
    • calico-node
      • is built by rancher/image-build-calico
      • using go build tool
      • according to projectcalico/calico
    • flexvoldriver
      • is built by rancher/image-build-calico
      • using go-build-static.sh
      • according to projectcalico/calico
    • kube-controllers
      • is built by rancher/image-build-calico
      • using go-build-static.sh
      • according to projectcalico/calico
    • check-status
      • is built by rancher/image-build-calico
      • using go-build-static.sh
      • according to projectcalico/calico

[matttrach]

Image Xref part 3

• rancher/hardened-cni-plugins
  • is built by rancher/image-build-cni-plugins repo
  • based on bci
  • has 18 binary that we build
    • bandwidth
      • is built by rancher/image-build-cni-plugins
      • using build_linux.sh
      • according to containernetworking/plugins
    • bridge
      • is built by rancher/image-build-cni-plugins
      • using build_linux.sh
      • according to containernetworking/plugins
    • dhcp
      • is built by rancher/image-build-cni-plugins
      • using build_linux.sh
      • according to containernetworking/plugins
    • dummy
      • is built by rancher/image-build-cni-plugins
      • using build_linux.sh
      • according to containernetworking/plugins
    • firewall
      • is built by rancher/image-build-cni-plugins
      • using build_linux.sh
      • according to containernetworking/plugins
    • host-device
      • is built by rancher/image-build-cni-plugins
      • using build_linux.sh
      • according to containernetworking/plugins
    • host-local
      • is built by rancher/image-build-cni-plugins
      • using build_linux.sh
      • according to containernetworking/plugins
    • ipvlan
      • is built by rancher/image-build-cni-plugins
      • using build_linux.sh
      • according to containernetworking/plugins
    • loopback
      • is built by rancher/image-build-cni-plugins
      • using build_linux.sh
      • according to containernetworking/plugins
    • macvlan
      • is built by rancher/image-build-cni-plugins
      • using build_linux.sh
      • according to containernetworking/plugins
    • portmap
      • is built by rancher/image-build-cni-plugins
      • using build_linux.sh
      • according to containernetworking/plugins
    • ptp
      • is built by rancher/image-build-cni-plugins
      • using build_linux.sh
      • according to containernetworking/plugins
    • sbr
      • is built by rancher/image-build-cni-plugins
      • using build_linux.sh
      • according to containernetworking/plugins
    • static
      • is built by rancher/image-build-cni-plugins
      • using build_linux.sh
      • according to containernetworking/plugins
    • tuning
      • is built by rancher/image-build-cni-plugins
      • using build_linux.sh
      • according to containernetworking/plugins
    • vlan
      • is built by rancher/image-build-cni-plugins
      • using build_linux.sh
      • according to containernetworking/plugins
    • vrf
      • is built by rancher/image-build-cni-plugins
      • using build_linux.sh
      • according to containernetworking/plugins
    • flannel
      • is built by rancher/image-build-cni-plugins
      • using make and build_flannel.sh
      • according to flannel-io/cni-plugin go.mod

[matttrach]

Image Xref part 4

• rancher/hardened-coredns
  • built by rancher/image-build-coredns repo
  • based on bci
  • has 1 bin that we build
    • coredns
      • built by rancher/image-build-coredns
      • using go-build-static.sh
      • according to coredns/coredns
• rancher/hardened-cluster-autoscaler
  • built by rancher/image-build-coredns
  • based on bci
  • has 1 bin that we build
    • cluster-proportional-autoscaler
      • built by rancher/image-build-coredns
      • using go-build-static.sh
      • according to kubernetes-sig/cluster-proportional-autoscaler
• rancher/hardened-dns-node-cache
  • built by rancher/image-build-dns-nodecache repo
  • based on bci
  • has 8 bin that we build
    • ipset
      • is copied from rancher/image-build-kubernetes
      • is built by k3s-io/root
      • according to buildroot ipset config
    • ebtables-legacy
      • is copied) from rancher/image-build-kubernetes
      • is built by k3s-io/root
      • according to buildroot ebtables config
    • ebtablesd
      • is copied from rancher/image-build-kubernetes
      • is built by k3s-io/root
      • according to buildroot ebtables config
    • ebtablesu
      • is copied from rancher/image-build-kubernetes
      • is built by k3s-io/root
      • according to buildroot ebtables config
    • iptables-apply
      • is copied from rancher/image-build-kubernetes
      • is built by k3s-io/root
      • according to buildroot iptables config
    • xtables-legacy-multi
      • is copied from rancher/image-build-kubernetes
      • is built by k3s-io/root
      • according to buildroot xtables-addons config
    • xtables-nft-multi
      • is copied from rancher/image-build-kubernetes
      • is built by k3s-io/root
      • according to buildroot xtables-addons config
    • node-cache
      • is built by rancher/image-build-dns-nodecache
      • using go-build-static.sh
      • according to kubernetes/dns
• rancher/hardened-etcd
  • is built by rancher/image-build-etcd repo
  • based on bci
  • has 2 bin that we build
    • etcd
      • is built by rancher/image-build-etcd
      • using go-build-static.sh
      • according to k3s-io/etcd go.mod
    • etcdctl
      • is built by rancher/image-build-etcd
      • using go-build-static.sh
      • according to k3s-io/etcd go.mod
• rancher/hardened-flannel
  • is built by rancher/image-build-flannel repo
  • based on bci
  • has 7 bin that we build
    • ebtables-legacy
      • is copied from k3s-io/k3s-root/releases/download
      • is built by k3s-io/root
      • according to buildroot ebtables config
    • ebtablesd
      • is copied from k3s-io/k3s-root/releases/download
      • is built by k3s-io/root
      • according to buildroot ebtables config
    • ebtablesu
      • is copied from k3s-io/k3s-root/releases/download
      • is built by k3s-io/root
      • according to buildroot ebtables config
    • iptables-apply
      • is copied from k3s-io/k3s-root/releases/download
      • is built by k3s-io/root
      • according to buildroot iptables config
    • xtables-legacy-multi
      • is copied from k3s-io/k3s-root/releases/download
      • is built by k3s-io/root
      • according to buildroot xtables-addons config
    • xtables-nft-multi
      • is copied from k3s-io/k3s-root/releases/download
      • is built by k3s-io/root
      • according to buildroot xtables-addons config
    • flanneld
      • is built by rancher/image-build-flannel
      • using go-build-static.sh
      • according to flannel-io/flannel go.mod

[matttrach]

Image Xref part 5

• rancher/hardened-ib-sriov-cni
  • is built by rancher/image-build-ib-sriov-cni repo
  • based on bci
  • has 1 bin that we build
    • ib-sriov
      • is built by rancher/image-build-ib-sriov-cni
      • using make
      • according to k8snetworkingplumbingwg/ib-sriov-cni go.mod
• rancher/hardened-k8s-metrics-server
  • is built by rancher/image-build-k8s-metrics-server repo
  • based on bci
  • has 1 bin that we build
    • metrics-server
      • is built by rancher/image-build-k8s-metrics-server
      • using go-build-static.sh
      • according to kubernetes-sigs/metrics-server go.mod
• rancher/hardened-multus-cni
  • is built by rancher/image-build-multus repo
  • based on bci
  • has 3 bin that we build
    • multus
      • is built by rancher/image-build-multus
      • using hack/build-go.sh
      • according to k8snetworkplumbingwg/multus-cni go.mod
    • multus-daemon
      • is built by rancher/image-build-multus
      • using hack/build-go.sh
      • according to k8snetworkplumbingwg/multus-cni go.mod
    • generate-kubeconfig
      • is built by rancher/image-build-multus
      • using hack/build-go.sh
      • according to k8snetworkplumbingwg/multus-cni go.mod
• rancher/rke2-cloud-provider
  • is built by rancher/image-build-rke2-cloud-provider repo
  • based on bci
  • has 1 bin that we build
    • rke2-cloud-provider
      • is built by rancher/image-build-rke2-cloud-provider
      • using go-build-static.sh
      • according to rancher/image-build-rke2-cloud-provider go.mod
• rancher/hardened-sriov-cni
  • is built by rancher/image-build-sriov-cni repo
  • based on bci
  • has 1 bin that we build
    • sriov
      • is built by rancher/image-build-sriov-cni
      • using make
      • according to k8snetworkplumbingwg/sriov-cni go mod

[matttrach]

Image Xref part 6

• rancher/hardened-sriov-cni
  • is built by rancher/image-build-sriov-cni repo
  • based on bci
  • has 1 bin that we build
    • sriov
      • is built by rancher/image-build-sriov-cni
      • using make
      • according to k8snetworkplumbingwg/sriov-cni go mod
• rancher/hardened-sriov-network-device-plugin
  • is built by rancher/image-build-sriov-network-device-plugin repo
  • based on bci
  • has 1 bin that we build
    • sriovdp
      • is built by rancher/image-build-sriov-network-device-plugin repo
      • using make
      • according to k8snetworkplumbingwg/sriov-network-device-plugin go mod
• rancher/hardened-sriov-network-resources-injector
  • is built by rancher/image-build-sriov-network-resources-injector repo
  • based on bci
  • has 2 bin that we build
    • webhook
      • is built by rancher/image-build-sriov-network-resources-injector repo
      • using make to call build.sh which then uses go install
      • according to k8snetworkplumbingwg/network-resources-injector go mod
    • installer
      • is built by rancher/image-build-sriov-network-resources-injector repo
      • using make to call build.sh which then uses go install
      • according to k8snetworkplumbingwg/network-resources-injector go mod
• rancher/hardened-sriov-network-operator
  • is built by rancher/image-build-sriov-operator repo
  • based on bci
  • has 1 bin that we build
    • manager
      • is built by rancher/image-build-sriov-operator repo
      • using make to call hack/build-go.sh which uses go build
      • according to k8snetworkplumbingwg/sriov-network-operator go mod
• rancher/hardened-sriov-network-config-daemon
  • is built by rancher/image-build-sriov-operator repo
  • based on bci
  • has 1 bin that we build
    • sriov-network-config-daemon
      • is built by rancher/image-build-sriov-operator repo
      • using make to call hack/build-go.sh which uses go build
      • according to k8snetworkplumbingwg/sriov-network-operator go mod
• rancher/hardened-sriov-network-webhook
  • is built by rancher/image-build-sriov-operator repo
  • based on bci
  • has 1 bin that we build
    • webhook
      • is built by rancher/image-build-sriov-operator repo
      • using make to call hack/build-go.sh which uses go build
      • according to k8snetworkplumbingwg/sriov-network-operator go mod
• rancher/hardened-whereabouts
  • is built by rancher/image-build-whereabouts repo
  • based on bci
  • has 2 bin that we build
    • whereabouts
      • is built by rancher/image-build-whereabouts repo
      • using go-build-static.sh
      • according to k8snetworkplumbingwg/whereabouts go mod
    • ip-control-loop
      • is built by rancher/image-build-whereabouts repo
      • using go-build-static.sh
      • according to k8snetworkplumbingwg/whereabouts go mod
• rancher/hardened-node-feature-discovery
  • is built by rancher/image-build-node-feature-discovery repo
  • based on bci-minimal
  • has 5 bin that we build
    • nfd-master
      • is built by rancher/image-build-node-feature-discovery repo
      • using go-build-static.sh
      • according to kubernetes-sigs/node-feature-discovery go mod
    • nfd-topology-gc
      • is built by rancher/image-build-node-feature-discovery repo
      • using go-build-static.sh
      • according to kubernetes-sigs/node-feature-discovery go mod
    • nfd-topology-updater
      • is built by rancher/image-build-node-feature-discovery repo
      • using go-build-static.sh
      • according to kubernetes-sigs/node-feature-discovery go mod
    • nfd-worker
      • is built by rancher/image-build-node-feature-discovery repo
      • using go-build-static.sh
      • according to kubernetes-sigs/node-feature-discovery go mod
    • grpc_health_probe
      • is built by rancher/image-build-node-feature-discovery repo
      • using go-build-static.sh
      • according to grpc-ecosystem/grpc-health-probe go mod
• rancher/system-agent-installer-rke2
  • is built by rancher/system-agent-installer-rke2 repo
  • based on scratch
  • has 1 bin that we build
    • rke2
      • is copied from rancher/rke2/releases/download
      • is built by rancher/rke2
      • using rancher/rke2/scripts/build-binary
      • according to rancher/rke2

[matttrach]

That is the last of them, now to compile it from the bin point of view and cross reference with dependency CVEs.

[caroline-suse-rancher]

@matttrach Can you give me a status update on this one and why it was moved to the Backlog? Thank you!

[matttrach]

yeah, sorry. I should have updated this earlier.

I pulled the csv of the image scans from https://github.com/rancher/image-scanning/tree/main/docs/_data, imported the data to Google sheets and used the QUERY function to aggregate the data a little better. This essentially made the sheet a mini database with the ability to use sql queries against it.
With that I found all of the packages that have CVEs and the count of images that we produce which contain those CVEs. We are not in control of theses packages, they are dependencies of upstream code bases that we build.

packages	image_count
golang.org/x/net	527
golang.org/x/crypto	229
golang.org/x/text	216
github.com/emicklei/go-restful	74
github.com/docker/distribution	53
github.com/prometheus/client_golang	43
gopkg.in/yaml.v3	27
github.com/containernetworking/cni	20
k8s.io/kubernetes	20
github.com/buger/jsonparser	12
github.com/docker/docker	10
github.com/gogo/protobuf	10
github.com/opencontainers/runc	9
github.com/caddyserver/caddy	6
github.com/containerd/containerd	6
github.com/dgrijalva/jwt-go	6
github.com/distribution/distribution	6
gopkg.in/yaml.v2	4
libcrypto3	4
libssl3	4
github.com/Masterminds/goutils	3
github.com/rancher/wrangler	2
k8s.io/apimachinery	2
ncurses-libs	1
ncurses-terminfo-base	1

[matttrach]

The data I have gathered and the processes I have used should give the team tools to more quickly discover false positives in the CVE scanning tools and quickly address any CVEs that are under our control.

[matttrach]

There is more that can be done with this, but I believe this resolves the immediate concerns.

[matttrach]

I moved this to the backlog because I was not actively working on it anymore and I am unaware of anyone else picking it up.

[matttrach]

sorry, I meant to also mention #4365 where I am adding the image data I collected to the RKE2 repo.

[caroline-suse-rancher]

We have resolved all CVEs that are within our direct control (in our codebase), and now we'll created a new issue for follow-on work that has more to do with dependencies.

[macedogm]

@matttrach thanks for the amazing work 👏🏻