Compute Ressources: Kube-Reserved & System-Reserved

[sass1997]

I would like to see some recommandations about the configuration of kube-reserved and system-reserved of the Kubelet. I found some references from providers like AWS, Azure & Google how they configure the kube-reserved according to the vm-size. https://learnk8s.io/allocatable-resources

Also I'm not sure which c-groups for kube-reserved and system-reserved must be referenced. According to this design recommandation from Kubernetes. https://github.com/kubernetes/design-proposals-archive/blob/main/node/node-allocatable.md#recommended-cgroups-setup. If I want to follow this design document I would need to start the kubelet/containerd process in a seperate c group. This is acutally not forseen in the current installation, as far as I saw.

Happy for every Input

[brandond]

I would like to see some recommandations about the configuration of kube-reserved and system-reserved of the Kubelet.

These are really highly dependent on your specific configuration. The resources reserved for the kubelet and host operating system will depend entirely on how you're deploying your workloads and what else is running on the node besides things managed by Kubernetes. In general I have not seen these as necessary to tune unless you are exercising VERY tight control over the CPU and memory allocations for your workload, and need to ensure that the kubernetes scheduler can also account for the overhead of other processes running on the node.

[sass1997]

Sorry for the late reply. As you can may think off I created this dicussion here in combinaton of #3592. I found out that if I restrict the system.slice c-group, where also the rke2-agent or rke2-server is running as well as all other processes. This was done via

        - --system-reserved=memory=1Gi
        - --system-reserved-cgroup=/system.slice
        - --enforce-node-allocatable=system-reserved

After setting up didn't have any issues anymore with kubelet filling the cache, also there were no performance issues noticable. Due to the design recommandation and as well for the configuration possiblity of kubelet it would be really nice if there is at least the possiblity that the rke2-agent or rke2-server will be installed on a dedicated c-group.