k3s is consuming 535 mb of memory consistently in a single node cluster on an idle state

[srinicrick65]

Environmental Info:
K3s Version:
k3s version v1.21.2+k3s1 (5a67e8d)
go version go1.16.4
2GB Ram

Node(s) CPU architecture, OS, and Version:
k3s Single Node cluster setup with default setup
Os : Ubuntu 18.04 and also 20.04

Cluster Configuration:
default k3s configuration
curl -sfL https://get.k3s.io | sh -

Describe the bug:
We ran the below curl command to install k3s for single node k3s cluster setup . Once the cluster came up we could see that the process "k3s server" is consuming 565 mb of memory . There seems to be a serious memory overhead even at the idle State . I have a plan of deploying the k3s in Raspberry pi 3 which is having 1 gb ram and because of the memory overhead of k3s , this may fail in bringing up the k3s cluster in Raspberry pi 3 device.

Please find the attachment for the footprint captured
k3s-memory-cpu.xlsx
k3slog.txt

Below is the command executed
curl -sfL https://get.k3s.io | sh -

Steps To Reproduce:

• Installed K3s by running url -sfL https://get.k3s.io | sh -
• Ran "top -i" command to check the process which are consuming more memory and cpu
• I could see k3s-server is consuming 27.7 % of memory (Assigned memory to the host is 2gb) which means k3s-server is consuming 552 Mb of memory usage.
• Please find the metrics below

<style> </style>

Command: free -h

total        used        free      shared  buff/cache   available
Mem:           1.9G        1.6G         77M         20M        256M        154M
Swap:            0B           0B              0B

<style> </style>

Command: top -i

PID         USER      %CPU        %MEM         TIME+ COMMAND
16800         root      18.6                 26.8        1:02.08 k3s-server
16844         root       1.0                   3.5          0:11.22 containerd
 
Command: top -i
PID           USER        %CPU    %MEM     TIME+ COMMAND
16800           root       9.3             25.9        1:31.07 k3s-server
16844           root       1.0              3.5         0:13.38 containerd
 
Command: kubectl  top pods -A
NAMESPACE     NAME                                                       CPU(cores)   MEMORY(bytes)
kube-system   coredns-7448499f4d-bk7sv                                5m           17Mi
kube-system   local-path-provisioner-5ff76fc89d-dqqjs       3m           12Mi
kube-system   metrics-server-86cbb8457f-4thzp                    2m           17Mi
kube-system   svclb-traefik-s68d8                                               0m           2Mi
kube-system   traefik-97b44b794-7xxdk                                    2m           35Mi

[brandond]

This doesn't sound unreasonable, have you taken a look at the resource profiling page in the documentation?
https://rancher.com/docs/k3s/latest/en/installation/installation-requirements/resource-profiling/

Also, if you're going to submit logs, please make them an attachment rather than inline to avoid making folks to scroll through pages and pages.

[srinicrick65]

Please find the attachments which contains the footprint

k3slog.txt
Uploading k3s-memory-cpu.xlsx…

[srinicrick65]

This doesn't sound unreasonable, have you taken a look at the resource profiling page in the documentation?
https://rancher.com/docs/k3s/latest/en/installation/installation-requirements/resource-profiling/

I did verify . It claims that K3s cluster with a single agent should be good with 768 M Ram . Below is my system configuration
Process : Intel(R) Core(TM) i7-9850H CPU @ 2.60GHz 2.59 GHz
Installed Ram in machine is : 16GB and I used VM in hyper v for testing k3s and I configured 2GB RAM

I also Tested the same in Raspberry Pi 3 Model B+
Processor : 64 bit quad-core ARMv8 CPU
Ram : 1GB

In both the system config I see that "k3s server" process is consuming 535 MB of memory with 15% CPU.

Also, if you're going to submit logs, please make them an attachment rather than inline to avoid making folks to scroll through pages and pages.

Thanks for that . I did.

[brandond]

I did verify . It claims that K3s cluster with a single agent should be good with 768 M Ram

I think you might be misunderstanding; that page says that the k3s server process with a single agent using embedded etcd, is expected to consume 768MB of RAM. It does not mean that the node itself needs only 768 MB of RAM. If you want any memory left for workloads, disk cache, etc you will need to size the nodes appropriately.

[srinicrick65]

What we are currently using is single node cluster with default setup where both agent and server are in the same node. Also what we can see after analysis is that the memory consumption is due to garbage collection from "k3s server" process when it runs and after we set the GOGC value to 10 in the k3s systemd service environment the memory drastically reduced to 300 MB from 535MB. Can someone from k3s look into this on why there is a lot of garbage collection and if there are any active threads which is not getting killed?

[brandond]

Server and agent cannot run on the same node; what you're describing is listed in that table as K3s server with a workload. Both Kubernetes and Go will aggressively cache things if possible; you can play with go GC and/or setting a lower memory limit on the systemd cgroup that k3s runs in, but (in my experience) both of those have negative performance impacts that outweigh the memory savings.

[srinicrick65]

Please reopen this issue as this is still not fixed

[brandond]

I didn't close it, I just converted it to a discussion since this is not a defect, but rather Q/A about tuning memory utilizaton.

[srinicrick65]

My Bad @brandond . I didn't notice it properly