How to Optimize and Manage Node Storage

[abhishek-ch]

Recently, I noticed a storage warning for my nodes, which is unusual. Upon investigating further, I discovered a significant increase in the node disk size.

 du -h / 2>/dev/null | sort -rh | head -20
57G	/host
57G	/
46G	/host/var/lib
46G	/host/var
45G	/host/var/lib/rancher/k3s/agent/containerd
45G	/host/var/lib/rancher/k3s/agent
45G	/host/var/lib/rancher/k3s
45G	/host/var/lib/rancher
33G	/host/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots
33G	/host/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs
13G	/host/var/lib/rancher/k3s/agent/containerd/io.containerd.content.v1.content/blobs/sha256
13G	/host/var/lib/rancher/k3s/agent/containerd/io.containerd.content.v1.content/blobs
13G	/host/var/lib/rancher/k3s/agent/containerd/io.containerd.content.v1.content
9.8G	/host/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io
9.8G	/host/run/k3s/containerd/io.containerd.runtime.v2.task
9.8G	/host/run/k3s/containerd
9.8G	/host/run/k3s
9.8G	/host/run
2.9G	/host/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/347/fs
2.9G	/host/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/347

Any recommendation or configuration to control this size, I am not sure about the cause

[vitobotta]

I can't confirm with an actual cluster right now since I am on the move, but I have this in my wiki: you can configure the garbage collection in Containerd by editing /etc/rancher/k3s/config.yaml and set:

containerd:
  snapshotter:
    prune_on_startup: true

Then restarting k3s:

systemctl restart k3s

However I am not sure if this now happens automatically with newer versions of k3s.

[vitobotta]

Other things I have in my wiki that might help:

# List all container images
sudo k3s crictl images

# Remove unused images
sudo k3s crictl rmi --prune

# Run garbage collection for containerd
sudo k3s crictl rmp --prune

Also, configure the GC for images in k3s: edit the K3s configuration file (usually located at /etc/rancher/k3s/config.yaml or /etc/k3s/config.yaml) and add or configure the following:

kubelet-arg:
  - "image-gc-high-threshold=85"
  - "image-gc-low-threshold=70"

image-gc-high-threshold: The percentage of disk usage at which garbage collection is triggered.
image-gc-low-threshold: The percentage of disk usage to which garbage collection will reduce disk usage.

After making changes, restart the K3s service:

sudo systemctl restart k3s

[abhishek-ch]

well in my node, its not showing any command systemctl or k3s, I managed to find k3s in /host/etc/rancher/k3s/ but no systemctl

[vitobotta]

which OS are you using?

[abhishek-ch]

default so ubuntu-24.04

kubeconfig_path: "./kubeconfig"
k3s_version: v1.31.4+k3s1

networking:
  ssh:
    port: 22
    use_agent: false # set to true if your key has a passphrase
    public_key_path: "~/.ssh/rsa.pub"
    private_key_path: "~/.ssh/rsa"
  allowed_networks:
    ssh:
      - 0.0.0.0/0
    api: # this will firewall port 6443 on the nodes; it will NOT firewall the API load balancer
      - 0.0.0.0/0
  public_network:
    ipv4: true
    ipv6: true
  private_network:
    enabled : true
    subnet: 10.0.0.0/16
    existing_network_name: ""
  cni:
    enabled: true
    encryption: false
    mode: flannel

  cluster_cidr: 10.244.0.0/16 # optional: a custom IPv4/IPv6 network CIDR to use for pod IPs
  service_cidr: 10.43.0.0/16 # optional: a custom IPv4/IPv6 network CIDR to use for service IPs. Warning, if you change this, you should also change cluster_dns!
  cluster_dns: 10.43.0.10 # optional: IPv4 Cluster IP for coredns service. Needs to be an address from the service_cidr range


  manifests:
    cloud_controller_manager_manifest_url: "https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/download/v1.20.0/ccm-networks.yaml"
    csi_driver_manifest_url: "https://raw.githubusercontent.com/hetznercloud/csi-driver/v2.9.0/deploy/kubernetes/hcloud-csi.yml"
    system_upgrade_controller_deployment_manifest_url: "https://github.com/rancher/system-upgrade-controller/releases/download/v0.13.4/system-upgrade-controller.yaml"
    system_upgrade_controller_crd_manifest_url: "https://github.com/rancher/system-upgrade-controller/releases/download/v0.13.4/crd.yaml"
    cluster_autoscaler_manifest_url: "https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/hetzner/examples/cluster-autoscaler-run-on-master.yaml"

datastore:
  mode: etcd

schedule_workloads_on_masters: false

masters_pool:
  instance_type: cx22
  instance_count: 1
  location: nbg1

worker_node_pools:
- name: small
  instance_type: cx32
  instance_count: 1
  labels:
    - key: instance_type
      value: cx32
  location: nbg1

[vitobotta]

I don't understand how that's possible considering that Ubuntu 24.04 uses systemd