bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid argument: unknown

[svendaems]

Description

Hi

I followed the procedure from https://docs.docker.com/engine/install/ubuntu/ to install Docker on my new Odroid C4 (ARM64). This Odroid is freshly installed with a Ubuntu 22.04 image (https://wiki.odroid.com/odroid-c4/getting_started/os_installation_guide?redirect=1#tab__odroid-c4hc4).

After running the command sudo docker run hello-world, I get following error
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid argument: unknown. ERRO[0000] error waiting for container:

I also tried to upgrade (apt full-upgrade) as suggested on #3481, but this also didn't worked.

Reproduce

sudo docker run hello-world
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid argument: unknown.
ERRO[0000] error waiting for container:

Expected behavior

No response

docker version

Client: Docker Engine - Community
 Version:           23.0.6
 API version:       1.42
 Go version:        go1.19.9
 Git commit:        ef23cbc
 Built:             Fri May  5 21:18:33 2023
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.6
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.9
  Git commit:       9dbdbd4
  Built:            Fri May  5 21:18:33 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.17.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 1
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 4.9.337-11
 Operating System: Ubuntu 22.04.2 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 3.622GiB
 Name: odroiDocker
 ID: 205c03e0-f4ab-4d12-8e20-74f1f8deaaa6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support

Additional Info

No response

[cpuguy83]

It is very strange you are running kernel 4.9 on Ubuntu 22.04?
A standard 22.04 kernel should be a much higher kernel version, which I suspect will fix this.

In any case this may be a bug in runc where it should use some fallback behavior. Though I'm no expert in this bit of code and I'm not sure if 4.9 is just too old to use with how cgroupv2 support is implemented.
I opened opencontainers/runc#3867 to look into this.

[cpuguy83]

The kernel you have installed is just missing functionality that runc currently assumes would be there.

[briansaycocie]

I'm getting the same error on my Odroid N2+ with the Ubuntu 22.04 image.

[MatthewSukyoung]

I got the same error now. Should the bpf program for cgroupv2 be written by myself and be installed?

[cpuguy83]

Upgrade your kernels.

[MatthewSukyoung]

My kernel version is 5.15. I might have to backport some part of the kernel.
Which part should be backported to figure this out? Can I get some more advice?

[thaJeztah]

I'm not sure about all the things that could be missing, but the check-config script could at least point out some options; https://github.com/moby/moby/blob/master/contrib/check-config.sh

[kolyshkin]

So, you want to use cgroup v2 with a very old kernel. Please see https://github.com/opencontainers/runc/blob/main/docs/cgroup-v2.md for runc cgroup v2 requirements (TL;DR: linux >= v5.2 is recommended, v4.15 is the absolute minimum).

If cgroup v2 is not a hard requirement, switch to cgroup v1 (e.g. by adding systemd.unified_cgroup_hierarchy=0 kernel boot parameter). If it is a hard requirement, use a newer kernel.

[cpuguy83]

Thanks @kolyshkin!

Going to go ahead and close this since it seems there's nothing actionable here.

[svendaems]

I couldn't upgrade the kernel, so I ended up in installing Armbian (https://www.armbian.com/odroid-c4/)

[briansaycocie]

This worked for me: hardkernel/linux#433 (comment)

I ended up using the Odroid community Ubuntu image which uses kernel v5.15.

[arghle]

For others in a similar situation, who may not be able to swap the whole OS image:

If cgroup v2 is not a hard requirement, switch to cgroup v1 (e.g. by adding systemd.unified_cgroup_hierarchy=0 kernel boot parameter). If it is a hard requirement, use a newer kernel.

This worked for me, I added systemd.unified_cgroup_hierarchy=0 to bootargs in /media/boot/boot.ini and docker is now back up and running. Thanks @kolyshkin !