The config_path of [plugins."io.containerd.grpc.v1.cri".registry] don't work.

[cyxinda]

Description

I config the containerd like this:

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = "/etc/containerd/certs.d"

And in the ~/.config/containerd/ is like this:

cyxinda@oldsix [~/.config/containerd] ➜  ls                                                                                                                                                                   [11:24:25]
certs.d  config.toml
cyxinda@oldsix [~/.config/containerd] ➜  tree                                                                                                                                                                 [11:24:25]
.
├── certs.d
│   └── docker.io
│       └── hosts.toml
└── config.toml

The content of certs.d/docker.io/hosts.toml is:

cyxinda@oldsix [~/.config/containerd] ➜  cat certs.d/docker.io/hosts.toml                                                                                                                                     [11:24:29]
server = "https://docker.io"
[host."https://jockerhub.com"]
  capabilities = ["pull", "resolve"]

I try to pull image from docker hub using the nerdctl in the version of 2.0.0-rc.1 and 2.0.0-rc.2,it works with the proxy server of https://jockerhub.com.
And like this:

cyxinda@oldsix [~/.config/containerd/certs.d/docker.io] ➜  nerdctl --version                                                                                                                                            [11:05:04]
nerdctl version 2.0.0-rc.1
cyxinda@oldsix [~/.config/containerd/certs.d/docker.io] ➜  containerd --version                                                                                                                                         [11:05:11]
containerd github.com/containerd/containerd/v2 v2.0.0-rc.3 27de5fea738a38345aa1ac7569032261a6b1e562
cyxinda@oldsix [~/.config/containerd/certs.d/docker.io] ➜  nerdctl pull docker.io/library/centos                                                                                                                        [11:06:30]
docker.io/library/centos:latest:                                               resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:a27fd8080b517143cbbbab9dfb7c8571c40d67d534bbdee55bd6c473f432b177: done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 56.2s                                                                 total:   0.0 B (0.0 B/s)                                         
FATA[0056] failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://jockerhub.com/v2/library/centos/manifests/sha256:a27fd8080b517143cbbbab9dfb7c8571c40d67d534bbdee55bd6c473f432b177?ns=docker.io": net/http: TLS handshake timeout 
---------------------------------------------------------------------------------------------------------------------------------------------------------
cyxinda@oldsix [~/.config/containerd/certs.d/docker.io] ➜  nerdctl --version                                                                                                                                            [11:08:36]
nerdctl version 2.0.0-rc.2
cyxinda@oldsix [~/.config/containerd/certs.d/docker.io] ➜  containerd --version                                                                                                                                         [11:09:47]
containerd github.com/containerd/containerd/v2 v2.0.0-rc.4 fa5bf66fdff44846dc475c022bf6b47197febffd
cyxinda@oldsix [~/.config/containerd/certs.d/docker.io] ➜  nerdctl pull docker.io/library/centos                                                                                                                        [11:12:28]
docker.io/library/centos:latest: resolving      |--------------------------------------| 
docker.io/library/centos:latest: resolving      |--------------------------------------| 
elapsed: 48.3s                   total:   0.0 B (0.0 B/s)                                         
INFO[0048] trying next host                              error="failed to do request: Head \"https://docker.io/v2/library/centos/manifests/latest\": dial tcp 162.125.32.12:443: i/o timeout" host=docker.io
FATA[0048] failed to resolve reference "docker.io/library/centos:latest": failed to do request: Head "https://jockerhub.com/v2/library/centos/manifests/latest?ns=docker.io": net/http: TLS handshake timeout 

When I do the same thing using the nerdctl in the version of 2.0.0-rc.3 and 2.0.0,it will not use the proxy server any more.

cyxinda@oldsix [~/.config/containerd/certs.d/docker.io] ➜  nerdctl --version                                                                                                                                            [11:13:50]
nerdctl version 2.0.0-rc.3
cyxinda@oldsix [~/.config/containerd/certs.d/docker.io] ➜  containerd --version                                                                                                                                         [11:13:58]
containerd github.com/containerd/containerd/v2 v2.0.0-rc.6 b70cce2085802e9e166ed8d92b42058c550f9ca3
cyxinda@oldsix [~/.config/containerd/certs.d/docker.io] ➜  nerdctl pull docker.io/library/centos                                                                                                                        [11:14:00]
docker.io/library/centos:latest: resolving      |--------------------------------------| 
elapsed: 29.9s                   total:   0.0 B (0.0 B/s)                                         
INFO[0030] trying next host                              error="failed to do request: Head \"https://registry-1.docker.io/v2/library/centos/manifests/latest\": dial tcp 31.13.88.169:443: i/o timeout" host=registry-1.docker.io
ERRO[0030] active check failed                           error="context canceled"
FATA[0030] failed to resolve reference "docker.io/library/centos:latest": failed to do request: Head "https://registry-1.docker.io/v2/library/centos/manifests/latest": dial tcp 31.13.88.169:443: i/o timeout 

--------------------------------------------------------------------------------------------------------------------------------------------------------------------
cyxinda@oldsix [~/.config/containerd/certs.d/docker.io] ➜  nerdctl --version                                                                                                                                  [11:40:31]
nerdctl version 2.0.0
cyxinda@oldsix [~/.config/containerd/certs.d/docker.io] ➜  containerd --version                                                                                                                               [11:40:34]
containerd github.com/containerd/containerd/v2 v2.0.0 207ad711eabd375a01713109a8a197d197ff6542
cyxinda@oldsix [~/.config/containerd/certs.d/docker.io] ➜  nerdctl pull docker.io/library/centos                                                                                                              [11:40:38]
docker.io/library/centos:latest: resolving      |--------------------------------------| 
elapsed: 29.9s                   total:   0.0 B (0.0 B/s)                                         
INFO[0030] fetch failed                                  error="failed to do request: Head \"https://registry-1.docker.io/v2/library/centos/manifests/latest\": dial tcp 64.13.192.76:443: i/o timeout" host=registry-1.docker.io
ERRO[0030] active check failed                           error="context canceled"
FATA[0030] failed to resolve reference "docker.io/library/centos:latest": failed to do request: Head "https://registry-1.docker.io/v2/library/centos/manifests/latest": dial tcp 64.13.192.76:443: i/o timeout 

I think this is a bug.

Steps to reproduce the issue

Describe the results you received and expected

I expect the nerdctl can use the proxy server refer to the config of hosts.toml .

What version of nerdctl are you using?

cyxinda@oldsix [/.config/containerd/certs.d/docker.io] ➜ nerdctl --version [11:40:31]
nerdctl version 2.0.0
cyxinda@oldsix [/.config/containerd/certs.d/docker.io] ➜ containerd --version [11:40:34]
containerd github.com/containerd/containerd/v2 v2.0.0 207ad711eabd375a01713109a8a197d197ff6542

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

cyxinda@oldsix [~/.config/containerd/certs.d/docker.io] ➜  nerdctl info                                                                                                                                       [11:41:16]
Client:
 Namespace:     default
 Debug Mode:    false

Server:
 Server Version: v2.0.0
 Storage Driver: overlayfs
 Logging Driver: json-file
  Cgroup Driver:  : systemd
  Cgroup Version: : 2
 Plugins:
  Log:     fluentd journald json-file none syslog
  Storage: native overlayfs
 Security Options:
  apparmor
  seccomp
   Profile:     builtin
  cgroupns
  rootless
 Kernel Version:   6.8.0-48-generic
 Operating System: Ubuntu 24.04.1 LTS
 OSType:           linux
 Architecture:     x86_64
 CPUs:             24
 Total Memory:     46.96GiB
 Name:             oldsix
 ID:               ca3e7d2c-4de1-4e7f-8f44-791f0d59cd75

WARNING: AppArmor profile "nerdctl-default" is not loaded.
         Use 'sudo nerdctl apparmor load' if you prefer to use AppArmor with rootless mode.
         This warning is negligible if you do not intend to use AppArmor.
WARNING: No cpuset support

[AkihiroSuda]

nerdctl is not a CRI application and does not consume CRI config:

nerdctl/docs/faq.md

Lines 89 to 93 in 7a0b958

	### nerdctl ignores `[plugins."io.containerd.grpc.v1.cri"]` config
	Expected behavior, because nerdctl does not use CRI (Kubernetes Container Runtime Interface) API.
	See the questions below for how to configure nerdctl.

[cyxinda]

nerdctl is not a CRI application and does not consume CRI config:

nerdctl/docs/faq.md

Lines 89 to 93 in 7a0b958

nerdctl ignores [plugins."io.containerd.grpc.v1.cri"] config

Expected behavior, because nerdctl does not use CRI (Kubernetes Container Runtime Interface) API.

See the questions below for how to configure nerdctl.

But there is a diffrent behivor between the version 2.0.0-rc.2 and the version 2.0.0.
And even if I use ctr ,it has the same behivor :

[root@sc-node-app-3 docker.io]# ctr -n k8s.io i pull docker.io/library/ubuntu:latest
WARN[0000] DEPRECATION: The `tracing` property of `[plugins."io.containerd.internal.v1".tracing]` is deprecated since containerd v1.6 and will be removed in containerd v2.0.Use OTEL environment variables instead: https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/ 
ctr: rpc error: code = DeadlineExceeded desc = failed to resolve image: failed to do request: Head "https://registry-1.docker.io/v2/library/ubuntu/manifests/latest": dial tcp 108.160.165.173:443: i/o timeout
[root@sc-node-app-3 docker.io]# ctr --version
ctr github.com/containerd/containerd/v2 v2.0.0

Will the ctr client also ignore this configuration plugins."io.containerd.grpc.v1.cri"?

[djdongjin]

@cyxinda you can try this hosts_dir nerdctl config:

https://github.com/containerd/nerdctl/blob/7a0b958f739cbe82d8f39c33471b76eec9bef96f/docs/config.md#:~:text=hosts_dir%20%20%20%20%20%20%3D%20%5B%22/etc/containerd/certs.d%22%2C%20%22/etc/docker/certs.d%22%5D

Will the ctr client also ignore this configuration plugins."io.containerd.grpc.v1.cri"?

Yes ctr also doesn't use containerd cri interface. ctr i pull has a --hosts-dir flag where your can specify that. crictl uses containerd's cri interface: https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md#container-runtime-interface-cri-cli