Unable to pull/push image from harbor registry with nerdctl

[Dan4ik7]

Description

Unable to pull/push image from harbor registry with nerdctl

Pre-condition:

1. Configure the harbor and expose it via ingress: example harbor.app.com
2. register it in containerd host.toml file:
   cat /etc/containerd/certs.d/harbor.app.com/hosts.toml
   server = "https://harbor.app.com"

[host."https://harbor.app.com/kubernetes"]
capabilities = ["pull", "resolve", "push"]
ca = "/etc/containerd/certs.d/harbor.app.com/ca.crt"

Steps to reproduce the issue

1. try to login to harbor.app.com with nerdctl: sudo nerdctl login harbor.app.com
2. try to push an image to it: sudo nerdctl push harbor.app.com/kubernetes/nginx:latest
3. try to pull an image from registry: sudo nerdctl pull harbor.app.com/kubernetes/hello-world:1.2

Describe the results you received and expected

Actual results:

nms@node-1:~$ sudo nerdctl login harbor.app.com
WARNING: Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

nms@node-1:$ sudo nerdctl push harbor.app.com/kubernetes/nginx:latest
INFO[0000] pushing as a reduced-platform image (application/vnd.oci.image.index.v1+json, sha256:3c2fec5591022af849f07e55835e15ac66258716b508fa6f1b82a5b5291c998f)
index-sha256:3c2fec5591022af849f07e55835e15ac66258716b508fa6f1b82a5b5291c998f: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:3d696e8357051647b844d8c7cf4a0aa71e84379999a4f6af9b8ca1f7919ade42: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:66f8bdd3810c96dc5c28aec39583af731b34a2cd99471530f53c8794ed5b423e: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:57b64962dd94d4818372adf30dc0e2ca4803c46d4f638b7712fe01a149c705c5: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:8cc1569e58f52d008e232130d8fca2411f417ea423305cd7d7b513fb96d22947: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:362f35df001b4bd6f8733cd4abe8e1493582782404fefc2393129a5dfb5e72df: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:bc0965b23a04fe7f2d9fb20f597008fcf89891de1c705ffc1c80483a1f098e4f: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:13e320bf29cd3ef51b06a3dfe259b2582d48be27a9ac4c6b7af6fbb99429d210: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:7b50399908e1c0958c409f3c844d61736fd41e37a58dca4832927715508dd3aa: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:650ee30bbe5efddbef9cc0245ba52b133d3c8897a6565faa6c5c87bc552b5305: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.1 s total: 913.0 (8.9 KiB/s)
FATA[0000] failed commit on ref "index-sha256:3c2fec5591022af849f07e55835e15ac66258716b508fa6f1b82a5b5291c998f": unexpected status from PUT request to https://harbor.app.com/kubernetes/v2/kubernetes/nginx/manifests/latest: 405 Not Allowed
nms@node-1:$ sudo nerdctl pull harbor.app.com/kubernetes/hello-world:1.2
WARN[0000] reference for unknown type: text/html digest="sha256:b6a377d90af951a0588dd82b1d02c6517ba29042f1f5136de1ebdb88df5b4602" mediatype=text/html size=785
FATA[0000] failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:b6a377d90af951a0588dd82b1d02c6517ba29042f1f5136de1ebdb88df5b4602: not found
nms@node-1:~$

What version of nerdctl are you using?

nms@node-1:~$ sudo nerdctl version
Client:
Version: v2.0.2
OS/Arch: linux/amd64
Git commit: 1220ce7
buildctl:
Version: v0.18.1
GitCommit: eb68885955169461d72dc2b7e6d084100fcaba86

Server:
containerd:
Version: 1.7.12
GitCommit:
runc:
Version: 1.2.2
GitCommit: v1.2.2-0-g7cb36325

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

However I have an external ubuntu machine where I have docker on it and harbor.app.com as insecure registry, and I am able to pull push images to that registry that runs on k8s cluster from where I use nerdctl trying to pull/push images.

nms@nfs:$ docker push harbor.app.com/kubernetes/hello-world:1.2
The push refers to repository [harbor.app.com/kubernetes/hello-world]
ac28800ec8bb: Layer already exists
1.2: digest: sha256:d37ada95d47ad12224c205a938129df7a3e52345828b4fa27b03a98825d1e2e7 size: 524
nms@nfs:$ cat /etc/docker/daemon.json
{
"insecure-registries": ["harbor.app.com"]
}
nms@nfs:~$

[apostasie]

405 is Harbor refusing to serve the requests.

This is somewhat typical when the ingress rules are pointing to the wrong location IIRC.

Can you share how exactly you started and configured Harbor?

Also, can you re-run your nerdctl pull and push commands with the extra argument --debug-full

Thanks.