Failed to pull image "nvcr.io/nvidia/driver:550.90.07-debian12" in nvidia-driver-daemonset Pod

[utsumi-fj]

In A100 GPU environment, when installing GPU Operator, the error Failed to pull image "nvcr.io/nvidia/driver:550.90.07-debian12" occurs in nvidia-driver-daemonset Pod.

The status of nvidia-driver-daemonset Pod is ImagePullBackOff as follows:

# kubectl get pod -n gpu-operator
NAME                                                              READY   STATUS             RESTARTS   AGE
gpu-feature-discovery-sz7tw                                       0/1     Init:0/1           0          34m
gpu-operator-1737329855-node-feature-discovery-gc-54495f58tffkd   1/1     Running            0          34m
gpu-operator-1737329855-node-feature-discovery-master-77876hrzc   1/1     Running            0          34m
gpu-operator-1737329855-node-feature-discovery-worker-dcsjb       1/1     Running            0          34m
gpu-operator-7b8f77b698-rp5q8                                     1/1     Running            0          34m
nvidia-container-toolkit-daemonset-z6pzk                          0/1     Init:0/1           0          34m
nvidia-dcgm-exporter-vpdqp                                        0/1     Init:0/1           0          34m
nvidia-device-plugin-daemonset-gqvpn                              0/1     Init:0/1           0          34m
nvidia-driver-daemonset-pwnhc                                     0/1     ImagePullBackOff   0          34m
nvidia-operator-validator-gz7j4                                   0/1     Init:0/4           0          34m

The image nvcr.io/nvidia/driver:550.90.07-debian12 is not found as follows:

# kubectl describe pod nvidia-driver-daemonset-pwnhc -n gpu-operator
(...snip...)
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  36m                  default-scheduler  Successfully assigned gpu-operator/nvidia-driver-daemonset-pwnhc to kubeflow-control-plane
  Normal   Pulled     36m                  kubelet            Container image "nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.6.10" already present on machine
  Normal   Created    36m                  kubelet            Created container k8s-driver-manager
  Normal   Started    36m                  kubelet            Started container k8s-driver-manager
  Normal   Pulling    34m (x4 over 36m)    kubelet            Pulling image "nvcr.io/nvidia/driver:550.90.07-debian12"
  Warning  Failed     34m (x4 over 36m)    kubelet            Failed to pull image "nvcr.io/nvidia/driver:550.90.07-debian12": rpc error: code = NotFound desc = failed to pull and unpack image "nvcr.io/nvidia/driver:550.90.07-debian12": failed to resolve reference "nvcr.io/nvidia/driver:550.90.07-debian12": nvcr.io/nvidia/driver:550.90.07-debian12: not found
  Warning  Failed     34m (x4 over 36m)    kubelet            Error: ErrImagePull
  Warning  Failed     34m (x5 over 35m)    kubelet            Error: ImagePullBackOff
  Normal   BackOff    71s (x152 over 35m)  kubelet            Back-off pulling image "nvcr.io/nvidia/driver:550.90.07-debian12"

The installation command:

# helm install --wait --generate-name \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --version=v24.6.2

The following is my environment information.

Environment:

• kind cluster with single node
• k8s version: v1.31.0
• node OS: Ubuntu 22.04.5 LTS
• Linux Kernel version: 5.15.0-130-generic
• A100 GPU passthrough
• GPU Operator version: v24.6.2

# kubectl get node
NAME                     STATUS   ROLES           AGE     VERSION
kubeflow-control-plane   Ready    control-plane   2d22h   v1.31.0

# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

# uname -r
5.15.0-130-generic

[klongmitre]

I'm having a similar issue with nvcr.io/nvidia/driver:550.144.03-ubuntu22.04. Were you able to figure this out @utsumi-fj ?

[neelamari]

i have similar issues on RHEL 8.10. Details below

kubectl get pods -n gpu-operator
NAME READY STATUS RESTARTS AGE
gpu-feature-discovery-pvb7d 0/1 Init:0/2 0 14s
gpu-operator-58dcc865fd-4ntdb 1/1 Running 0 64s
gpu-operator-node-feature-discovery-gc-7f546fd4bc-jshjv 1/1 Running 0 64s
gpu-operator-node-feature-discovery-master-8448c8896c-gdz2q 1/1 Running 0 64s
gpu-operator-node-feature-discovery-worker-rg7fm 1/1 Running 0 64s
gpu-operator-node-feature-discovery-worker-rr5xs 1/1 Running 0 64s
gpu-operator-node-feature-discovery-worker-xldq5 1/1 Running 0 64s
nvidia-container-toolkit-daemonset-rxms2 0/1 Init:0/1 0 14s
nvidia-dcgm-exporter-jm4mq 0/1 Init:0/1 0 14s
nvidia-dcgm-exporter-wxqng 0/1 Terminating 0 26s
nvidia-device-plugin-daemonset-6p75m 0/1 Terminating 0 27s
nvidia-device-plugin-daemonset-v9jlq 0/1 Init:0/1 0 14s
nvidia-driver-daemonset-4.18.0-553.16.1.el8.10-rhel8.10-5jrds 0/1 Init:0/1 0 28s
nvidia-driver-daemonset-4.18.0-553.16.1.el8.10-rhel8.10-f2m7r 0/1 PodInitializing 0 28s
nvidia-operator-validator-vm6sn 0/1 Init:0/4 0 14s

kubectl describe pod nvidia-driver-daemonset-pwnhc -n gpu-operator
(...snip...)
Events:
Type Reason Age From Message

---

Normal Scheduled 36m default-scheduler Successfully assigned gpu-operator/nvidia-driver-daemonset-pwnhc to kubeflow-control-plane
Normal Pulled 36m kubelet Container image "nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.6.10" already present on machine
Normal Created 36m kubelet Created container k8s-driver-manager
Normal Started 36m kubelet Started container k8s-driver-manager
Normal Pulling 34m (x4 over 36m) kubelet Pulling image "nvcr.io/nvidia/driver:550.90.07-debian12"
Warning Failed 34m (x4 over 36m) kubelet Failed to pull image "nvcr.io/nvidia/driver:550.90.07-debian12": rpc error: code = NotFound desc = failed to pull and unpack image "nvcr.io/nvidia/driver:550.90.07-debian12": failed to resolve reference "nvcr.io/nvidia/driver:550.90.07-debian12": nvcr.io/nvidia/driver:550.90.07-debian12: not found
Warning Failed 34m (x4 over 36m) kubelet Error: ErrImagePull
Warning Failed 34m (x5 over 35m) kubelet Error: ImagePullBackOff
Normal BackOff 71s (x152 over 35m) kubelet Back-off pulling image "nvcr.io/nvidia/driver:550.90.07-debian12"
The installation command:

--command used to install
helm install --wait gpu-operator
-n gpu-operator --create-namespace
nvidia/gpu-operator
--version=v24.9.2
--set driver.usePrecompiled=true
--set driver.version="570.86.15-rhel8.10"

The following is my environment information.

Environment:
3 node k8s cluster
k8s version: v1.31.0
node OS: rhel 8.10
GPU Operator version: v24.6.2

I am logged in to docker repo on nvcr.io. with API Key