Docker image ID verification fails with containerd engine enabled Docker Desktop

[hannes-ucsc]

In order to free up disk space on my Mac, I clicked the Clean/Purge Data button on the Troubleshooting page of the Docker Desktop Dashboard. This reset some setting and caused the new containerd image store to be enabled. When I later ran the tests in test_indexer.py, the Elasticsearch image would be pulled successfully, but our assertion that checks the image ID failed. The image ID returned by Docker (via the Docker Python client) was the mirror digest instead of the image ID (aka config digest).

In this state, all of the IDs displayed by Docker were the repo digests, not the image ID:

$ docker inspect 15df3a90b5521f1ea9ae6f9d596fe2da9ea5a1539314db2c877b3dac360c7061
[
    {
        "Id": "sha256:15df3a90b5521f1ea9ae6f9d596fe2da9ea5a1539314db2c877b3dac360c7061",
        "RepoTags": [
            "ucscgi/azul-elasticsearch:7.17.24-29"
        ],
        "RepoDigests": [
            "ucscgi/azul-elasticsearch@sha256:15df3a90b5521f1ea9ae6f9d596fe2da9ea5a1539314db2c877b3dac360c7061"
        ],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2024-10-16T18:05:01.218931381Z",
        "DockerVersion": "27.2.0",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "9200/tcp": {},
                "9300/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "ELASTIC_CONTAINER=true"
            ],
            "Cmd": [
                "eswrapper"
            ],
            "ArgsEscaped": true,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "/usr/share/elasticsearch",
            "Entrypoint": [
                "/bin/tini",
                "--",
                "/usr/local/bin/docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "org.label-schema.build-date": "2024-09-05T07:34:51.812485320Z",
                "org.label-schema.license": "Elastic-License-2.0",
                "org.label-schema.name": "Elasticsearch",
                "org.label-schema.schema-version": "1.0",
                "org.label-schema.url": "https://www.elastic.co/products/elasticsearch",
                "org.label-schema.usage": "https://www.elastic.co/guide/en/elasticsearch/reference/index.html",
                "org.label-schema.vcs-ref": "fcf25fff740db6ab3ed5d145c58d70e4c3528ea7",
                "org.label-schema.vcs-url": "https://github.com/elastic/elasticsearch",
                "org.label-schema.vendor": "Elastic",
                "org.label-schema.version": "7.17.24",
                "org.opencontainers.image.created": "2024-09-05T07:34:51.812485320Z",
                "org.opencontainers.image.documentation": "https://www.elastic.co/guide/en/elasticsearch/reference/index.html",
                "org.opencontainers.image.licenses": "Elastic-License-2.0",
                "org.opencontainers.image.ref.name": "ubuntu",
                "org.opencontainers.image.revision": "fcf25fff740db6ab3ed5d145c58d70e4c3528ea7",
                "org.opencontainers.image.source": "https://github.com/elastic/elasticsearch",
                "org.opencontainers.image.title": "Elasticsearch",
                "org.opencontainers.image.url": "https://www.elastic.co/products/elasticsearch",
                "org.opencontainers.image.vendor": "Elastic",
                "org.opencontainers.image.version": "7.17.24"
            }
        },
        "Architecture": "arm64",
        "Os": "linux",
        "Size": 389844935,
        "GraphDriver": {
            "Data": null,
            "Name": "overlayfs"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:171652ecd561458ec6fcca88016f1d5552155ecaff2445a38a97f49d6b672904",
                "sha256:0d960a5642e05d4bb0b928019fb2f56b36494a0e6a6e00517599d698e0c86708",
                "sha256:daeb83f8de63e712b3a0c9c3b5716903b69a9ce45917be4935dc954e1cb2e5fa",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
                "sha256:f23acc1a346c9f05f995b3e10998d98d3591bb1a7a71db32eb0f82e3b397f269",
                "sha256:0545ed68bcc3224379dc4db0655833dc344960259263ef364deb728b1cd7dbd7",
                "sha256:0b001537c997a8d84f674051dd5773dbc34af58f7f024525467bc282cdaec545",
                "sha256:8a9649b794c30625803411dafaf9a3e4d62d7d98f5b2781c0eafeb402366f825",
                "sha256:ea40a75200fc1875f2fff00ed0420f463b413cbc20f9114554e388847670b19d",
                "sha256:7e1b634dbd2940762861d132798c7308fdea2631434d891e473d65e7a78130c5",
                "sha256:dd49d3f2946d75c2d919b33db528c4138b3d5de0a93e5a9137c50a682ff0f424",
                "sha256:5c3f04dae206acf7f66067d5428e24adb861af8a4b8e587cf83bb60b35fbe64e",
                "sha256:0b8380f66cac0e081d0c60e8410985b8389901176b1699fcfc64d8d6ab7ffcc6",
                "sha256:519349b7f3711867fb2afec45e6454a93089ede30d63e81d971717ff50184622",
                "sha256:3277629cecfcb2bdfa84e2cb6b38572b8e10007e0f7f768bdd4203982048b2d0"
            ]
        },
        "Metadata": {
            "LastTagTime": "2024-10-25T04:33:57.795001179Z"
        }
    }
]

Compare this with the output of docker inspect after disabling the containerd
image storage again.

$ docker inspect 00352eb62158
[
    {
        "Id": "sha256:00352eb621589b9ba28912cccbb0cb40326c63758668d5947ce9d74cbe1e4347",
        "RepoTags": [],
        "RepoDigests": [
            "122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/ucscgi/azul-elasticsearch@sha256:acd4ca93dce8693fec18cc0caea3ed62c0e9cf057a621d67f9d7fd06ef31ae2f"
        ],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2024-10-16T18:05:01.218931381Z",
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "9200/tcp": {},
                "9300/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "ELASTIC_CONTAINER=true"
            ],
            "Cmd": [
                "eswrapper"
            ],
            "ArgsEscaped": true,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "/usr/share/elasticsearch",
            "Entrypoint": [
                "/bin/tini",
                "--",
                "/usr/local/bin/docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "org.label-schema.build-date": "2024-09-05T07:34:51.812485320Z",
                "org.label-schema.license": "Elastic-License-2.0",
                "org.label-schema.name": "Elasticsearch",
                "org.label-schema.schema-version": "1.0",
                "org.label-schema.url": "https://www.elastic.co/products/elasticsearch",
                "org.label-schema.usage": "https://www.elastic.co/guide/en/elasticsearch/reference/index.html",
                "org.label-schema.vcs-ref": "fcf25fff740db6ab3ed5d145c58d70e4c3528ea7",
                "org.label-schema.vcs-url": "https://github.com/elastic/elasticsearch",
                "org.label-schema.vendor": "Elastic",
                "org.label-schema.version": "7.17.24",
                "org.opencontainers.image.created": "2024-09-05T07:34:51.812485320Z",
                "org.opencontainers.image.documentation": "https://www.elastic.co/guide/en/elasticsearch/reference/index.html",
                "org.opencontainers.image.licenses": "Elastic-License-2.0",
                "org.opencontainers.image.ref.name": "ubuntu",
                "org.opencontainers.image.revision": "fcf25fff740db6ab3ed5d145c58d70e4c3528ea7",
                "org.opencontainers.image.source": "https://github.com/elastic/elasticsearch",
                "org.opencontainers.image.title": "Elasticsearch",
                "org.opencontainers.image.url": "https://www.elastic.co/products/elasticsearch",
                "org.opencontainers.image.vendor": "Elastic",
                "org.opencontainers.image.version": "7.17.24"
            }
        },
        "Architecture": "arm64",
        "Os": "linux",
        "Size": 665975876,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/ecd7b11b854da6dae0d35d75c8415b79c5f09c7e4bc5d390bd8566a7440eba36/diff:/var/lib/docker/overlay2/5fec58d8a7ea2055feabb85b43178eb15fc2b23a3e8bcf609b926e129c11165e/diff:/var/lib/docker/overlay2/f2d684f2cbdb309c1710ac79536dffaf13df326bdbccf239e62179f0c3ba27ae/diff:/var/lib/docker/overlay2/c0b0c41713e9d94ce7feee187f99ac5d4eeac1396cf875d520b5c0e8051c067e/diff:/var/lib/docker/overlay2/f49405aa1b9783707da57d21ffbe9a36b99ecb86374605f60c843336c4239f75/diff:/var/lib/docker/overlay2/e51611cd241cdc5f236db57133df572a813ed3501723321af0b06defd96bfc70/diff:/var/lib/docker/overlay2/20c93503ad96248cc527d44177046b0cb7c7cb8cf3834dc8c1408bbe50b01cdc/diff:/var/lib/docker/overlay2/f6e9247798de6e550f6a840a77b64399738d49028e05ca5b118af781483fd5e1/diff:/var/lib/docker/overlay2/0c6f0567ccd2d8b56ea2658364562886edfe49714031a5b10089109c3e6e7421/diff:/var/lib/docker/overlay2/e3f075599eddb74c177fbdf4e830684fdb46b05061501b5982e318e0433e291b/diff:/var/lib/docker/overlay2/eda982e0774884c970115a00b2e9326d92e706cce4e8faf5e5878eddc758ce94/diff:/var/lib/docker/overlay2/d48f2ffd7f31018daa46043f51247a36cb34be42da7275d39385c8852542c9a3/diff:/var/lib/docker/overlay2/587807647b2f94eee7ab265cad4e91ce20c2d521b3aa1f98b160879b0dc7de2c/diff:/var/lib/docker/overlay2/d66da794e9d135dded74185f23b822f49962f598e73eab83cdd7e8a6e9f30215/diff",
                "MergedDir": "/var/lib/docker/overlay2/f482b963785a916119b59a42c484d165d12e3923de6a6bcaaaf222dcec31f308/merged",
                "UpperDir": "/var/lib/docker/overlay2/f482b963785a916119b59a42c484d165d12e3923de6a6bcaaaf222dcec31f308/diff",
                "WorkDir": "/var/lib/docker/overlay2/f482b963785a916119b59a42c484d165d12e3923de6a6bcaaaf222dcec31f308/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:171652ecd561458ec6fcca88016f1d5552155ecaff2445a38a97f49d6b672904",
                "sha256:0d960a5642e05d4bb0b928019fb2f56b36494a0e6a6e00517599d698e0c86708",
                "sha256:daeb83f8de63e712b3a0c9c3b5716903b69a9ce45917be4935dc954e1cb2e5fa",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
                "sha256:f23acc1a346c9f05f995b3e10998d98d3591bb1a7a71db32eb0f82e3b397f269",
                "sha256:0545ed68bcc3224379dc4db0655833dc344960259263ef364deb728b1cd7dbd7",
                "sha256:0b001537c997a8d84f674051dd5773dbc34af58f7f024525467bc282cdaec545",
                "sha256:8a9649b794c30625803411dafaf9a3e4d62d7d98f5b2781c0eafeb402366f825",
                "sha256:ea40a75200fc1875f2fff00ed0420f463b413cbc20f9114554e388847670b19d",
                "sha256:7e1b634dbd2940762861d132798c7308fdea2631434d891e473d65e7a78130c5",
                "sha256:dd49d3f2946d75c2d919b33db528c4138b3d5de0a93e5a9137c50a682ff0f424",
                "sha256:5c3f04dae206acf7f66067d5428e24adb861af8a4b8e587cf83bb60b35fbe64e",
                "sha256:0b8380f66cac0e081d0c60e8410985b8389901176b1699fcfc64d8d6ab7ffcc6",
                "sha256:519349b7f3711867fb2afec45e6454a93089ede30d63e81d971717ff50184622",
                "sha256:3277629cecfcb2bdfa84e2cb6b38572b8e10007e0f7f768bdd4203982048b2d0"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

The respective section of docker_images.json at that time:

    "docker.io/ucscgi/azul-elasticsearch:7.17.24-29": {
        "digest": "sha256:15df3a90b5521f1ea9ae6f9d596fe2da9ea5a1539314db2c877b3dac360c7061",
        "mirror_digest": "sha256:acd4ca93dce8693fec18cc0caea3ed62c0e9cf057a621d67f9d7fd06ef31ae2f",
        "parts": {
            "linux/amd64": {
                "digest": "sha256:e306230f4b04bc48d4f155be9316cc992c4e0e1e6849e176ec28edff8c23a760",
                "id": "sha256:6ae5f90233036b6c5ea6fcb498b7c431d6b3d1e9eaa810d5b1cf02e259f31203",
                "platform": "linux/amd64"
            },
            "linux/arm64": {
                "digest": "sha256:f5a536692efe2c9f6eb3c6c264211049f30023154c9ec177a549df391ef3733c",
                "id": "sha256:00352eb621589b9ba28912cccbb0cb40326c63758668d5947ce9d74cbe1e4347",
                "platform": "linux/arm64"
            }
        }
    },

To work around this, disable the containerd image store. You should delete all images beforehand since merely disabling the store will just hide them in which case I suspect they'll continue to use up space.