"404 Not Found" on layer push

[grzleadams]

Version

{                                                                                                                                                                                              
  "versions": [                                                                                                                                                                                             
    {                                                                                                                                                                                                       
      "component": "core",                                                                            
      "version": "3.49.1",                                                                            
      "package": "pulpcore",                                                                                                                                                                                
      "module": "pulpcore.app",                                                                       
      "domain_compatible": true                                                                       
    },                                                                                                                                                                                                      
<snip>                                                                                                                                                                                                
    {                                                                                                                                                                                                       
      "component": "container",                                                                                                                                                                             
      "version": "2.19.2",                                                                                                                                                                                  
      "package": "pulp-container",                                                                                                                                                                          
      "module": "pulp_container.app",
      "domain_compatible": false
    },
<snip>
}

Describe the bug
We occasionally (with no real discernible pattern) see 404 Not Found during image pushes. Anecdotally, it seems like it happens most when we're pushing to the registry cache, but we've seen it during the image push steps too. For example:

#71 exporting to image
#71 exporting layers
#71 exporting layers 173.3s done
#71 exporting manifest sha256:a6f47bd5cb76be7346fd1cb99a90492ec79b3f4e00f0fb98195c808d3c6cb34b 0.0s done
#71 exporting config sha256:cb21c8ceb7ba7113890ab5ca388ceb5a1ea3c216acb6f073d9ab70464f923736
#71 exporting config sha256:cb21c8ceb7ba7113890ab5ca388ceb5a1ea3c216acb6f073d9ab70464f923736 0.0s done
#71 exporting attestation manifest sha256:993d77c985cfb8a5cc673f91824972d447ee48602329fc3912c6b01abf4fd8c8
#71 exporting attestation manifest sha256:993d77c985cfb8a5cc673f91824972d447ee48602329fc3912c6b01abf4fd8c8 0.1s done
#71 exporting manifest list sha256:e87e0eb4b59769dff6012b80bda2508bd6698f2539bbdc2b9c37ff5dc7b14cad 0.0s done
#71 pushing layers
#71 ...

#72 [auth] <image>:pull,push token for pulp.<domain>
#72 DONE 0.0s

#71 exporting to image
#71 pushing layers 109.3s done
#71 pushing manifest for pulp.<domain>/<image>:7cc3b17@sha256:e87e0eb4b59769dff6012b80bda2508bd6698f2539bbdc2b9c37ff5dc7b14cad
#71 pushing manifest for pulp.<domain>/<image>:7cc3b17@sha256:e87e0eb4b59769dff6012b80bda2508bd6698f2539bbdc2b9c37ff5dc7b14cad 3.6s done
#71 pushing layers 3.1s done
#71 pushing manifest for pulp.<domain>/<image>:<tag>@sha256:e87e0eb4b59769dff6012b80bda2508bd6698f2539bbdc2b9c37ff5dc7b14cad
#71 pushing manifest for pulp.<domain>/<image>:<tag>@sha256:e87e0eb4b59769dff6012b80bda2508bd6698f2539bbdc2b9c37ff5dc7b14cad 1.3s done
#71 DONE 291.0s

#73 exporting cache to registry
#73 preparing build cache for export
#73 writing layer sha256:01525355d735e7b6b8ebe60b82051c37843c2902ac1340e108ab7643ada6d470
#73 ...

#74 [auth] <image>-cache:pull,push token for pulp.<domain>
#74 DONE 0.0s

#73 exporting cache to registry
#73 writing layer sha256:01525355d735e7b6b8ebe60b82051c37843c2902ac1340e108ab7643ada6d470 0.7s done
#73 writing layer sha256:044175318ebb77afb048d2cdee8b43bf6c10b9a97f84a506a10fa797cd069075
#73 writing layer sha256:044175318ebb77afb048d2cdee8b43bf6c10b9a97f84a506a10fa797cd069075 0.3s done
#73 writing layer sha256:04965b09d9e9d14c554e36af33412acf01cfadc606d42bfbd8de496222eba0d0
#73 writing layer sha256:04965b09d9e9d14c554e36af33412acf01cfadc606d42bfbd8de496222eba0d0 0.3s done
#73 writing layer sha256:057c7aeeaed7ba721ef12ef58e0dceb05300a010b2b7d6e20f8ff4455779c671
#73 writing layer sha256:057c7aeeaed7ba721ef12ef58e0dceb05300a010b2b7d6e20f8ff4455779c671 0.3s done
#73 writing layer sha256:05a1af9fcb157da3afbe9d538ff487094a9f63356c54aca342faf727b38cf618
#73 writing layer sha256:05a1af9fcb157da3afbe9d538ff487094a9f63356c54aca342faf727b38cf618 0.2s done
#73 writing layer sha256:0752ce7f36560900e5a2c8b0b3316aed5e0db73b3c31306c0d287378fcb61598
#73 writing layer sha256:0752ce7f36560900e5a2c8b0b3316aed5e0db73b3c31306c0d287378fcb61598 0.5s done
#73 writing layer sha256:11459ca91b8320af3fa401ab8b5fb7127b40cf833157e0530b181e1cf932f12f
#73 writing layer sha256:11459ca91b8320af3fa401ab8b5fb7127b40cf833157e0530b181e1cf932f12f 0.4s done
#73 writing layer sha256:11fbbb80a4095f5516b8b36e9d054b209b36efd6e33a59221f270a43313737a8
#73 writing layer sha256:11fbbb80a4095f5516b8b36e9d054b209b36efd6e33a59221f270a43313737a8 0.4s done
#73 writing layer sha256:147ec5507b774ccc8ef4fe9754aeba607ece2bf50dbced9d7a31044afebbacf1
#73 writing layer sha256:147ec5507b774ccc8ef4fe9754aeba607ece2bf50dbced9d7a31044afebbacf1 0.5s done
#73 writing layer sha256:14e0bc89db8c8d47f5e4724d0042c1a51aaa6e05dc56643e0d6f8ffa8a08379d
#73 writing layer sha256:14e0bc89db8c8d47f5e4724d0042c1a51aaa6e05dc56643e0d6f8ffa8a08379d 0.4s done
#73 writing layer sha256:1603a056623feed9fb3fdbaa3c5c594b69e4cdafd8fb9d20ac8bd5dbd4f8877b
#73 writing layer sha256:1603a056623feed9fb3fdbaa3c5c594b69e4cdafd8fb9d20ac8bd5dbd4f8877b 0.4s done
#73 writing layer sha256:1da6f14f7b3514a93217a48bc23eb8500c228ed39fd0c5421881b8d37f1704c2
#73 writing layer sha256:1da6f14f7b3514a93217a48bc23eb8500c228ed39fd0c5421881b8d37f1704c2 0.2s done
#73 writing layer sha256:23a3e03ff261f8ca2dcede3c7bf0299b3d56118a80db60e9eb4cb92831c9eb4f
#73 writing layer sha256:23a3e03ff261f8ca2dcede3c7bf0299b3d56118a80db60e9eb4cb92831c9eb4f 0.2s done
#73 writing layer sha256:271b50b52a597a82945601c1f84b6e433786a6688b44b67e0043d6ff6a0d618f
#73 writing layer sha256:271b50b52a597a82945601c1f84b6e433786a6688b44b67e0043d6ff6a0d618f 0.4s done
#73 writing layer sha256:2933e4be0f6ef9a5f5097fc60cfcba8908b48c5a9ded6a40cc2187bd3b953e91
#73 writing layer sha256:2933e4be0f6ef9a5f5097fc60cfcba8908b48c5a9ded6a40cc2187bd3b953e91 0.2s done
#73 writing layer sha256:2dd22f0f3f70e32c0d9a0848ef2233f138b3ff24aabbd2b6d65c3223cc88c8eb
#73 writing layer sha256:2dd22f0f3f70e32c0d9a0848ef2233f138b3ff24aabbd2b6d65c3223cc88c8eb 0.4s done
#73 writing layer sha256:301d8d85a74898fcde2f88731a25806f2a10240872d97162db720122e131a4df
#73 writing layer sha256:301d8d85a74898fcde2f88731a25806f2a10240872d97162db720122e131a4df 0.3s done
#73 writing layer sha256:3444fa0b749d7d49d00d137cdbb3485d098230b015f9f348bd259c80a6a58558
#73 writing layer sha256:3444fa0b749d7d49d00d137cdbb3485d098230b015f9f348bd259c80a6a58558 0.6s done
#73 writing layer sha256:3a9309322bf58066ff345cffe4418ef5435cfb5af8943addb3e3e4db9b5c3e9c
#73 writing layer sha256:3a9309322bf58066ff345cffe4418ef5435cfb5af8943addb3e3e4db9b5c3e9c 0.2s done
#73 writing layer sha256:3a94c75b493b94ffdb28449eac31a245340aa8ec26e9bb7731ca7cdce81b09e9
#73 writing layer sha256:3a94c75b493b94ffdb28449eac31a245340aa8ec26e9bb7731ca7cdce81b09e9 0.7s done
#73 writing layer sha256:3be33b84fd6ca863088e22422fd8ddc8fe6278c6f73e66b5fb89470eaecf3886
#73 writing layer sha256:3be33b84fd6ca863088e22422fd8ddc8fe6278c6f73e66b5fb89470eaecf3886 0.3s done
#73 writing layer sha256:3e0031da2274159fa4e8747a9e8202a9ce8a598e1221c3a992632cf2a75504c8
#73 writing layer sha256:3e0031da2274159fa4e8747a9e8202a9ce8a598e1221c3a992632cf2a75504c8 0.3s done
#73 writing layer sha256:3ebab2c93c040ee710d7be09f2e74d201813e503ce9929abacca6e4449b42adb
#73 writing layer sha256:3ebab2c93c040ee710d7be09f2e74d201813e503ce9929abacca6e4449b42adb 0.7s done
#73 writing layer sha256:4292130be076c53f474142322ddf427000266e58feed6b261a533d7376270c95
#73 writing layer sha256:4292130be076c53f474142322ddf427000266e58feed6b261a533d7376270c95 0.6s done
#73 writing layer sha256:44d748b6c893e95079466940f482b0bac1273a34d2141a48a64dbe875bd0eada
#73 writing layer sha256:44d748b6c893e95079466940f482b0bac1273a34d2141a48a64dbe875bd0eada 0.2s done
#73 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
#73 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 done
#73 writing layer sha256:548866466dfa0128eab53d54197a5528928744090d857740dff1c592adacc1bf
#73 writing layer sha256:548866466dfa0128eab53d54197a5528928744090d857740dff1c592adacc1bf 0.4s done
#73 writing layer sha256:5d7cc58b1e126f8b3eed9d4aa0e0205253ea676cbb8ce85c1048bd3ea9aa8128
#73 writing layer sha256:5d7cc58b1e126f8b3eed9d4aa0e0205253ea676cbb8ce85c1048bd3ea9aa8128 0.2s done
#73 writing layer sha256:613f27c8fc3487c7e80da67b057f958ba4c4d983d0d1e2f2fe8b04440d3fb15c
#73 writing layer sha256:613f27c8fc3487c7e80da67b057f958ba4c4d983d0d1e2f2fe8b04440d3fb15c 0.5s done
#73 writing layer sha256:62b335c807980415166fd2e7e0e7b8da8b6fb948472b94924809dac88feffcd7
#73 writing layer sha256:62b335c807980415166fd2e7e0e7b8da8b6fb948472b94924809dac88feffcd7 0.3s done
#73 writing layer sha256:76442822e81453aaa3daf77d81fb0474fe8a9a25cb43b44f3f1273ca14f3cd37
#73 writing layer sha256:76442822e81453aaa3daf77d81fb0474fe8a9a25cb43b44f3f1273ca14f3cd37 0.5s done
#73 writing layer sha256:7702444192410b37cc6881737edbd6b480663b157b189ba12c5e776ae6162227
#73 writing layer sha256:7702444192410b37cc6881737edbd6b480663b157b189ba12c5e776ae6162227 0.3s done
#73 writing layer sha256:7bb9afb4836ed45074cf765c806c8d739472a847421b2e2b272568a371fe4186
#73 writing layer sha256:7bb9afb4836ed45074cf765c806c8d739472a847421b2e2b272568a371fe4186 0.3s done
#73 writing layer sha256:7c411cbb8a01bb9570bd4ab599fe64314c3d3ad9768e9bfcad376be33ded74f2
#73 writing layer sha256:7c411cbb8a01bb9570bd4ab599fe64314c3d3ad9768e9bfcad376be33ded74f2 0.2s done
#73 writing layer sha256:85bb4fbc01b0529a085d28603009e6d1adb6dd223ed60eb4c75296cbe057f8ce
#73 writing layer sha256:85bb4fbc01b0529a085d28603009e6d1adb6dd223ed60eb4c75296cbe057f8ce 0.3s done
#73 writing layer sha256:8ce96305f7851956f43a3a73659df6b03167904ad9060b227680a8c33662aa4d
#73 writing layer sha256:8ce96305f7851956f43a3a73659df6b03167904ad9060b227680a8c33662aa4d 1.0s done
#73 writing layer sha256:933973d4631be36d6b5e63c19c61a0874186f8538acae78714ad26663b5e04a2
#73 writing layer sha256:933973d4631be36d6b5e63c19c61a0874186f8538acae78714ad26663b5e04a2 1.1s done
#73 writing layer sha256:9644f61a7012c8deeea6c3819a4b1256ccd7da65e8a83944f721603ef5980b66
#73 writing layer sha256:9644f61a7012c8deeea6c3819a4b1256ccd7da65e8a83944f721603ef5980b66 0.3s done
#73 writing layer sha256:96ac0aad715b02af84e45dd18d975d84d49e284c0831bb8e5a30f7db1a5c8091
#73 writing layer sha256:96ac0aad715b02af84e45dd18d975d84d49e284c0831bb8e5a30f7db1a5c8091 0.2s done
#73 writing layer sha256:982cc79e315cf54a2311a02deaadce3d93c40d2558c8aa2d861a93ebdbc5027b
#73 writing layer sha256:982cc79e315cf54a2311a02deaadce3d93c40d2558c8aa2d861a93ebdbc5027b done
#73 writing layer sha256:9c704ecd0c694c4cbdd85e589ac8d1fc3fd8f890b7f3731769a5b169eb495809
#73 writing layer sha256:9c704ecd0c694c4cbdd85e589ac8d1fc3fd8f890b7f3731769a5b169eb495809 0.3s done
#73 writing layer sha256:a5a8cd1f01948da8b68deb51ecbc1d6b1293a802e568e12c07c7aaf2c308bf0d
#73 writing layer sha256:a5a8cd1f01948da8b68deb51ecbc1d6b1293a802e568e12c07c7aaf2c308bf0d 0.9s done
#73 writing layer sha256:a7b6159a0d575406996b940ccee1676c635c096775236ae2b5fad3d416240567
#73 writing layer sha256:a7b6159a0d575406996b940ccee1676c635c096775236ae2b5fad3d416240567 1.2s done
#73 writing layer sha256:a8cfe203da21316a0fe252c4e8d6eafdd8cc4f8f6a2b60245bf3c962690014ef
#73 writing layer sha256:a8cfe203da21316a0fe252c4e8d6eafdd8cc4f8f6a2b60245bf3c962690014ef 0.2s done
#73 writing layer sha256:a960b3505d8fbc3f11cee972d2f762a001419bcc4c14a059447635fc18ce50b9
#73 writing layer sha256:a960b3505d8fbc3f11cee972d2f762a001419bcc4c14a059447635fc18ce50b9 0.2s done
#73 writing layer sha256:abc050978704621da291ac46e34056d0a9597a8d92911233f87edad0aa2d4751
#73 writing layer sha256:abc050978704621da291ac46e34056d0a9597a8d92911233f87edad0aa2d4751 0.2s done
#73 writing layer sha256:adb5125413e7a34e398cedd9ca40af2c4616b8e2dcd887eb6249a89340c8bf7d
#73 writing layer sha256:adb5125413e7a34e398cedd9ca40af2c4616b8e2dcd887eb6249a89340c8bf7d 0.2s done
#73 writing layer sha256:ae1e4d7dcf67887a2820666fc4e9da9d9acfd44c76a40522460cc04300c0bcf6
#73 writing layer sha256:ae1e4d7dcf67887a2820666fc4e9da9d9acfd44c76a40522460cc04300c0bcf6 0.6s done
#73 writing layer sha256:b3fdb29e47594490c2dbd399a410e7807c1395c4d6e16b9737c932e75c671276
#73 writing layer sha256:b3fdb29e47594490c2dbd399a410e7807c1395c4d6e16b9737c932e75c671276 0.2s done
#73 writing layer sha256:b7b10d77efed6fb3af4dfb67e8e96a91501585d3f2a93ec56370babff115de5e
#73 writing layer sha256:b7b10d77efed6fb3af4dfb67e8e96a91501585d3f2a93ec56370babff115de5e 0.9s done
#73 writing layer sha256:b85dfe13830c2c171913a96589f75f4ed64f376b607cc8f8d7500565932ae9f3
#73 writing layer sha256:b85dfe13830c2c171913a96589f75f4ed64f376b607cc8f8d7500565932ae9f3 0.2s done
#73 writing layer sha256:b90b1aca9742cd809d645af217f50d692663fd9bd683f97aaf88f19829681483
#73 writing layer sha256:b90b1aca9742cd809d645af217f50d692663fd9bd683f97aaf88f19829681483 0.2s done
#73 writing layer sha256:bd540637d68a563ce8881363cbbfbbc9d371e36d0e75ba9d2210c7c12498e88c
#73 writing layer sha256:bd540637d68a563ce8881363cbbfbbc9d371e36d0e75ba9d2210c7c12498e88c 0.5s done
#73 writing layer sha256:be90e53f8898cd7adab7387819d1d7bd7a14acd85ac601ad0a5740576ae66282
#73 writing layer sha256:be90e53f8898cd7adab7387819d1d7bd7a14acd85ac601ad0a5740576ae66282 0.4s done
#73 writing layer sha256:c4a79b465ab1a94daec52b0e89cdd532218cdd52cd090eb455931aa550692fda
#73 writing layer sha256:c4a79b465ab1a94daec52b0e89cdd532218cdd52cd090eb455931aa550692fda 0.2s done
#73 writing layer sha256:c5e244e3ce5ceaf7f9e3c8081b6a6dfd0e6d04b6f836ca951ad1e2a51ae87102
#73 writing layer sha256:c5e244e3ce5ceaf7f9e3c8081b6a6dfd0e6d04b6f836ca951ad1e2a51ae87102 0.6s done
#73 writing layer sha256:c650db8f793d468e4575079673507a3e463a69f81a28c53b5e387c45a1041ef3
#73 writing layer sha256:c650db8f793d468e4575079673507a3e463a69f81a28c53b5e387c45a1041ef3 0.2s done
#73 writing layer sha256:c9ec06ca22e319f945d75dfcab94f28f442ea44fa091e897d563b44e432247e3
#73 writing layer sha256:c9ec06ca22e319f945d75dfcab94f28f442ea44fa091e897d563b44e432247e3 0.4s done
#73 writing layer sha256:cb8b2e5edc73fddd08a0d8f5554042a1cdc89310954e90b3369e7db4737bd953
#73 writing layer sha256:cb8b2e5edc73fddd08a0d8f5554042a1cdc89310954e90b3369e7db4737bd953 0.3s done
#73 writing layer sha256:ccf1d123eaa327de4eaf0ce55057d50445c022f5e39a874fef2132aab412bd1a
#73 writing layer sha256:ccf1d123eaa327de4eaf0ce55057d50445c022f5e39a874fef2132aab412bd1a 0.2s done
#73 writing layer sha256:d30bc156f85726423638929d88c58acef2f053605b687568330d3b6ffa4163c7
#73 writing layer sha256:d30bc156f85726423638929d88c58acef2f053605b687568330d3b6ffa4163c7 0.4s done
#73 writing layer sha256:d4e486087e5be1f082358a41e95a1dd4470dd71e20dd21f138e678f28a5fc2d0
#73 writing layer sha256:d4e486087e5be1f082358a41e95a1dd4470dd71e20dd21f138e678f28a5fc2d0 0.2s done
#73 writing layer sha256:dc3c5333306d79d5fe6269fcd3920c41e938be3cc78ac58b44f945c91f937b4b
#73 writing layer sha256:dc3c5333306d79d5fe6269fcd3920c41e938be3cc78ac58b44f945c91f937b4b 0.4s done
#73 writing layer sha256:ddde008fd93da8a6af166e2de8b6b5cbed2fa3246676b5a47495f9dd59016424
#73 writing layer sha256:ddde008fd93da8a6af166e2de8b6b5cbed2fa3246676b5a47495f9dd59016424 0.5s done
#73 writing layer sha256:df23c83256a294417fd186a825fd9513ccec1cbe9eba5170fd2b3633a0cf9a93
#73 writing layer sha256:df23c83256a294417fd186a825fd9513ccec1cbe9eba5170fd2b3633a0cf9a93 0.5s done
#73 writing layer sha256:e0f7b3100b1d6c5b44dbf0e195ac1e1e58eb0e121fffded34139b0ec66a9148f
#73 writing layer sha256:e0f7b3100b1d6c5b44dbf0e195ac1e1e58eb0e121fffded34139b0ec66a9148f 0.4s done
#73 writing layer sha256:e384321238c7a263cdf16ee8f23c4a69264bcbae1ff1dd1d8811c4c6b1501431
#73 writing layer sha256:e384321238c7a263cdf16ee8f23c4a69264bcbae1ff1dd1d8811c4c6b1501431 0.4s done
#73 writing layer sha256:e69a5db34ec4dc6fdab8fad9e70e6b2de69c719ff39210fa163afa4e548d1d37
#73 writing layer sha256:e69a5db34ec4dc6fdab8fad9e70e6b2de69c719ff39210fa163afa4e548d1d37 0.4s done
#73 writing layer sha256:ef31a6814a468b58bb3011b59c0d529ad75954b94504b3c91ae1e50afe876552
#73 writing layer sha256:ef31a6814a468b58bb3011b59c0d529ad75954b94504b3c91ae1e50afe876552 0.6s done
#73 writing layer sha256:efa2402de0932c3b65dab1eeab3c665fee0d8b0aa36c11d80ee53560ddedd0a3
#73 writing layer sha256:efa2402de0932c3b65dab1eeab3c665fee0d8b0aa36c11d80ee53560ddedd0a3 0.5s done
#73 writing layer sha256:f092c6d375ed0eab80ffc491d24e8cfb4ce1304355d003b081b26257c4b35b59
#73 writing layer sha256:f092c6d375ed0eab80ffc491d24e8cfb4ce1304355d003b081b26257c4b35b59 0.5s done
#73 writing layer sha256:f28a00134dd3044b063336da7a98baedfbbc3bfcb5fc0636023f72fe61e3c752
#73 preparing build cache for export 290.8s done
#73 writing layer sha256:f28a00134dd3044b063336da7a98baedfbbc3bfcb5fc0636023f72fe61e3c752 204.5s done
#73 ERROR: error writing layer blob: failed commit on ref "layer-sha256:f28a00134dd3044b063336da7a98baedfbbc3bfcb5fc0636023f72fe61e3c752": unexpected status from PUT request to https://pulp.<domain>/v2/<image>-cache/blobs/uploads/01912e58-af0e-7f79-b5ca-56eb659b5f6e?digest=sha256%3Af28a00134dd3044b063336da7a98baedfbbc3bfcb5fc0636023f72fe61e3c752: 404 Not Found
------
 > exporting cache to registry:
------
ERROR: failed to solve: error writing layer blob: failed commit on ref "layer-sha256:f28a00134dd3044b063336da7a98baedfbbc3bfcb5fc0636023f72fe61e3c752": unexpected status from PUT request to https://pulp.<domain>/v2/<image>-cache/blobs/uploads/01912e58-af0e-7f79-b5ca-56eb659b5f6e?digest=sha256%3Af28a00134dd3044b063336da7a98baedfbbc3bfcb5fc0636023f72fe61e3c752: 404 Not Found
Error: buildx failed with: ERROR: failed to solve: error writing layer blob: failed commit on ref "layer-sha256:f28a00134dd3044b063336da7a98baedfbbc3bfcb5fc0636023f72fe61e3c752": unexpected status from PUT request to https://pulp.<domain>/v2/<image>-cache/blobs/uploads/01912e58-af0e-7f79-b5ca-56eb659b5f6e?digest=sha256%3Af28a00134dd3044b063336da7a98baedfbbc3bfcb5fc0636023f72fe61e3c752: 404 Not Found

To Reproduce
Unclear, since it doesn't seem to happen all the time or in any identifiable situations.

Expected behavior
The push should succeed.

Additional context
At first I thought it could be related to #1587 but I verified that image-manifest=true was on the cache-to: line. It almost feels like the token expires and so the push fails, but I haven't found anything to indicate that in logs, and we have token expiration set to 3600 (which is far longer than the workflow takes to run).

[grzleadams]

I should mention that we're seeing this when using docker/build-push-action, and I'm not entirely convinced this is a Pulp issue, but wanted to open this in case it is.

[grzleadams]

In a previous job I remember seeing something like this in Artifactory, and it was related to the client sending a PUT vs. a PATCH. Not sure if this could be something similar.

For reference, here's our client information after setting buildx up with docker/setup-buildx-action:

  Client:
   Version:    25.0.4
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Docker Buildx (Docker Inc.)
      Version:  v0.13.1
      Path:     /usr/local/lib/docker/cli-plugins/docker-buildx

[ipanova]

I am not familiar that much with buildx, can you share if available all the calls it makes to pulp?
Usually when a layer is being uploaded, first an upload-id is being created with POST, that's the https://pulp.<domain>/v2/<image>-cache/blobs/uploads/01912e58-af0e-7f79-b5ca-56eb659b5f6e
Then chunks are being uploaded to it via PATCH.
PUT is a sign that blob upload completes with the upload of last chunk. After that a DELETE is issued to the upload-id.

This log info suggests that:
#73 writing layer sha256:f28a00134dd3044b063336da7a98baedfbbc3bfcb5fc0636023f72fe61e3c752 ----> PATCH happens
#73 preparing build cache for export 290.8s done
#73 writing layer sha256:f28a00134dd3044b063336da7a98baedfbbc3bfcb5fc0636023f72fe61e3c752 204.5s done --->PUT happens, layer upload is complete ( then upload-id DELETE should happen)
#73 ERROR: error writing layer blob: failed commit on ref "layer-sha256:f28a00134dd3044b063336da7a98baedfbbc3bfcb5fc0636023f72fe61e3c752": unexpected status from PUT request to https://pulp./v2/-cache/blobs/uploads/01912e58-af0e-7f79-b5ca-56eb659b5f6e?digest=sha256%3Af28a00134dd3044b063336da7a98baedfbbc3bfcb5fc0636023f72fe61e3c752: 404 Not Found ----> again PUT?

[grzleadams]

I assume that's the PUT for the image manifest (since this is caching to the registry) but I'll see if there's a way to enable buildx debugging in the workflow to get all those calls. Later runs of the same workflow succeeded with the push of this cache image, for what it's worth.

[ipanova]

No no, the error is clear that call to the blob upload endpoint failed and not manifest endpoint.
Yeah please get us the api call logs if possible.

[ipanova]

Later runs of the same workflow succeeded with the push of this cache image, for what it's worth.

That's a sign of some race condition. But for what it's worth I see such bug report for the first time, have not observed such via podman or docker, so there is a chance there is something fishy with buildx?

[grzleadams]

Later runs of the same workflow succeeded with the push of this cache image, for what it's worth.

That's a sign of some race condition. But for what it's worth I see such bug report for the first time, have not observed such via podman or docker, so there is a chance there is something fishy with buildx?

I agree, it feels like some kind of race condition. buildx does implement some parallelization of builds but as far as I know/can tell the push of the layers is single-threaded, so I'm not exactly sure where such a race condition would come in. Either way, I'll see about exporting the buildx logs from the workflow so we can try to get more information.