Added suport for monolithic upload. #1274

ipanova · 2023-05-03T10:33:56Z

ipanova · 2023-05-03T13:21:43Z

pulp_container/app/registry_api.py

        """
        _, repository = self.get_dr_push(request, path, create=True)

        if self.tries_to_mount_blob(request):
            response = self.mount_blob(request, path, repository)
+            return response
+        elif digest := request.query_params.get("digest"):


this is a theoretical part, based only on docs specs https://docs.docker.com/registry/spec/api/
I am not aware of any client that would be using this

I am quite puzzled by the specs. We can complete a monolithic upload in either a PUT or POST request. This is what you meant by the theoretical part, correct?

https://stackoverflow.com/a/57068286

you not the only one ;)

monolithic upload can be done 2 ways:

Make a POST request with digest as request param and complete in one request the blob upload ( this is the theoretical part because i have no idea what clients use this, so basically this is implemented only based on docs without a way to prove this works fine). Also note that I am returning a 201 BlobResponse and not UploadResponse

Make a POST request without digest request param, then make a PUT request and send the chunk in the body.

ipanova · 2023-05-03T13:22:52Z

pulp_container/app/registry_api.py

@@ -684,11 +747,15 @@ def put(self, request, path, pk=None):
        """
        Create a blob from uploaded chunks.

-        Note: We do not support monolithic upload.
+        This request makes the upload complete. It can whether carry a zero-length


here, this can be tested with podman( which is sending empty body in the PUT request) and helm client that sends last ( in its case the only chunk) in the PUT request.

And, the last chunk can also be the "main" chunk in the monolithic upload, right?

To carry out a “monolithic” upload, one can simply put the entire content blob to the provided URL: PUT /v2//blobs/uploads/?digest=<digest>

https://docs.docker.com/registry/spec/api/#monolithic-upload

In the monolithic upload it is always the last and main chunk, that's why it is called monolithic.
It never issues a PATCH request where chunks are being sent ( as in chunked upload).

To make it even more confusing, podman claims to use chunked upload, but in reality it sens only one chunk in the PATCH request and no body in the PUT. It also does not send the content-range header in the PATCH request that would indicate a legal chunked upload ;) So that if branch in the PATCH code with the content-range header is also theoretical ;)

lubosmj

I added a few comments. It might be beneficial to merge this PR after merging pending blobs first, as discussed before.

lubosmj · 2023-05-03T21:04:43Z

pulp_container/app/registry_api.py


-        return response
+    def create_single_chunk_artifact(self, chunk):


I miss basic error handling in this method (e.g., digest invalid, omitted content-type, etc.).

Digest validation is done in init_and_validate.

Oh, I interchanged the method I wanted to reference. I meant to place this comment in single_request_upload. I wanted rather to validate the format of the digest.

There are few places that would benefit from digest validation format, at least in the PUT request.
I would not bother with that, it would just give a 404. Or at least not in the scope of this PR.

lubosmj · 2023-05-03T21:11:48Z

pulp_container/app/registry_api.py

@@ -684,11 +747,15 @@ def put(self, request, path, pk=None):
        """
        Create a blob from uploaded chunks.

-        Note: We do not support monolithic upload.
+        This request makes the upload complete. It can whether carry a zero-length


And, the last chunk can also be the "main" chunk in the monolithic upload, right?

To carry out a “monolithic” upload, one can simply put the entire content blob to the provided URL: PUT /v2//blobs/uploads/?digest=<digest>

https://docs.docker.com/registry/spec/api/#monolithic-upload

lubosmj · 2023-05-03T21:18:26Z

pulp_container/app/registry_api.py

        """
        _, repository = self.get_dr_push(request, path, create=True)

        if self.tries_to_mount_blob(request):
            response = self.mount_blob(request, path, repository)
+            return response
+        elif digest := request.query_params.get("digest"):


I am quite puzzled by the specs. We can complete a monolithic upload in either a PUT or POST request. This is what you meant by the theoretical part, correct?

https://stackoverflow.com/a/57068286

lubosmj · 2023-05-03T21:47:14Z

I think we need to handle overloading too. This is however not specified in the documentation. May a helm client decide to flood and freeze our API with a huge monolithic upload once we will support immediate tasks?

mdellweg · 2023-05-04T06:36:28Z

pulp_container/app/registry_api.py

+            try:
+                blob_artifact = ContentArtifact(
+                    artifact=artifact, content=blob, relative_path=digest
+                )
+                blob_artifact.save()


I think, this should be part of the first try. The blob cannot exist without it's content artifact. So finding the blob in 608 means the ContentArtifact must already be in place.

You're right, I will change this.
I mostly moved around the code that was repeated in few places without looking too thoroughly.

Actually, I will not. Here is the reason to it! c06c025
Should I rather leave a comment in the code?

mdellweg · 2023-05-04T06:37:22Z

pulp_container/app/registry_api.py

+            except IntegrityError:
+                ca = ContentArtifact.objects.get(content=blob, relative_path=digest)
+                if not ca.artifact:
+                    ca.artifact = artifact
+                    ca.save(update_fields=["artifact"])


Which means we would need none of this.

Am i missing not yet downloaded remote artifacts here?

there are no remote artifacts in the uploaded content

ipanova · 2023-05-04T13:25:31Z

I think we need to handle overloading too. This is however not specified in the documentation. May a helm client decide to flood and freeze our API with a huge monolithic upload once we will support immediate tasks?

There is an issue that can address your concern with the limitation of the max body #532

closes pulp#1219

ipanova commented May 3, 2023

View reviewed changes

lubosmj reviewed May 3, 2023

View reviewed changes

mdellweg reviewed May 4, 2023

View reviewed changes

ipanova force-pushed the i1219 branch 2 times, most recently from 35aa23a to 4d5b05d Compare May 26, 2023 13:10

Added suport for monolithic upload.

3616957

closes pulp#1219

ipanova force-pushed the i1219 branch from 4d5b05d to 3616957 Compare May 26, 2023 13:15

lubosmj approved these changes May 26, 2023

View reviewed changes

lubosmj merged commit cd37507 into pulp:main May 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added suport for monolithic upload. #1274

Added suport for monolithic upload. #1274

ipanova commented May 3, 2023

ipanova May 3, 2023

lubosmj May 3, 2023

ipanova May 4, 2023

ipanova May 3, 2023

lubosmj May 3, 2023

ipanova May 4, 2023 •

edited

Loading

lubosmj left a comment •

edited

Loading

lubosmj May 3, 2023

mdellweg May 4, 2023

lubosmj May 4, 2023

ipanova May 4, 2023

lubosmj May 3, 2023

lubosmj May 3, 2023

lubosmj commented May 3, 2023

mdellweg May 4, 2023

ipanova May 4, 2023 •

edited

Loading

ipanova May 4, 2023

mdellweg May 4, 2023

mdellweg May 4, 2023

ipanova May 4, 2023

ipanova commented May 4, 2023


		return response
		def create_single_chunk_artifact(self, chunk):

Added suport for monolithic upload. #1274

Added suport for monolithic upload. #1274

Conversation

ipanova commented May 3, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ipanova May 4, 2023 • edited Loading

Choose a reason for hiding this comment

lubosmj left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lubosmj commented May 3, 2023

Choose a reason for hiding this comment

ipanova May 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ipanova commented May 4, 2023

ipanova May 4, 2023 •

edited

Loading

lubosmj left a comment •

edited

Loading

ipanova May 4, 2023 •

edited

Loading