Cloud API: Support selecting multiple upload targets and add Pulp OSTree uploads

[achilleas-k]

This PR is a follow-up to #3636 and replaces #3672. It requires #3636 but isn't based on it (it wont compile until it is).

---

The purpose of this PR is to add the ability to upload edge and iot commits built through the cloud API to a Pulp server that supports ostree repos. However, at the same time, we need to be able to support the old behaviour of uploading ostree commits to AWS S3. This is a required part of HMS-1798.

I also think it's a good idea in general to be able to specify alternative upload targets for certain artifacts. We could support uploading any artifact to S3 for example.

The main change in this PR is that an image request now includes an (optional) upload_target option which is an enum of the names of all the upload targets we support in the cloud API (in the form org.osbuild.<target>, e.g., org.osbuild.aws.s3). Omitting the upload_target maintains the current behaviour, which means each image type has a default target. Also, we define a mapping between upload targets and image types that they support and any invalid combination is a bad request.

The commit that adds the pulp ostree upload target to the cloud API was cherry picked from #3672 (thanks @kingsleyzissou).

---

This pull request includes:

• adequate testing for the new functionality or fixed issue
• adequate documentation informing people about the change such as
  • submit a PR for the guides repository if this PR changed any behavior described there: https://www.osbuild.org/guides/

[achilleas-k]

#3636 has been merged. Rebased and marked as ready.

[achilleas-k]

Converted to draft to block this while I work on supporting multiple targets.

[achilleas-k]

Sorry this took so long. Multiple upload targets added.

This now supports what @croissanne suggested and also adds the default upload options with the default target for the image type if specified, so all the following examples are valid:

Single explicit upload target:

{
  "architecture": "x86_64",
  "image_type": "edge-commit",
  "upload_targets": [
    {
      "type": "org.osbuild.aws.s3",
      "upload_options": { ... }
    }
  ]
}

{
  "architecture": "x86_64",
  "image_type": "edge-commit",
  "upload_targets": [
    {
      "type": "org.osbuild.pulp.ostree",
      "upload_options": { ... }
    }
  ]
}

{
  "architecture": "x86_64",
  "image_type": "guest-image",
  "upload_targets": [
    {
      "type": "org.osbuild.aws.s3",
      "upload_options": { ... }
    }
  ]
}

Default only (will select the default for the image type):

{
  "architecture": "x86_64",
  "image_type": "edge-commit",
  "upload_options": { ... },
  "upload_targets": []
}

{
  "architecture": "x86_64",
  "image_type": "guest-image",
  "upload_options": { ... },
  "upload_targets": []
}

One explicit upload target + a default:

{
  "architecture": "x86_64",
  "image_type": "edge-commit",
  "upload_options": { ... },
  "upload_targets": [
    {
      "type": "org.osbuild.pulp.ostree",
      "upload_options": { ... }
    }
  ]
}

{
  "architecture": "x86_64",
  "image_type": "guest-image",
  "upload_options": { ... },
  "upload_targets": [
    {
      "type": "org.osbuild.aws.s3",
      "upload_options": { ... }
    }
  ]
}

Multiples of the same upload target are supported.

[achilleas-k]

We should add some tests for this. I'm thinking we can change some of the compose requests in the API tests to use the new UploadTargets and for types that get uploaded to S3, we can define multiple upload targets and check that all of them succeeded.

[achilleas-k]

Edge test failures are failures to fetch from the rpm repos for the osbuild-composer build. A glitch, I guess?

[achilleas-k]

The response for the compose status is

    ComposeStatus:
      allOf:
      - $ref: '#/components/schemas/ObjectReference'
      - type: object
        required:
          - status
          - image_status
        properties:
          status:
            $ref: '#/components/schemas/ComposeStatusValue'
          image_status:
            $ref: '#/components/schemas/ImageStatus'
          image_statuses:
            type: array
            items:
              $ref: '#/components/schemas/ImageStatus'
          koji_status:
            $ref: '#/components/schemas/KojiStatus'

which means we can use the image_statuses array to send responses for the multiple uploads:

    ImageStatus:
      required:
       - status
      properties:
        status:
          $ref: '#/components/schemas/ImageStatusValue'
        upload_status:
          $ref: '#/components/schemas/UploadStatus'
        error:
          $ref: '#/components/schemas/ComposeStatusError'

but what should we do with the image_status at the top level? Should it mirror the request? Meaning, if the request defines upload_options in the old top level part of the request, the result of that specific upload request goes into the image_status. Any upload defined in upload_targets goes in the image_statuses. Do we need to keep the order the same so they match the request? Am I overcomplicating this?

EDIT: Spoke with @croissanne and we decided instead to add upload_statuses to image_status.

[thozza]

Nice work, the PR looks good! I have just one small nitpick...

[achilleas-k]

Added support for multiple upload targets and the response now includes an array of upload_status objects under upload_statuses. The top-level upload_status is always a copy of the first element of upload_statuses, so in the most common case where we only have one upload target, the upload_statuses array has only one element that is the same as the existing upload_status.

The structure was discussed and agreed with @croissanne.

[croissanne]

Rebased to prevent OCI private key leaking. Apologies :(

[achilleas-k]

I think the RHEL 8.8 RPM builds are getting hungry and killed again:

Nov 08 17:40:46 ip-172-31-22-179.ec2.internal kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/machine.slice/machine-cb421a227be348e29fe19c158933ca72.scope,task=compile,pid=29577,uid=1000
Nov 08 17:40:46 ip-172-31-22-179.ec2.internal kernel: Out of memory: Killed process 29577 (compile) total-vm:2798720kB, anon-rss:1645056kB, file-rss:0kB, shmem-rss:1088kB, UID:1000 pgtables:640kB oom_score_adj:0

From RHEL 8.8 aarch64 rpm build journal log: https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/jobs/5490961946

This time it's happening on all 8.8 builds, regardless of architecture. The last time we had this issue it was on all EL8 builds for aarch64. Weird.

[croissanne]

thank you! this was a big effort ._.