Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add repository_version param as a building context #1687

Merged
merged 1 commit into from
Oct 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGES/479.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
The `build_image` endpoint has been refactored to accept `build_context`
(i.e., a file repository version) instead of raw artifacts. The same applies to Containerfile."
57 changes: 41 additions & 16 deletions docs/admin/guides/build-image.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,47 +6,72 @@
guaranteed.

Users can add new images to a container repository by uploading a Containerfile. The syntax for
Containerfile is the same as for a Dockerfile. The same REST API endpoint also accepts a JSON
string that maps artifacts in Pulp to a filename. Any artifacts passed in are available inside the
build container at `/pulp_working_directory`.
Containerfile is the same as for a Dockerfile.

## Create a Repository
To pass arbitrary files or artifacts to the image building context, the `build_context` property (a reference to a file repository) can be provided in the request payload.
These files can be referenced in Containerfile by passing their `relative-path`:
```
ADD/COPY <file relative-path> <location in container>
```

It is possible to define the Containerfile in two ways:
* from a [local file](site:pulp_container/docs/admin/guides/build-image#build-from-a-containerfile-uploaded-during-build-request) and pass it during build request
* from an [existing file](site:pulp_container/docs/admin/guides/build-image#upload-the-containerfile-as-a-file-content) in the `build_context`

## Create a Container Repository

```bash
REPO_HREF=$(pulp container repository create --name building | jq -r '.pulp_href')
CONTAINER_REPO=$(pulp container repository create --name building | jq -r '.pulp_href')
```

## Create an Artifact
## Create a File Repository and populate it

```bash
FILE_REPO=$(pulp file repository create --name bar --autopublish | jq -r '.pulp_href')

echo 'Hello world!' > example.txt

ARTIFACT_HREF=$(http --form POST http://localhost/pulp/api/v3/artifacts/ \
file@./example.txt \
| jq -r '.pulp_href')
pulp file content upload --relative-path foo/bar/example.txt \
--file ./example.txt --repository bar
```

## Create a Containerfile

```bash
echo 'FROM centos:7

# Copy a file using COPY statement. Use the relative path specified in the 'artifacts' parameter.
# Copy a file using COPY statement. Use the path specified in the '--relative-path' parameter.
COPY foo/bar/example.txt /inside-image.txt

# Print the content of the file when the container starts
CMD ["cat", "/inside-image.txt"]' >> Containerfile
```

## Build an OCI image

## Build from a Containerfile uploaded during build request

```bash
TASK_HREF=$(http --form POST ${BASE_ADDR}${CONTAINER_REPO}'build_image/' "containerfile@./Containerfile" \
build_context=${FILE_REPO}versions/1/ | jq -r '.task')
```

## Upload the Containerfile to a File Repository and use it to build

### Upload the Containerfile as a File Content

```bash
pulp file content upload --relative-path MyContainerfile --file ./Containerfile --repository bar
```

### Build an OCI image from a Containerfile present in build_context

```bash
TASK_HREF=$(http --form POST :$REPO_HREF'build_image/' containerfile@./Containerfile \
artifacts="{\"$ARTIFACT_HREF\": \"foo/bar/example.txt\"}" | jq -r '.task')
TASK_HREF=$(http --form POST ${BASE_ADDR}${CONTAINER_REPO}'build_image/' containerfile_name=MyContainerfile \
build_context=${FILE_REPO}versions/2/ | jq -r '.task')
```


!!! warning

Non-staff users, lacking read access to the `artifacts` endpoint, may encounter restricted
functionality as they are prohibited from listing artifacts uploaded to Pulp and utilizing
them within the build process.
File repositories synced with the on-demand policy will not automatically download the missing artifacts.
Trying to build an image using a file that has not yet been downloaded will fail.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we intend to lift that limitation some day?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be a good improvement.
We could “force” (or automate) the download by Pulp if the artifacts were not found, or maybe do not allow the usage of on-demand repositories for the build machinery.

107 changes: 59 additions & 48 deletions pulp_container/app/serializers.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
from gettext import gettext as _
import os
import re

from django.core.validators import URLValidator
from rest_framework import serializers

from pulpcore.plugin.models import (
Artifact,
ContentRedirectContentGuard,
Remote,
Repository,
Expand All @@ -30,9 +28,11 @@
ValidateFieldsMixin,
)

from pulp_file.app.models import FileContent
from pulp_container.app import models
from pulp_container.constants import SIGNATURE_TYPE


VALID_SIGNATURE_NAME_REGEX = r"^sha256:[0-9a-f]{64}@[0-9a-f]{32}$"
VALID_TAG_REGEX = r"^[A-Za-z0-9][A-Za-z0-9._-]*$"
VALID_BASE_PATH_REGEX_COMPILED = re.compile(r"^[a-z0-9]+(?:(?:[._]|__|[-]*)[a-z0-9])*$")
Expand Down Expand Up @@ -758,13 +758,12 @@ class OCIBuildImageSerializer(ValidateFieldsMixin, serializers.Serializer):
A repository must be specified, to which the container image content will be added.
"""

containerfile_artifact = RelatedField(
many=False,
lookup_field="pk",
view_name="artifacts-detail",
queryset=Artifact.objects.all(),
containerfile_name = serializers.CharField(
required=False,
allow_blank=True,
help_text=_(
"Artifact representing the Containerfile that should be used to run podman-build."
"Name of the Containerfile, from build_context, that should be used to run "
"podman-build."
),
)
containerfile = serializers.FileField(
Expand All @@ -774,65 +773,77 @@ class OCIBuildImageSerializer(ValidateFieldsMixin, serializers.Serializer):
tag = serializers.CharField(
git-hyagi marked this conversation as resolved.
Show resolved Hide resolved
required=False, default="latest", help_text="A tag name for the new image being built."
)
artifacts = serializers.JSONField(
build_context = RepositoryVersionRelatedField(
required=False,
help_text="A JSON string where each key is an artifact href and the value is it's "
"relative path (name) inside the /pulp_working_directory of the build container "
"executing the Containerfile.",
help_text=_("RepositoryVersion to be used as the build context for container images."),
allow_null=True,
lubosmj marked this conversation as resolved.
Show resolved Hide resolved
queryset=RepositoryVersion.objects.filter(repository__pulp_type="file.file"),
)

def __init__(self, *args, **kwargs):
"""Initializer for OCIBuildImageSerializer."""
super().__init__(*args, **kwargs)
self.fields["containerfile_artifact"].required = False

def validate(self, data):
"""Validates that all the fields make sense."""
data = super().validate(data)

if "containerfile" in data:
if "containerfile_artifact" in data:
raise serializers.ValidationError(
_("Only one of 'containerfile' and 'containerfile_artifact' may be specified.")
)
data["containerfile_artifact"] = Artifact.init_and_validate(data.pop("containerfile"))
elif "containerfile_artifact" in data:
data["containerfile_artifact"].touch()
else:
if bool(data.get("containerfile", None)) == bool(data.get("containerfile_name", None)):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Poor mans xor... 🤣

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤣

raise serializers.ValidationError(
_("Exactly one of 'containerfile' or 'containerfile_name' must be specified.")
)

if "containerfile_name" in data and "build_context" not in data:
raise serializers.ValidationError(
_("'containerfile' or 'containerfile_artifact' must " "be specified.")
_("A 'build_context' must be specified when 'containerfile_name' is present.")
)
artifacts = {}
if "artifacts" in data:
for url, relative_path in data["artifacts"].items():
if os.path.isabs(relative_path):

# TODO: this can be removed after https://github.com/pulp/pulpcore/issues/5786
if data.get("build_context", None):
data["repository_version"] = data["build_context"]
git-hyagi marked this conversation as resolved.
Show resolved Hide resolved

return data

def deferred_files_validation(self, data):
"""
Defer the validation of on_demand_artifacts and the `Containerfile` to avoid rerunning
unnecessary database queries when checking permissions (DRF Access Policy).
"""
if build_context := data.get("build_context", None):

# check if the on_demand_artifacts exist
for on_demand_artifact in build_context.on_demand_artifacts.iterator():
if not on_demand_artifact.content_artifact.artifact:
raise serializers.ValidationError(
_("Relative path cannot start with '/'. " "{0}").format(relative_path)
_(
"It is not possible to use File content synced with on-demand "
"policy without pulling the data first."
)
)

# check if the containerfile_name exists in the build_context (File Repository)
if (
data.get("containerfile_name", None)
and not FileContent.objects.filter(
repositories__in=[build_context.repository.pk],
relative_path=data["containerfile_name"],
).exists()
):
raise serializers.ValidationError(
_(
'Could not find the Containerfile "'
+ data["containerfile_name"]
+ '" in the build_context provided'
)
artifactfield = RelatedField(
view_name="artifacts-detail",
queryset=Artifact.objects.all(),
source="*",
initial=url,
)
try:
artifact = artifactfield.run_validation(data=url)
artifact.touch()
artifacts[str(artifact.pk)] = relative_path
except serializers.ValidationError as e:
# Append the URL of missing Artifact to the error message
e.detail[0] = "%s %s" % (e.detail[0], url)
raise e
data["artifacts"] = artifacts

data["build_context_pk"] = build_context.repository.pk

return data

class Meta:
fields = (
"containerfile_artifact",
"containerfile_name",
"containerfile",
"repository",
"tag",
"artifacts",
"build_context",
)


Expand Down
2 changes: 1 addition & 1 deletion pulp_container/app/tasks/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
from .download_image_data import download_image_data # noqa
from .builder import build_image_from_containerfile # noqa
from .builder import build_image_from_containerfile, build_image # noqa
from .recursive_add import recursive_add_content # noqa
from .recursive_remove import recursive_remove_content # noqa
from .sign import sign # noqa
Expand Down
Loading