🏭 Consolidate osbuilder and Auroraboot #1633

jimmykarily · 2023-07-24T08:26:08Z

We currently have 2 different tools to build Kairos images:

Auraboot, meant to be used outside Kubernetes
osbuilder, meant to be used as a controller inside Kubernetes

Both of them have more responsibilities than building (e.g. Auroraboot handles netbooting, osbuilder handles upgrades) but they share the building part.

The current situation and the desired is described in the image above.

This is an epic to collect all the steps/issues toward the desired state.

The last item needs to be discussed and it will be more clear when we have everything consolidated in one image.

jimmykarily · 2023-08-01T11:57:37Z

One more image describing the desired state of the various tools (osbuilder, auroraboot, enki) and the relations between them.

jimmykarily · 2023-11-27T07:25:13Z

We will soon be able to build Kairos container images using just Dockerfiles (no Earthly or other tools). There are 2 things missing to have the full "factory" experience:

A tool that produces the various other artifacts from the container image (e.g. iso, img)
A tool that wraps the whole flow in some nice UX

Right now the creation of additional artifacts happens in Earthly using osbuilder-tools image. Above we describe how Autoraboot should provide both the UX and the artifact building functionality. Let's consider splitting as such:

enki is the tool that builds all types of artifacts given a container image
Auroraboot is a cli interface that calls out to enki. It's meant to be used by humans.\
Osbuilder controller also calls out to enki. It's meant to be used in Kubernetes in a declarative way.

The new suggestion above is that osbuilder doesn't use Auroraboot but enki directly. This allows us to focus the UX of Auroraboot to humans and not try to satisfy the needs of osbuilder too. enki should be able to cover both since it will be an internal tool, with no focus on UX.

jimmykarily · 2024-10-16T14:50:53Z

Work happening here:

As part of kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]>

so that it can be used as a library from Auroraboot as part of kairos-io/kairos#1633 The following anti-patterns make it very hard to re-use the enki functions: - viper is used as a global store, randomly accessing everything from anywhere - we pass around too many "configs": the BuildConfig, the embedded kairos-agent config, the BuildUKIAction (which is essentially yet another config), It's nearly impossible to construct the proper input for any function because we don't know what part of the configs available, is actually needed and used. This is a first refactoring that should at least let the caller use the BuildUKIAction without needing to populate a global "viper" object. Signed-off-by: Dimitris Karakasilis <[email protected]>

As part of kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]>

as part of kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]>

part of: kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]>

* Port sysext command from enki part of: kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]> * Don't use viper Signed-off-by: Dimitris Karakasilis <[email protected]> * Use global "--debug" option and some minor fixes - args that weren't used - command output was not printed Signed-off-by: Dimitris Karakasilis <[email protected]> --------- Signed-off-by: Dimitris Karakasilis <[email protected]>

as part of kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]>

* Port genkey command from enki as part of kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]> * Port genkey e2e test and don't use viper Signed-off-by: Dimitris Karakasilis <[email protected]> * Run the genkey tests in CI Signed-off-by: Dimitris Karakasilis <[email protected]> * Allow certs with negative serial numbers Not sure why the asus certs have a negative serial number but they do. If the asus box had that others might too, so we should better support it. The alternative would be to generate certs with positive serial number for the tests. https://github.com/golang/go/blob/master/src/crypto/x509/parser.go#L1014-L1018 microsoft/mssql-docker#895 (comment) Signed-off-by: Dimitris Karakasilis <[email protected]> * Remove focus Signed-off-by: Dimitris Karakasilis <[email protected]> --------- Signed-off-by: Dimitris Karakasilis <[email protected]>

jimmykarily · 2024-11-12T10:53:08Z

New auroraboot release: https://github.com/kairos-io/AuroraBoot/releases/tag/v0.3.0
We should be good to archive the enki repository now. If there is any importing of code from that repo, we will port to auroraboot or kairos-sdk as soon as we discover it. No changes or fixes should be made on enki from now on.

jimmykarily · 2024-11-12T15:07:20Z

Cut another release which completely removes enki as a dependency: https://github.com/kairos-io/AuroraBoot/releases/tag/v0.3.1

also the enki repository is now archived.

So we can deprecate the osbuilder-tools image in favor of this one. Part of kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]>

because it has been merged with auroraboot here: kairos-io/AuroraBoot#110 as part of kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]>

part of kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]>

…110) * Copy over the osbuilder-tools image assets and merge with auroraboot So we can deprecate the osbuilder-tools image in favor of this one. Part of kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]> * Print command output when things fail Signed-off-by: Dimitris Karakasilis <[email protected]> * Build versioned image with earthly Signed-off-by: Dimitris Karakasilis <[email protected]> * Fix ARG being in the global scope and not visible in the stage this making the `auroraboot --version` command fail https://docs.docker.com/build/building/variables/#scoping Signed-off-by: Dimitris Karakasilis <[email protected]> * Set the correct arg in pipeline Signed-off-by: Dimitris Karakasilis <[email protected]> * Add back qemu (needed by some test) Signed-off-by: Dimitris Karakasilis <[email protected]> * Remove duplicate block in dockerfile and keep luet repos up to date Signed-off-by: Dimitris Karakasilis <[email protected]> * Support images in the form of "dir:", in build-iso Signed-off-by: Dimitris Karakasilis <[email protected]> * Let elemental handle the schema and create missing tmp dir Signed-off-by: Dimitris Karakasilis <[email protected]> * Remove not necessary flag it seems that it only fails when the user building is not the same as the owner of the .git directory: golang/go#53532 I only saw it when running: ``` docker run --rm -it -v $PWD:/work --workdir /work golang go build . ``` With `docker build` it just works (maybe that command handles the users differently?). In any case, it doesn't affect CI or anything else so I'll remove it. Signed-off-by: Dimitris Karakasilis <[email protected]> --------- Signed-off-by: Dimitris Karakasilis <[email protected]>

jimmykarily · 2024-11-15T13:21:44Z

I extracted the last item to a separate ticket. Since it has nothing to do with consolidation of osbuilder and auroraboot, I don't think it is part of this epic. Let's close this one as soon as the other 2 items are done.

* Remove references to osbuilder-tools image because it has been merged with auroraboot here: kairos-io/AuroraBoot#110 as part of kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]> * WIP Signed-off-by: Dimitris Karakasilis <[email protected]> * Fix test command Signed-off-by: Dimitris Karakasilis <[email protected]> * go mod tidy Signed-off-by: Dimitris Karakasilis <[email protected]> * Bump linting action Signed-off-by: Dimitris Karakasilis <[email protected]> * Don't look for artifacts in build/ subdir because it has been fixed here: kairos-io/AuroraBoot#115 Signed-off-by: Dimitris Karakasilis <[email protected]> --------- Signed-off-by: Dimitris Karakasilis <[email protected]>

* Use auroraboot image as tools image part of kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]> * Update crds from osbuilder repo Signed-off-by: Dimitris Karakasilis <[email protected]> * Bump action Signed-off-by: Dimitris Karakasilis <[email protected]> * Bump chart version Signed-off-by: Dimitris Karakasilis <[email protected]> --------- Signed-off-by: Dimitris Karakasilis <[email protected]>

jimmykarily added this to 🧙Issue tracking board Jul 24, 2023

jimmykarily converted this from a draft issue Jul 24, 2023

This was referenced Jul 24, 2023

Osbuilder should use the Auroraboot image #1634

Closed

Deprecate osbuilder-tools image in favor or Auroraboot #1635

Closed

📢 Kairos release v2.4.0 #1313

Closed

jimmykarily added epic lane/factory labels Jul 24, 2023

jimmykarily changed the title ~~Consolidate osbuilder and Auroraboot~~ 🏭 Consolidate osbuilder and Auroraboot Jul 24, 2023

jimmykarily mentioned this issue Jul 24, 2023

🌱 Split elemental-cli into a lifecycle manager for the system (install/upgrade) and a building component #711

Closed

6 tasks

mudler mentioned this issue Sep 5, 2023

📢 Kairos release v2.5.0 #1314

Closed

22 tasks

jimmykarily mentioned this issue Sep 14, 2023

AuroraBoot missing dependencies when running binary, not starting pixieboot when running in container. #1757

Closed

jimmykarily removed the status in 🧙Issue tracking board Oct 23, 2023

This was referenced Nov 27, 2023

enki should be able to convert an image to Kairos #1721

Closed

enki should be able to built any type of artifact from an oci image #2027

Open

mudler mentioned this issue Dec 11, 2023

📢 Kairos release v3.2.0 #2052

Closed

38 tasks

jimmykarily mentioned this issue Jan 8, 2024

Build UKI's with osbuilder #870

Closed

5 tasks

mudler mentioned this issue Apr 18, 2024

📢 Kairos release v3.3.0 #2127

Open

34 tasks

jimmykarily moved this to In Progress 🏃 in 🧙Issue tracking board Oct 14, 2024

jimmykarily mentioned this issue Oct 14, 2024

Consider replacing Auroraboot for netbooting #2928

Open

jimmykarily self-assigned this Oct 14, 2024

This was referenced Oct 24, 2024

Add commands interface kairos-io/AuroraBoot#76

Merged

Introduce build-iso command kairos-io/AuroraBoot#77

Merged

jimmykarily added a commit to kairos-io/AuroraBoot that referenced this issue Nov 1, 2024

[WIP] Import the build-uki command from enki

fa7a4aa

As part of kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]>

jimmykarily mentioned this issue Nov 1, 2024

Refactor enki build-uki command kairos-io/enki#224

Closed

jimmykarily added a commit to kairos-io/AuroraBoot that referenced this issue Nov 6, 2024

[WIP] Import the build-uki command from enki

61a1e8e

As part of kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]>

jimmykarily added a commit to kairos-io/AuroraBoot that referenced this issue Nov 7, 2024

[WIP] Import the build-uki command from enki

b4f528b

As part of kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]>

jimmykarily mentioned this issue Nov 11, 2024

1633 port genkey command kairos-io/AuroraBoot#98

Merged

jimmykarily added a commit to kairos-io/AuroraBoot that referenced this issue Nov 12, 2024

Port genkey command from enki

0379904

as part of kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]>

jimmykarily added a commit to kairos-io/AuroraBoot that referenced this issue Nov 12, 2024

Port sysext command from enki

e0f7e0c

part of: kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]>

jimmykarily mentioned this issue Nov 12, 2024

1633 port sysext command from enki kairos-io/AuroraBoot#101

Merged

jimmykarily added a commit to kairos-io/AuroraBoot that referenced this issue Nov 12, 2024

Port genkey command from enki

13530ac

as part of kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]>

This was referenced Nov 12, 2024

1633 no more enki dependencies kairos-io/AuroraBoot#102

Merged

[WIP] Use diskfs for uki iso build kairos-io/enki#186

Closed

Bundle bios boot artifacts kairos-io/enki#219

Closed

jimmykarily mentioned this issue Nov 13, 2024

Add auroraboot kairos-io/packages#1138

Closed

jimmykarily mentioned this issue Nov 13, 2024

Copy over the osbuilder-tools image assets and merge with auroraboot kairos-io/AuroraBoot#110

Merged

jimmykarily mentioned this issue Nov 13, 2024

Remove tools image references and opt for auroraboot kairos-io/osbuilder#192

Merged

jimmykarily added a commit to kairos-io/helm-charts that referenced this issue Nov 14, 2024

Use auroraboot image as tools image

c01b847

part of kairos-io/kairos#1633 Signed-off-by: Dimitris Karakasilis <[email protected]>

This was referenced Nov 14, 2024

Use auroraboot image as tools image kairos-io/helm-charts#22

Merged

Fix indentation and add a TODO to fix the curl command kairos-io/kairos-docs#311

Merged

jimmykarily mentioned this issue Nov 15, 2024

Stop shelling out from auroraboot #3019

Open

jimmykarily closed this as completed Nov 15, 2024

github-project-automation bot moved this from In Progress 🏃 to Done ✅ in 🧙Issue tracking board Nov 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🏭 Consolidate osbuilder and Auroraboot #1633

🏭 Consolidate osbuilder and Auroraboot #1633

jimmykarily commented Jul 24, 2023 •

edited

Loading

jimmykarily commented Aug 1, 2023 •

edited

Loading

jimmykarily commented Nov 27, 2023

jimmykarily commented Oct 16, 2024 •

edited

Loading

jimmykarily commented Nov 12, 2024

jimmykarily commented Nov 12, 2024

jimmykarily commented Nov 15, 2024

🏭 Consolidate osbuilder and Auroraboot #1633

🏭 Consolidate osbuilder and Auroraboot #1633

Comments

jimmykarily commented Jul 24, 2023 • edited Loading

jimmykarily commented Aug 1, 2023 • edited Loading

jimmykarily commented Nov 27, 2023

jimmykarily commented Oct 16, 2024 • edited Loading

jimmykarily commented Nov 12, 2024

jimmykarily commented Nov 12, 2024

jimmykarily commented Nov 15, 2024

jimmykarily commented Jul 24, 2023 •

edited

Loading

jimmykarily commented Aug 1, 2023 •

edited

Loading

jimmykarily commented Oct 16, 2024 •

edited

Loading