feat: rewrite release system to support multiple images per commit

[innovaker]

A critical limitation of the first design was the assumption that there would only be one Docker build per commit. As such, software packages were often only refreshed when Zephyr was upgraded.

This new design opens the door to better CI practices. It allows regular rebuilds of the Docker images irrespective of version control. This is critical for incorporating the latest security fixes and bug patches as soon as possible.

Maintainers are still required to trigger stable releases (via tags), but this can be revisited in the future if further automation is necessary.

[petejohanson]

A critical limitation of the first design was the assumption that there would only be one Docker build per commit. As such, software packages were often only refreshed when Zephyr was upgraded.

This new design opens the door to better CI practices. It allows regular rebuilds of the Docker images irrespective of version control. This is critical for incorporating the latest security fixes and bug patches as soon as possible.

Maintainers are still required to trigger stable releases (via tags), but this can be revisited in the future if further automation is necessary.

Can you please help me understand the workflow here? Seems like you're promposing a tag system where we tag the same commit again with updated zephyr/sdk versions, timestamp, etc and by pushing multiple tags, we build and push those equivalent docker image tags w/ the appropriate bundled software in it?

[innovaker]

Can you please help me understand the workflow here? Seems like you're promposing a tag system where we tag the same commit again with updated zephyr/sdk versions, timestamp, etc and by pushing multiple tags, we build and push those equivalent docker image tags w/ the appropriate bundled software in it?

That's correct. As was done previously, to release a particular candidate image we take its docker tag and push an equivalent git tag to the associated commit. That then triggers the release process. Said tag can then either be left there or deleted. It's not ideal and a GitHub API (release?) trigger would probably be more appropriate, but this was the quickest solution.

The key takeaway is that it lets us run regular (nightly?) builds to pick up security fixes and patches, from which we can periodically release stable versions outside the Zephyr lifecycle.

Given that we've now got comprehensive integration tests (which didn't exist when I developed this PR), we may also wish to automate regular releases for security and patches, maybe the base images too. In which case the release system would become redundant.

[innovaker]

@petejohanson, our discussion on #development got me thinking, and I realised it was worth recording the reasoning/purpose for each field in the tag ...

• branch
  • alphabetical grouping of candidates
  • filtering of candidates
  • soon to be the tag for the "latest" cache for each branch
• run_id
  • locating logs on GitHub, i.e. https://github.com/zmkfirmware/zmk-docker/actions/runs/#########
  • 1/2 of the unique identifier
• timestamp
  • 1/2 of the unique identifier (because of the potential for re-runs of the same workflow run)
  • maybe for human-readable searches?
• sha
  • finding the commit it was built from
  • checking it matches the Docker tag as a safeguard in the release trigger job
    • it prevents releases via an unrelated commit which may have an older or incompatible release process implemented.
• Zephyr version
  • easy reference
• Zephyr SDK version
  • easy reference

In terms of ease of use:

• GitHub
  • often chops the git reference off after 25 characters.
    • For PR branches, that often doesn't even cover the branch name.
    • For maintainence branches that's enough for the major.minor (e.g. 2.4) plus something else
  • You can find the sha if you know the run_id, but not the other way around?
    • I'm not sure if you can do either programmatically? Probably via the GitHub API?
• Docker Hub
  • shows the full tag
  • one entry/row per tag - i.e. multiple rows per image
  • has a text filter
  • has a sort by (not always reliable):
    • newest & oldest
      • I presume this is by push timestamp rather than last pull timestamp?
    • alphabetical
• GitHub Packages (Container Registyr)
  • shows the full tag
  • one row/entry per image - i.e. tags are grouped together
  • no filters or sorting?

Perhaps that sways your thoughts about which are necessary (or can be dropped?) and/or the ordering. I'm still thinking it through.

[innovaker]

I'm also open to shortening the sha if desirable. My understanding is that both git CLI and GitHub (search, links, etc.) can handle it. GitHub currently limits it to 7 characters but supports any length.

[innovaker]

I'm happy enough with the current approach. Candidate tags are relative shortlived and we can tweak it later if necessary.

[innovaker]

I guess another approach to this is to make the branch less prominent, so these ...

• main
  • candidate image: main-20210522132909-2.5.0-0.12.4-7615f98a3a37ed403a5ed40d656c0c2901318fad-866653742
  • cache: main
• feature branch / PR
  • candidate: feat_blah-blah-blah-20210522132909-2.5.0-0.12.4-7615f98a3a37ed403a5ed40d656c0c2901318fad-866653742
  • cache: feat_blah-blah-blah
• maintainence branch
  • candidate: 2.5-20210522132909-2.5.0-0.12.4-7615f98a3a37ed403a5ed40d656c0c2901318fad-866653742
  • cache: 2.5

... would become ...

• main
  • candidate: 2.5.0-0.12.4-20210522132909-main-7615f98a3a37ed403a5ed40d656c0c2901318fad-866653742
  • cache: main
• feature branch / PR
  • candidate: 2.5.0-0.12.4-20210522132909-feat_blah-blah-blah-7615f98a3a37ed403a5ed40d656c0c2901318fad-866653742
  • cache: feat_blah-blah-blah
• maintainence branch
  • candidate: 2.5.0-0.12.4-20210522132909-2.5-7615f98a3a37ed403a5ed40d656c0c2901318fad-866653742
  • cache: 2.5

with the option of dropping branch from the candidate name entirely.

( side note: IFAICT, the cache must be per branch to avoid potential conflicts - see #56 )

I'm not sure how I feel about this, but it's certainly another option.

[innovaker]

I've been experimenting with abbreviated SHA codes and I think I prefer them. I'll squash that in soon.

[innovaker]

I've also been giving some thought as to how (major.minor) 2.x tags should play nicely with 2.x branches. Both for GitHub and Docker tags.

This is important given the "latest" Docker tag for each branch is now the branch name. So there's potential for overlap/conflict.

There's a few options to consider, some involving #17 and some not. I'm still firming up my thoughts around it.

[innovaker]

@petejohanson, with #62 now concluded, how are you feeling about this PR please?

[petejohanson]

@innovaker This all seems reasonable to me. I suppose the only question I have is do we want to have any enforcement to be sure that any tag is "compatible" with the branch that contains the commit in question that has that tag applied to it?

[innovaker]

After some discussion on Discord, it was decided to add a guard.
https://discord.com/channels/719497620560543766/719544230497878116/852190615848091669

[innovaker]

Further discussion concluded the existing protection should be sufficient:
https://discord.com/channels/719497620560543766/719544230497878116/852567300170842153

[innovaker]

@petejohanson, I think this is ready for approval.