[kube-prometheus-stack] upgrade CRDs using helm

[jkroepke]

What this PR does / why we need it

Since helm does not support upgrading CRDs, it's an manual task for our end-users. However the velero chart https://github.com/vmware-tanzu/helm-charts/tree/main/charts/velero/templates/upgrade-crds implement a upgrade job using helm hooks that works for at least 1 year.

Which issue this PR fixes

(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

• fixes #

Special notes for your reviewer

I would leave it disabled for now in order to have the ability to test the job on real upgrades.

Checklist

• DCO signed
• Chart Version bumped
• Title of the PR starts with chart name (e.g. [prometheus-couchdb-exporter])

[jkroepke]

@QuentinBisson @GMartinez-Sisti this is super WIP (but working) what did you think? It based on https://github.com/vmware-tanzu/helm-charts/tree/main/charts/velero/templates/upgrade-crds?rgh-link-date=2025-01-15T09%3A52%3A20Z

[GMartinez-Sisti]

@QuentinBisson @GMartinez-Sisti this is super WIP (but working) what did you think? It based on https://github.com/vmware-tanzu/helm-charts/tree/main/charts/velero/templates/upgrade-crds?rgh-link-date=2025-01-15T09%3A52%3A20Z

I'm all up for it! 🙌 -> #4507

[jkroepke]

I'm all up for it! 🙌 -> #4507

Please take note that the CRDs needs to be pre-compressed, All CRDs in gzip format are approx. 300KB, while xz just using 80KB.

The creation of the tar.xz is part of hack scripts, but how we protect us against suspicious tar files?

[GMartinez-Sisti]

I'm all up for it! 🙌 -> #4507

Please take note that the CRDs needs to be pre-compressed, All CRDs in gzip format are approx. 300KB, while xz just using 80KB.

The creation of the tar.xz is part of hack scripts, but how we protect us against suspicious tar files?

Checksum? We can add a checksum file next to it, then run the same on ci and check the match.

[jkroepke]

I give-up after an hour to setup an reproducible tar archive which can be generated on any os (e.g. mac and linux)

For example, I can't find a tar option on mac where I can drop the mtime of the files.

GNU tar adds some pax header while bsd tar does not do it.

[GMartinez-Sisti]

I give-up after an hour to setup an reproducible tar archive which can be generated on any os (e.g. mac and linux)

For example, I can't find a tar option on mac where I can drop the mtime of the files.

GNU tar adds some pax header while bsd tar does not do it.

Yeah, it’s hard to find a way with native commands, I usually fix these issues by running the commands in a docker container.

[jkroepke]

What about check,

I give-up after an hour to setup an reproducible tar archive which can be generated on any os (e.g. mac and linux)
For example, I can't find a tar option on mac where I can drop the mtime of the files.
GNU tar adds some pax header while bsd tar does not do it.

Yeah, it’s hard to find a way with native commands, I usually fix these issues by running the commands in a docker container.

please offer a solution.

While tar isn't the big issue, xz might be. Seems not possible to get the same archive from macos/linux using xz compression. Even with gnu tar on mac, the archive is different. gz is not an option, because the compression rate is really, really bad.

Tests with zstd works fine, however busybox does not support zstd.

Then, I tried docker. The ubuntu docker image does not have xz pre installed. starting with apt-get first feels terrible slow. busybox have xz embedded, but does not offer capabilities to strip the mtime.

[GMartinez-Sisti]

please offer a solution.

While tar isn't the big issue, xz might be. Seems not possible to get the same archive from macos/linux using xz compression. Even with gnu tar on mac, the archive is different. gz is not an option, because the compression rate is really, really bad.

Tests with zstd works fine, however busybox does not support zstd.

Then, I tried docker. The ubuntu docker image does not have xz pre installed. starting with apt-get first feels terrible slow. busybox have xz embedded, but does not offer capabilities to strip the mtime.

Sure, I can test this during the weekend.

[jkroepke]

Ready to review!

[QuentinBisson]

Is it expected that your PR carry the binary #5175 (comment)?

EDIT: Yes it is.

Overall I quite like the job approach but i'm not really fond of the archive though. Is there a specific reason why we cannot use curl to fetch the crds from this chart app version instead?

[jkroepke]

Overall I quite like the job approach but i'm not really fond of the archive though. Is there a specific reason why we cannot use curl to fetch the crds from this chart app version instead?

Yeah, sideloading is not good in enterprise context where internet is restricted. That lead to unexpected situations.

One possible solution would be that we provide the CRDs as OCI image. Then, we can mount the image as PVC. but It will take some time: https://kubernetes.io/blog/2024/08/16/kubernetes-1-31-image-volume-source/

[aidanleuck]

This is awesome! I started looking at ways I could implement something similar in our wrapper chart. We use Terraform to deploy this chart and manage 20-30 Kubernetes clusters. Performing the CRD upgrades manually when we needed to take a major version upgrade had become tedious. This will help out our org a ton and resolve our biggest pain point with the deployment.

[jkroepke]

@aidanleuck great to know. Once implemented, we are happy about feedback.

[muffl0n]

Really looking forward to this, too! If you need any early feedback, let me know!

[aidanleuck]

Overall I quite like the job approach but i'm not really fond of the archive though. Is there a specific reason why we cannot use curl to fetch the crds from this chart app version instead?

Yeah, sideloading is not good in enterprise context where internet is restricted. That lead to unexpected situations.

One possible solution would be that we provide the CRDs as OCI image. Then, we can mount the image as PVC. but It will take some time: https://kubernetes.io/blog/2024/08/16/kubernetes-1-31-image-volume-source/

I like mounting an OCI image, but as you mentioned this is a cutting-edge feature and it won't work with older clusters. If I understand this PR correctly, the busybox image is doing the decompression of the CRD archive, the results of which are placed in a PVC that the kubectl pod uses to apply. Would it be simpler if you created a Dockerfile and extend the kubectl image? That way you don't need a shared PVC at all and you just use a single container image to run the job.

CI/Deployment you can build the custom kubectl image and the CRDS bundled together. You can tag the image with the same version as the appVersion. This will get rid of the tar archive issue as well.

Here's the pseudocode (Note use a multi-stage build if you want to merge all the crds into a single file)
Dockerfile

FROM registry.k8s.io/kubectl
COPY ./charts/crds/crds /var/crds

job.yaml

image: <custom-built-image>
workingDir: /var/
command:    
  - kubectl
args:
  - apply
  - --server-side
  - --filename
  - crds/*.yaml

Let me know if I am off base here and am missing something, but this seems like a simpler solution to the problem. I can whip up a POC if you like this idea.