Sidecar TLS Certificate Reloader

[treyhendon]

Would it be possible to put a secret watching sidecar in the pods which refreshes the mounted Kubernetes secret (certificate) and registers the certificate in the primary container?

(I'm not sure what the implications to Kafka itself would be if the certificate changes at the OS level without restarting the app.)

The goal on this would be to prevent rolling the cluster every two months when Let's Encrypt certs renew.

[scholzj]

We do not support any sidecars. You would need to inject it using a webhook. But I think the secret files are updated automatically without any sidecar. But Kafka does not reload them just because the file changes.

[treyhendon]

But Kafka does not reload them just because the file changes.

That's what I was wondering. I saw in the pod startup that the projected certificate file (from secret) gets registered in the OS. I wasn't sure if that was enough for Kafka to use the new cert version at the next SASL auth request or not.

We might play more with this, thanks!

[scholzj]

Well, first of all, the loaded certificates are currently converted to PKCS12 stores at startup. But even without that, Kafka does not reload it just because the file changes. You would need to use the Kafka Admin API to trigger the reload. There are also some limitations to what might have changed in the certificate to be able to reload it dynamically (for example no changes to the certificate subject).

Strimzi currently does not support this. We might in the future, but because of the limitations, it is, to be honest not high on our priority list and it is not trivial as you need to be able to do it both through rolling updates as well when needed.