Configuring ClusterId via Custom Resources: Exploring Options

[aswinayyolath]

Strimzi generates a unique clusterId for each Kafka cluster it manages. This identifier can be retrieved using the kubectl get kafka command, where you'll observe a randomly generated string by the Strimzi operator.

kubectl get kafka my-cluster-2 -n strimzi-2 -o=jsonpath='{.status.clusterId}'

Similarly, you can obtain the clusterId for Kafka node pools with:

kubectl get kafkanodepool my-pool-2 -o=jsonpath='{.status.clusterId}'

Now, let's discuss the possibility of overriding the clusterId with a user-defined value via Strimzi's CR configurations. Is this feature currently supported?

This will be useful for scenarios where you want more control over the cluster id. One such example could be creating stretch clusters where all brokers across different Kubernetes environments must use the same cluster id.

[scholzj]

The cluster ID is generated as unique. So I'm not sure what is the exact problem you are trying to solve.

[aswinayyolath]

Sure, let me elaborate. I was conducting an exploratory spike to delve into the process of creating a logical stretch cluster. Here's what I did:

• Set up 2 Kubernetes clusters.
• Installed Kafka with KRaft and Nodepool enabled.
• Temporarily halted the operator reconciliation using kubectl annotate Kafka <clustername> strimzi.io/pause-reconciliation="true".
• Edited the ConfigMap generated by Strimzi for each broker/controller in both clusters to utilize the same clusterId. This involved adjusting the controller.quorum.voters property in both clusters to include information about Pods running on both Kubernetes clusters.
• Initially, for the spike, we disabled security and made several other alterations to establish a logical cluster.

As I proceeded, I started thinking about a method to enable users to configure the clusterId via Custom Resources, allowing them to specify it in the Kafka Custom Resource.

I would greatly appreciate your insights and perspective on this topic.

[scholzj]

Well, that is fine. But that does not add value to anyone else because you are basically just hacking around. So I do not think it justifies any effort to support this.

Leaving that aside, I'm not sure how would that be configured because it is not really ocnfigurable beyond the initial deployment. So having an option for it that will be ignored all the time apart from the initial deployment would be probably very confusing.

But what ou can try is to edit the status of the Kafka resource when you create it and set your desired cluster Id there.

[aswinayyolath]

Thank you for your thoughtful response and insights. I completely understand your perspective. My intention was to explore the manual adjustments required initially, with the aim of potentially incorporating them into the operator logic later on. I agree that having an option for configuring only during the initial deployment could indeed lead to confusion. Additionally, I'm fully aware that implementing stretch clusters involves considerable effort and considerations. Your guidance is invaluable in shaping our approach moving forward. Thank you for your patience and assistance in this matter.

[siegenthalerroger]

So after running in to an issue with our KRaft cluster, we ended up needing to restore our cluster from a backup. Our B/DR system uses velero to backup our PVCs & PVs and relies on FluxCD to restore the k8s resources to the same point in time as the backup was made. This implies that strimzi's Kafka CR would be deterministic in it's creation of a Kafka resource.

However this is not a given due to the fact that the clusterID is randomly generated. This makes restoring a strimzi Kafka cluster from data disks a painful process with a race-condition after FluxCD applies the Kafka CR (between the user setting the clusterID within the Kafka.status subresource and strimzi's cluster-operator starting the cluster).

I'm not sure what the benefit of having the clusterID be random is tbh and would really like to see the Kafka CR spec expanded to allow manually setting a clusterID value. An alternative could also be to change the operator to not generate a random ID but rather something deterministic based on the name, possibly requiring the configuration of a random-seed strimzi-wide. Is there any reason not to allow manually setting this value, seeing as you currently can do it by directly manipulating Kafka.status which just seems like an absolute hack tbh.

---

Slack conversation about restoring and the clusterID problem: https://cloud-native.slack.com/archives/CMH3Q3SNP/p1720463076090579?thread_ts=1720455627.117929&cid=CMH3Q3SNP

[siegenthalerroger]

The read-only (after creation) field would not help if I set a wrong cluster ID. That is correct. If the user actively does something wrong you have the same scenario as you do today essentially. HOWEVER; if you set the cluster ID correctly, you solve an issue with the workaround that requires a multi-step fix (pausing strimzi reconcilliation, deleting all the unnecessarily created resources, setting the Kafka.status.clusterID AND KafkaNodePool.status.clusterID field, only now restoring the original disks, restarting strimzi reconcilliation). In our scenario where all k8s resources are version controlled and deployed with FluxCD, it's unlikely for us to end up setting the wrong clusterID if it's a part of the CR definition.

I don't really see how the solution of adding a field to manually set the clusterID is so "out there", to an extent that it's unusable (and unmaintainable). I don't know if adding the field would be a massive effort, I honestly don't know where the current clusterID random generation takes place (in strimzi or somewhere in Kafka itself) so I can't make any assumptions about what implementing this idea would entail. It just seems like a bit of a pessimistic take to be so negative regarding this; there are two use-cases here (my B/DR one and @aswinayyolath 's more experimental one) which would both benefit from it and I'm sure there's more that would be enabled by this.

[a1exus]

How does a read-only field help if you set a wrong cluster ID?

Look, I understand the problem you have. But there is a workaround you can use today. And any solution has to be usable and maintainable for everyone. Not just for you.

@scholzj - would you please describe such a workaround? or if you can point into the right direction, that'd also help) thanks in advance!

[siegenthalerroger]

@a1exus check out the linked slack convoy for the full details. You can manually set the clusterId by applying it to the Kafka.status.clusterId field. You'll need to explicitly tell kubectl to apply to the status subresource and temporarily pause any automatic reconciliation you have running (strimzi & any further like fluxcd).

[a1exus]

@a1exus check out the linked slack convoy for the full details.

i tried to access it, however

my email doesn’t have an account on this workspace.

[scholzj]

You can register for the CNCF Slack here: https://slack.cncf.io/

But in general, you can set the cluster ID in the .status section of the Kafka CR. You can do it for example with something like kubectl edit kafka my-cluster --subresource=status.

[cashol]

Hi.
I also run into a problem relating to clusterID, when redeploying kafka cluster (using KRaft) after first deployment:
Exception in thread "main" java.lang.RuntimeException: Invalid cluster.id in: /var/lib/kafka/data-0/kafka-log0/meta.properties. Expected cfuiDEZDSK6r10UaZL3Idw, but read 9FlnsttMTUerIhRLoEiZ4g
I am using Docker Desktop with Kubernetes enabled on Windows 10 Prof.

[cashol]

Workaround:
Remove meta.properties from all kafka related PVCs, e.g. data-0-kafka-cluster-broker-0 etc.

BTW: On Docker Desktop, deployed via docker-compose (confluent images used), this way does NOT help!