StrimziPodSets and Broker IDs

[Manicben]

Hey everyone!

With StrimziPodSets coming soon™, I was wondering whether it would allow us to customise the Broker IDs used by each pod.
I imagine by default it will be similar to the current approach, where each broker spun up is assigned an incremental ID starting from 0, but would it be possible to have the ID start from another arbitrary number, or even assign arbitrary numbers per broker via broker-specific configuration?

The main use case I can think of for this is helping with zero downtime migrations from an existing Kafka cluster to a Strimzi-managed cluster. As long as the existing cluster shares the same Zookeeper as the destination cluster (which is possible by copying the data over), then spinning up Strimzi-managed brokers with specific IDs that don't conflict with the existing brokers will help massively with migration, via moving partitions or temporarily increasing replication factor.
This would be much more preferable to using MirrorMaker2 to replicate data, then orchestrate a complex migration with minimal downtime and zero data loss.

I imagine there must be other use cases, as I've seen this be supported by a certain other operator, I can only speculate that it must be due to needing to be able to set specific configuration for a specific broker ID. Why though I'm not sure.

Completely understand if this behaviour isn't desirable. The migration method would probably become void as soon as kRaft is production ready anyway. I'm just gauging all the available options for migrating to Strimzi, but it's slowly looking like some downtime is unavoidable.

Keep up the amazing work btw, we're really excited for StrimziPodSets and being able to perform rolling updates more gracefully!

[scholzj]

With StrimziPodSets coming soon™, I was wondering whether it would allow us to customise the Broker IDs used by each pod.
I imagine by default it will be similar to the current approach, where each broker spun up is assigned an incremental ID starting from 0, but would it be possible to have the ID start from another arbitrary number, or even assign arbitrary numbers per broker via broker-specific configuration?

I think it would eventually. However, the initial implementation which is now in the main branch more or less copies the StatefulSet behaviour. And it would be quite hard to maintain the StatefulSet support while having the broker IDs fully configurable etc. So this is something what might happen only after we drop StatefulSets completely. So I think it might be still some time before the broker ID configuration is possible.

The main use case I can think of for this is helping with zero downtime migrations from an existing Kafka cluster to a Strimzi-managed cluster. As long as the existing cluster shares the same Zookeeper as the destination cluster (which is possible by copying the data over), then spinning up Strimzi-managed brokers with specific IDs that don't conflict with the existing brokers will help massively with migration, via moving partitions or temporarily increasing replication factor.
This would be much more preferable to using MirrorMaker2 to replicate data, then orchestrate a complex migration with minimal downtime and zero data loss.

TBH, I think this would be a way more complicated than just the broker IDs since Strimzi currently doesn't expect any outsdie brokers joining etc. So while I can undertsand the motivation, I would not count on this anytime soon.

[Manicben]

Thanks for the very swift response. That is nice to hear and I'm glad to have some certainty.
I had a feeling this would be the case, but didn't hurt to ask :)

Just to confirm, once StrimziPodSets are in use, is the plan to optimise the rollout of pods to ensure the active controller broker is rolled last and that each pod is rolled out based on metrics like underreplicated partitions, thus ensuring minimal disruption to clients?

fwiw, we will most likely use the MirrorMaker2 approach detailed here: https://strimzi.io/blog/2021/11/22/migrating-kafka-with-mirror-maker2/
I'm considering writing my own article on how we achieved it in production when we're done, as migrating Kafka clusters in production is no simple task and is way more complex than what that article covers.

[scholzj]

Just to confirm, once StrimziPodSets are in use, is the plan to optimise the rollout of pods to ensure the active controller broker is rolled last and that each pod is rolled out based on metrics like underreplicated partitions, thus ensuring minimal disruption to clients?

This is the case already now with StatefulSets - even with StatefulSets, we do not use the Kubernetes rollung update features but implement it inside Strimzi to achieve this. It will for sure keep being used even with the StrimziPodSets.

[ppatierno]

@Manicben are you willing to write the article on the official Strimzi blog? It would be great to see how the community is able to solve real problems using Strimzi and MM2 :-)

[Manicben]

@ppatierno Absolutely! Probably will be quite a while, still fleshing out the design and I'll probably write the article after we've migrated some of our production environments without issues.