Roll back not working as expected

[kannan4183]

HI team, We are testing Strimzi Operator & Kafka version upgrade and rollback . Following are the steps performed.

1. Upgraded the Strimzi Operator from "0.29.0" to "0.33.2" and Kafka version from "3.0.0" to "3.2.3" with "inter.broker.protocol.version" set to "3.0.0" . We have removed the configuration set for "log.message.format.version", since from version "3.0.0" this configuration is ignored.
2. Everything is looking good after performing the above step with "inter.broker.protocol.version" still set to "3.0" and noticed that in broker pod's "strimzi.properties" files has both "inter.broker.protocol.version" & "log.message.format.version" set to "3.0".

auto.create.topics.enable=false
default.replication.factor=3
group.initial.rebalance.delay.ms=0
inter.broker.protocol.version=3.0
log.retention.check.interval.ms=300000
log.retention.hours=168
log.segment.bytes=1073741824
num.io.threads=8
num.network.threads=3
num.partitions=1
num.recovery.threads.per.data.dir=1
num.replica.fetchers=1
offsets.topic.replication.factor=3
replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
ssl.cipher.suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
ssl.enabled.protocols=TLSv1.2
ssl.protocol=TLSv1.2
transaction.state.log.min.isr=2
transaction.state.log.replication.factor=3
zookeeper.connection.timeout.ms=6000
metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
log.message.format.version=3.0

3. Now started testing on roll back. Since Strimzi Operator version "0.33.2" doesn't support the Kafka version "3.0.0", we have decided to follow 3 step downgrading as follows.

a. Keep Strimzi operator version to "0.33.2" and downgrade Kafka version to "3.2.0" which is a supported version in both Strimzi Operator versions "0.33.2" & "0.29.0".
b. Downgrade Strimzi Operator version from "0.33.2" to 0.29.0" and keep Kafka version to "3.2.0"
c. Keep Strimzi operator version to "0.29.0" and downgrade kafka version from "3.2.0" to "3.0.0"

After performing the first roll back step mentioned above with "inter.broker.protocol.version" still set to "3.0", we noticed strange behaviour that "log.message.format.version" changed to "3.2" and "inter.broker.protocol.version" still set to "3.0" in broker pod properties file.

allow.everyone.if.no.acl.found=true
auto.create.topics.enable=true
default.replication.factor=3
group.initial.rebalance.delay.ms=0
inter.broker.protocol.version=3.0
log.retention.check.interval.ms=300000
log.retention.hours=168
log.segment.bytes=1073741824
num.io.threads=8
num.network.threads=3
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=3
replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
ssl.cipher.suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
ssl.enabled.protocols=TLSv1.2
ssl.protocol=TLSv1.2
transaction.state.log.min.isr=2
transaction.state.log.replication.factor=3
zookeeper.connection.timeout.ms=6000
metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
log.message.format.version=3.2

Could you please let me know if I am missing something here?

[scholzj]

The log.message.format-version is auto-configured based on the Kafka version and your inter.broker.protocol.version setting. But to be honest, I do not remember exactly how it worked in these old versions, but I think it might correspond to those things. As you said, with 3.0, the log message format is ignored, so it should not be a problem I think.

[kannan4183]

Thanks @scholzj for the response.

The reconciliation appears to be stalled as a result of this behaviour when I perform the previously indicated last roll back step.

2024-08-02 06:23:30 WARN  AbstractOperator:532 - Reconciliation #28(watch) Kafka(kite-emm-ns/kite-emm): Failed to reconcile
io.strimzi.operator.cluster.KafkaUpgradeException: log.message.format.version (3.2) and inter.broker.protocol.version (3.0) used by the brokers have to be set and be lower or equal to the Kafka broker version we downgrade to (3.0.0)

However, I was able to successfully roll back the upgrade by adding "log.message.format.version" to the Kafka configurations before starting the rollback process.

Is Strimzi Operator updating the configuration's default value during operator rollback?

[scholzj]

I don't think so. But to be honest, these are very old versions and I do not remember every detail of what they did and how.

[kannan4183]

thanks