[Bug]: MirrorMaker2 KafkaAdminClient keystore password was incorrect #10760

barrytam20 · 2024-10-24T19:35:54Z

barrytam20
Oct 24, 2024

Bug Description

Hey there - I am trying to set up MirrorMaker2 on k8s. I am intending to use it to rename topics that live on the same cluster, and I will eventually implement my own ReplicationPolicy. As of now, I want to set it up so that the sourceCluster and the targetCluster are the same.

I've attached the manifest I used to create the KafkaMirrorMaker2 resource, as well as the logs. It seems like it creates two keys

/tmp/kafka/cluster.truststore.p12
/tmp/kafka/clusters/default-cluster.truststore.p12

It seems like one of those keys work, since I do see the follow MirrorMaker topics created on my Kafka cluster

mirrormaker2-cluster-status
mirrormaker2-cluster-configs
mirrormaker2-cluster-offsets
And the status and configs topics both have messages in them.

Looking at the pod logs, it fails to start the connector since it fails to create a KafkaAdminClient

19:02:05.537 ERROR org.apache.kafka.connect.runtime.WorkerConnector:192 WorkerConnector{id=default-cluster->default-cluster.MirrorSourceConnector} Error while starting connector
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:545)
	at org.apache.kafka.clients.admin.Admin.create(Admin.java:143)
	at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49)
	at org.apache.kafka.connect.mirror.MirrorSourceConnector.start(MirrorSourceConnector.java:117)
	at org.apache.kafka.connect.runtime.WorkerConnector.doStart(WorkerConnector.java:184)
	at org.apache.kafka.connect.runtime.WorkerConnector.start(WorkerConnector.java:209)
	at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:348)
	at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:331)
	at org.apache.kafka.connect.runtime.WorkerConnector.doRun(WorkerConnector.java:140)
	at org.apache.kafka.connect.runtime.WorkerConnector.run(WorkerConnector.java:117)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /tmp/kafka/clusters/default-cluster.truststore.p12 of type PKCS12
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:377)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:349)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createTruststore(DefaultSslEngineFactory.java:322)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:168)
	at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:140)
	at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:97)
	at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:73)
	at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
	at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
	at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:516)
	... 14 more
Caused by: java.io.IOException: keystore password was incorrect
	at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159)
	at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)
	at java.base/java.security.KeyStore.load(KeyStore.java:1473)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:374)
	... 24 more
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
	... 28 more

Exec-ing into the pod also shows that the two keys are the same size; I'm assuming one must work since I see topics created on the cluster. How come the it seems like /tmp/kafka/clusters/default-cluster.truststore.p12 doesn't work? Is there any way around this?

Steps to reproduce

Create a configmap with the logging config
Create a secret that holds the TLS cert
Apply the attached manifest

Expected behavior

I would expect the KafkaMirrorMaker2 task to get into a running state

Strimzi version

0.33.2

Kubernetes version

1.29

Installation method

YAML files

Infrastructure

Amazon EKS

Configuration files and logs

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: my-mirror-maker
spec:
  version: 3.2.0
  replicas: 1
  connectCluster: "default-cluster"
  clusters:
    - alias: "default-cluster"
      bootstrapServers: my.kafka.bootstrap.servers:8093, my.kafka.bootstrap.servers:8094, my.kafka.bootstrap.servers:8095
      tls:
        trustedCertificates:
          - secretName: bbt-mirror-maker
            certificate: crt.crt
      config:
        config.storage.replication.factor: 3
        offset.storage.replication.factor: 3
        status.storage.replication.factor: 3
        config.providers: secrets
        config.providers.secrets.class: org.apache.kafka.common.config.provider.DirectoryConfigProvider
        config.reload.action: restart
        ssl.cipher.suites: "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
        ssl.enabled.protocols: "TLSv1.2"
        ssl.protocol: "TLSv1.2"
        ssl.endpoint.identification.algorithm: HTTPS
  mirrors:
    - sourceCluster: "default-cluster"
      targetCluster: "default-cluster"
      sourceConnector:
        tasksMax: 10
        config:
          replication.factor: 3
          offset-syncs.topic.replication.factor: 3
          sync.topic.acls.enabled: "false"
          refresh.topics.interval.seconds: 60
          replication.policy.separator: ""
      topicsPattern: "my-topic"
      groupsPattern: ""
  resources:
    requests:
      cpu: "1"
      memory: 2Gi
    limits:
      cpu: "2"
      memory: 2Gi
  logging:
    type: external
    valueFrom:
      configMapKeyRef:
        name: mirrormaker-logging
        key: log4j.properties
  readinessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
  livenessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
  jvmOptions:
    "-Xmx": "1g"
    "-Xms": "1g"
  template:
    connectContainer:
      env:
        - name: STRIMZI_IMAGE_PULL_POLICY
          value: Always

my-mirror-maker-mirrormaker2-97856fd6d-gjcrb.log

Additional context

No response

scholzj · 2024-10-24T22:53:29Z

scholzj
Oct 24, 2024
Maintainer

We had similar bugs to this in the past. So I think upgrading to latest version and seing if you still have the issue would be a good start. Also, make sure how you use the base container image from the same version of Strimzi as you use as this also tends to cause this kind of issues.

0 replies

barrytam20 · 2024-10-24T23:19:18Z

barrytam20
Oct 24, 2024
Author

Ok I'll give that a try, but it might be longer pull as we have other components that are dependent on Strimzi.

Are you able to find the commit where this was fixed? I'm saw you fixed a similar issue three years ago, but I'm pretty sure that was included in the version we're currently using

1 reply

scholzj Oct 24, 2024
Maintainer

I do not have an exact commit. And I definitely do not know on 100% if this was one of the things we fixed. In any case, we would need to be able to reproduce it on some recent version to be able to analyze and fix it.

barrytam20 · 2024-10-25T17:53:22Z

barrytam20
Oct 25, 2024
Author

Hey there - I upgraded our Strimzi Kafka Operator to 0.43.0 and it looks like I'm running into the same issue. When I describe my kafkamirrormaker2s resource, I see that the connector failed with the same status as before

Status:
  Conditions:
    Last Transition Time:  2024-10-25T17:50:00.119087937Z
    Message:               One or more connectors are in FAILED state
    Reason:                NoStackTraceThrowable
    Status:                True
    Type:                  NotReady
  Connectors:
    Connector:
      State:  FAILED
      Trace:  org.apache.kafka.common.KafkaException: The constructor of org.apache.kafka.clients.admin.ForwardingAdmin threw an exception
              at org.apache.kafka.common.utils.Utils.newParameterizedInstance(Utils.java:480)
              at org.apache.kafka.connect.mirror.MirrorConnectorConfig.forwardingAdmin(MirrorConnectorConfig.java:232)
              at org.apache.kafka.connect.mirror.MirrorSourceConnector.start(MirrorSourceConnector.java:162)
              at org.apache.kafka.connect.runtime.WorkerConnector.doStart(WorkerConnector.java:215)
              at org.apache.kafka.connect.runtime.WorkerConnector.start(WorkerConnector.java:244)
              at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:403)
              at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:384)
              at org.apache.kafka.connect.runtime.WorkerConnector.doRun(WorkerConnector.java:164)
              at org.apache.kafka.connect.runtime.WorkerConnector.run(WorkerConnector.java:124)
              at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:237)
              at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
              at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
              at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
              at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
              at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
  at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:554)
  at org.apache.kafka.clients.admin.Admin.create(Admin.java:147)
  at org.apache.kafka.clients.admin.ForwardingAdmin.<init>(ForwardingAdmin.java:52)
  at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
  at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
  at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
  at org.apache.kafka.common.utils.Utils.newParameterizedInstance(Utils.java:469)
  ... 14 more
Caused by: org.apache.kafka.common.KafkaException: Failed to create new NetworkClient
  at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:254)
  at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:189)
  at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:538)
  ... 22 more
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /tmp/kafka/clusters/default-cluster.truststore.p12 of type PKCS12
  at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:382)
  at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:354)
  at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createTruststore(DefaultSslEngineFactory.java:327)
  at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:171)
  at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:141)
  at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:98)
  at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:70)
  at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
  at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
  at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:119)
  at org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:223)
  ... 24 more
Caused by: java.io.IOException: keystore password was incorrect
  at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159)
  at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)
  at java.base/java.security.KeyStore.load(KeyStore.java:1473)
  at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:379)
  ... 34 more
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
  ... 38 more

Since I'm not specifying an image to use in my manifest, I see that the image used in the pod is quay.io/strimzi/kafka:0.43.0-kafka-3.8.0 which maches the Strimzi Kafka Operator version.

Here is the manifest I'm using, and attached are pod logs

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: my-mirror-maker
spec:
  version: 3.8.0
  replicas: 1
  connectCluster: "default-cluster"
  clusters:
    - alias: "default-cluster"
      bootstrapServers: my-bootstrap-servers:8093,my-bootstrap-servers:8094,my-bootstrap-servers:8095
      tls:
        trustedCertificates:
          - secretName: bbt-mirror-maker
            certificate: crt.crt
      config:
        config.storage.replication.factor: 3
        offset.storage.replication.factor: 3
        status.storage.replication.factor: 3
        config.providers: secrets
        config.providers.secrets.class: org.apache.kafka.common.config.provider.DirectoryConfigProvider
        config.reload.action: restart
        ssl.cipher.suites: "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
        ssl.enabled.protocols: "TLSv1.2"
        ssl.protocol: "TLSv1.2"
        ssl.endpoint.identification.algorithm: HTTPS
  mirrors:
    - sourceCluster: "default-cluster"
      targetCluster: "default-cluster"
      sourceConnector:
        tasksMax: 10
        config:
          replication.factor: 3
          offset-syncs.topic.replication.factor: 3
          sync.topic.acls.enabled: "false"
          refresh.topics.interval.seconds: 60
          replication.policy.separator: ""
      topicsPattern: "test"
      groupsPattern: ""
  resources:
    requests:
      cpu: "1"
      memory: 2Gi
    limits:
      cpu: "2"
      memory: 2Gi
  logging:
    type: external
    valueFrom:
      configMapKeyRef:
        name: mirrormaker-logging
        key: log4j.properties
  readinessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
  livenessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
  jvmOptions:
    "-Xmx": "1g"
    "-Xms": "1g"
  template:
    connectContainer:
      env:
        - name: STRIMZI_IMAGE_PULL_POLICY
[streaming-store-topic-cloner-mirrormaker2-0.log](https://github.com/user-attachments/files/17525876/streaming-store-topic-cloner-mirrormaker2-0.log)

          value: Always

5 replies

scholzj Oct 25, 2024
Maintainer

So, how do you reproduce it? It does not reproduce for me with the config you shared.

scholzj Oct 25, 2024
Maintainer

Wait ... you have just one cluster and use it both as source and target. That does not make any sense. You need two clusters.

barrytam20 Oct 25, 2024
Author

Is it not possible to use MirrorMaker to mirror topics in the same cluster?

scholzj Oct 25, 2024
Maintainer

Well, let me put it differently ... I think this is the first time such an idea was raised by someone and we never designed / developed / tested it with this in mind. So I do not know if Mirror Maker 2 is capable of this. But from your error it seems it is not something Strimzi supports at this point.

barrytam20 Oct 27, 2024
Author

Ah got it. Thanks for looking into this for us :)

barrytam20 · 2024-10-28T13:21:30Z

barrytam20
Oct 28, 2024
Author

closing as this issues stems from the source and target cluster being the same, which is not something Strimzi currently supports

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

[Bug]: MirrorMaker2 KafkaAdminClient keystore password was incorrect #10760

{{title}}

Replies: 4 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Strimzi

[Bug]: MirrorMaker2 KafkaAdminClient keystore password was incorrect #10760

barrytam20 Oct 24, 2024

Bug Description

Steps to reproduce

Expected behavior

Strimzi version

Kubernetes version

Installation method

Infrastructure

Configuration files and logs

Additional context

Replies: 4 comments · 6 replies

scholzj Oct 24, 2024 Maintainer

barrytam20 Oct 24, 2024 Author

scholzj Oct 24, 2024 Maintainer

barrytam20 Oct 25, 2024 Author

scholzj Oct 25, 2024 Maintainer

scholzj Oct 25, 2024 Maintainer

barrytam20 Oct 25, 2024 Author

scholzj Oct 25, 2024 Maintainer

barrytam20 Oct 27, 2024 Author

barrytam20 Oct 28, 2024 Author

barrytam20
Oct 24, 2024

Replies: 4 comments 6 replies

scholzj
Oct 24, 2024
Maintainer

barrytam20
Oct 24, 2024
Author

scholzj Oct 24, 2024
Maintainer

barrytam20
Oct 25, 2024
Author

scholzj Oct 25, 2024
Maintainer

scholzj Oct 25, 2024
Maintainer

barrytam20 Oct 25, 2024
Author

scholzj Oct 25, 2024
Maintainer

barrytam20 Oct 27, 2024
Author

barrytam20
Oct 28, 2024
Author