KafkaProducer with mtls shows error, failed authentication due to: Failed to process post-handshake messages, javax.crypto.AEADBadTagException: Tag mismatch! at

[seemasanjaisinghani]

I have enabled mtls in my setup using our own certificates for Client CA.
The kafka cluster came up fine.
But when KafKa producer is connecting to it, getting below exceptions:

2021-12-10T16:14:42.506558285Zjavax.net.ssl|ERROR|0C|kafka-producer-network-thread | console-producer|2021-12-10 16:14:42.506 GMT|TransportContext.java:341|Fatal (BAD_RECORD_MAC): Tag mismatch! (

2021-12-10T16:14:42.507284992Zorg.apache.kafka.common.errors.SslAuthenticationException: Failed to process post-handshake messages Caused by: javax.net.ssl.SSLException: Tag mismatch! at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:123) at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681) at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636) at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454) at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433) at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637) at org.apache.kafka.common.network.SslTransportLayer.read(SslTransportLayer.java:567) at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:95) at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452) at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402) at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576) at org.apache.kafka.common.network.Selector.poll(Selector.java:481) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:327) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:242) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: javax.crypto.AEADBadTagException: Tag mismatch! at java.base/com.sun.crypto.provider.GaloisCounterMode.decryptFinal(GaloisCounterMode.java:623) at java.base/com.sun.crypto.provider.CipherCore.finalNoPadding(CipherCore.java:1116) at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1053) at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:941) at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:491) at java.base/javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:779) at java.base/javax.crypto.CipherSpi.engineDoFinal(CipherSpi.java:730) at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2497) at java.base/sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1903) at java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240) at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197) at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111) ... 16 more

Any help appriciated!

[seemasanjaisinghani]

Any input from anyone?

[seemasanjaisinghani]

@scholzj any heads up for this issue?

[scholzj]

I don't kniw, you did not shared much information. It is probably some misconfiguration?

[seemasanjaisinghani]

@scholzj here is the setup done :

NOTE : Cluster CA is default strimzi one and Client CA secret is created from our own Certificates.

1. setup kafka cluster with -
   authorization:
   type: simple
   superUsers:
   - CN=acl-test-user
   listeners:
   • name: plain
     port: 9092
     type: internal
     tls: false
   • name: tls
     port: 9093
     type: internal
     tls: true
     authentication:
     type: tls
   • name: external
     port: 9094
     type: loadbalancer
     tls: true
     authentication:
     type: tls
     configuration:
     loadBalancerSourceRanges:
     - {{ .Values.kafka_listeners_external_load_balancer_source_ranges }}
     bootstrap:
     annotations:
     cloud.google.com/load-balancer-type: "Internal"
     brokers:
     • broker: 0
       annotations:
       cloud.google.com/load-balancer-type: "Internal"
     • broker: 1
       annotations:
       cloud.google.com/load-balancer-type: "Internal"
     • broker: 2
       annotations:
       cloud.google.com/load-balancer-type: "Internal"
2. create kafka user

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaUser
metadata:
name: acl-test-user
labels:
strimzi.io/cluster: test-cluster
spec:
authentication:
type: tls
authorization:
type: simple
acls:
# Consumer Acls for topic acl-topic
- resource:
type: topic
name: acl-topic
patternType: literal
operation: Read
host: ""
- resource:
type: topic
name: acl-topic
patternType: literal
operation: Describe
host: ""
# Producer Acls for topic acl-topic
- resource:
type: topic
name: acl-topic
patternType: literal
operation: Write
host: ""
- resource:
type: topic
name: acl-topic
patternType: literal
operation: Create
host: ""
- resource:
type: topic
name: acl-topic
patternType: literal
operation: Describe
host: "*"

3. start kafka client producer and consumer -

./kafka-console-producer.sh --broker-list test-cluster-kafka-bootstra:9093 --topic acl-topic --producer-property security.protocol=SSL --producer-property ssl.truststore.location=ca.p12 --producer-property ssl.truststore.type=PKCS12 --producer-property ssl.truststore.password=STOREPASSW0RD --producer-property ssl.keystore.location=producer.keystore.p12 --producer-property ssl.keystore.password=123456 --producer-property ssl.keystore.type=PKCS12

After starting Kafka producer it shows that error

2021-12-10T16:14:42.506558285Zjavax.net.ssl|ERROR|0C|kafka-producer-network-thread | console-producer|2021-12-10 16:14:42.506 GMT|TransportContext.java:341|Fatal (BAD_RECORD_MAC): Tag mismatch! (

2021-12-10T16:14:42.507284992Zorg.apache.kafka.common.errors.SslAuthenticationException: Failed to process post-handshake messages Caused by: javax.net.ssl.SSLException: Tag mismatch! at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:123) at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681) at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636) at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454) at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433) at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637) at

[seemasanjaisinghani]

@scholzj any help please

[scholzj]

Can you please properly format the YAMLs and logs as code? Without that, nobody can read your configuration and see whether it is correct or not.