kafka-connect is failing with java.lang.NoClassDefFoundError: org/bouncycastle/jcajce/provider/BouncyCastleFipsProvider

[hanlin1412sp]

Hi Team,

I'm trying to setup a standalone snowflake-sink-connector with strimzi with minikube in my local machine. I've established kafak-cluster, and created a sample topic with some messages. I also have kafka-connect cluster in ready state, but when I kicked off snowflake-kafka-connector.yaml, I'm getting this error message in the kafka-connect pod.

javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: org/bouncycastle/jcajce/provider/BouncyCastleFipsProvider
       at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
       at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
       at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
       at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
       at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
       at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
       at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
       at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
       at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
       at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
       at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
       at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
       at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
       at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
       at org.eclipse.jetty.server.Server.handle(Server.java:516)
       at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
       at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
       at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
       at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
       at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
       at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
       at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
       at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
       at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
       at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
       at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: org/bouncycastle/jcajce/provider/BouncyCastleFipsProvider
       at org.glassfish.jersey.servlet.internal.ResponseWriter.rethrow(ResponseWriter.java:255)
       at org.glassfish.jersey.servlet.internal.ResponseWriter.failure(ResponseWriter.java:237)
       at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:445)
       at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:264)
       at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
       at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
       at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
       at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
       at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
       at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
       at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
       at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
       at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
       ... 35 more
Caused by: java.lang.NoClassDefFoundError: org/bouncycastle/jcajce/provider/BouncyCastleFipsProvider
       at com.snowflake.kafka.connector.internal.InternalUtils.createProperties(InternalUtils.java:167)
       at com.snowflake.kafka.connector.internal.SnowflakeConnectionServiceFactory$SnowflakeConnectionServiceBuilder.setProperties(SnowflakeConnectionServiceFactory.java:75)
       at com.snowflake.kafka.connector.SnowflakeSinkConnector.validate(SnowflakeSinkConnector.java:223)
       at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:509)
       at org.apache.kafka.connect.runtime.AbstractHerder.lambda$validateConnectorConfig$2(AbstractHerder.java:390)
       at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
       ... 1 more
Caused by: java.lang.ClassNotFoundException: org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
       at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
       at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
       at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
       ... 10 more
2023-09-11 18:11:29,263 INFO 10.244.0.3 - - [11/Sep/2023:18:11:29 +0000] "PUT /connectors/snowflake-connector/config HTTP/1.1" 500 0 "-" "-" 127 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp785240035-38]
2023-09-11 18:11:38,261 INFO 10.244.0.1 - - [11/Sep/2023:18:11:38 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.27" 17 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp785240035-54]
2023-09-11 18:11:48,255 INFO 10.244.0.1 - - [11/Sep/2023:18:11:48 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.27" 11 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp785240035-53]
2023-09-11 18:11:58,257 INFO 10.244.0.1 - - [11/Sep/2023:18:11:58 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.27" 11 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp785240035-55]
2023-09-11 18:12:05,210 INFO 10.244.0.3 - - [11/Sep/2023:18:12:05 +0000] "GET /connector-plugins HTTP/1.1" 200 405 "-" "-" 91 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp785240035-52]
2023-09-11 18:12:05,210 INFO 10.244.0.3 - - [11/Sep/2023:18:12:05 +0000] "GET /admin/loggers/ HTTP/1.1" 200 142 "-" "-" 89 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp785240035-33]
2023-09-11 18:12:05,210 INFO 10.244.0.3 - - [11/Sep/2023:18:12:05 +0000] "GET /connectors HTTP/1.1" 200 2 "-" "-" 91 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp785240035-35]
2023-09-11 18:12:05,486 INFO 10.244.0.3 - - [11/Sep/2023:18:12:05 +0000] "GET /connectors/snowflake-connector/config HTTP/1.1" 404 70 "-" "-" 87 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp785240035-38] ```


Here is my custom docker image used in my kafka-connect CRD:
``` FROM quay.io/strimzi/kafka:0.37.0-kafka-3.5.1-arm64
# quay.io/strimzi/kafka:0.37.0-kafka-3.5.1

USER root:root

RUN mkdir -p /opt/kafka/plugins/snowflake

# Add Snowflake connector jar
RUN curl -sSL "https://repo1.maven.org/maven2/com/snowflake/snowflake-kafka-connector/2.0.1/snowflake-kafka-connector-2.0.1.jar" \
   -o /opt/kafka/plugins/snowflake/snowflake-kafka-connector-2.0.1.jar
   
USER 1001 ``` 

Here is my kafka-connect CRD:
``` apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
 name: my-connect-cluster
 annotations:
 # use-connector-resources configures this KafkaConnect
 # to use KafkaConnector resources to avoid
 # needing to call the Connect REST API directly
   strimzi.io/use-connector-resources: "true"
spec:
 replicas: 1
 bootstrapServers: my-cluster-kafka-bootstrap:9092
 image: hlinsolo/sf-kafka-connect:v1.0
 template:
   pod:
     imagePullSecrets:
       - name: dockerpass
 config:
   config.storage.replication.factor: 1
   offset.storage.replication.factor: 1
   status.storage.replication.factor: 1
   config.providers: file
   config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
 logging:
   type: inline
   loggers:
     log4j.rootLogger: INFO
 readinessProbe:
   initialDelaySeconds: 1000
   timeoutSeconds: 1000
 livenessProbe:
   initialDelaySeconds: 1200
   timeoutSeconds: 1200
 externalConfiguration:
   volumes:
     - name: connector-config
       secret:
         secretName: snowflake-kafka-credentials

I'm trying to establish snowflake-sink-connector so, is the above config correct?
Here is my snowflake-kafka-connector.yaml:

kind: KafkaConnector
metadata:
  name: snowflake-connector
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: com.snowflake.kafka.connector.SnowflakeSinkConnector
  tasksMax: 1
  config:
    topics: "samples"
    snowflake.topic2table.map: "samples:SAMPLE_ARCHIVE"
    buffer.count.records: "10000"
    buffer.flush.time: "240"
    buffer.size.bytes: "2000000"
    snowflake.url.name: "sucb14399.prod3.us-west-2.aws.snowflakecomputing.com"
    snowflake.user.name: "${file:/opt/kafka/external-configuration/connector-config/snowflake-kafka-credentials.properties:snowflake_username}"
    snowflake.private.key: "${file:/opt/kafka/external-configuration/connector-config/snowflake-kafka-credentials.properties:snowflake_privatekey}"
    snowflake.private.key.passphrase: "${file:/opt/kafka/external-configuration/connector-config/snowflake-kafka-credentials.properties:snowflake_privatekey_passphrase}"
    snowflake.database.name: "${file:/opt/kafka/external-configuration/connector-config/snowflake-kafka-credentials.properties:snowflake_database_name}"
    snowflake.schema.name: "KAFKA_SNOWFLAKE"
    key.converter: "org.apache.kafka.connect.storage.StringConverter"
    value.converter: "org.apache.kafka.connect.storage.StringConverter"
    key.converter.schemas.enable: "false"
    value.converter.schemas.enable: "false"

Please let me know if you need any more info. I'm hitting my head against the wall trying to solve this issue for the last 3 days. Really appreciate your time if you could help me guide to the right path. Thank you.

[scholzj]

That looks like you are missing some dependencies. Are you sure that downloading the single JAR is sufficient? Aren't there any additional dependencies the connector has?

[scholzj]

bcpkix-fips-1.0.3-javadoc.jar sounds like a wrong JAR? That would contain only the docs and not the actual code. There needs to be something like bcpkix-fips-1.0.3.jar.

[hanlin1412sp]

After bringing in all the right bouncy castle jar files, I am seeing a new error.

2023-09-12 00:07:53,545 ERROR [SF_KAFKA_CONNECTOR] Validate: Error connecting to snowflake:[SF_KAFKA_CONNECTOR] Exception: Invalid encrypted private key or passphrase
Error Code: 0018
Detail: failed to decrypt private key. Please verify input private key and passphrase. Snowflake Kafka Connector only supports encryption algorithms in FIPS 140-2
Message: unable to decode base64 string: invalid characters encountered in base64 data
org.bouncycastle.util.encoders.Base64.decode(Unknown Source)
org.bouncycastle.util.io.pem.PemReader.loadObject(Unknown Source)
org.bouncycastle.util.io.pem.PemReader.readPemObject(Unknown Source)
org.bouncycastle.openssl.PEMParser.readObject(PEMParser.java:92)
com.snowflake.kafka.connector.internal.EncryptionUtils.parseEncryptedPrivateKey(EncryptionUtils.java:34)
com.snowflake.kafka.connector.internal.InternalUtils.createProperties(InternalUtils.java:167)
com.snowflake.kafka.connector.internal.SnowflakeConnectionServiceFactory$SnowflakeConnectionServiceBuilder.setProperties(SnowflakeConnectionServiceFactory.java:75)
com.snowflake.kafka.connector.SnowflakeSinkConnector.validate(SnowflakeSinkConnector.java:223)
org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:509)
org.apache.kafka.connect.runtime.AbstractHerder.lambda$validateConnectorConfig$2(AbstractHerder.java:390)
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base/java.lang.Thread.run(Thread.java:833), errorCode:0018 (com.snowflake.kafka.connector.SnowflakeSinkConnector) [pool-2-thread-1]
2023-09-12 00:07:53,599 INFO AbstractConfig values: 
 (org.apache.kafka.common.config.AbstractConfig) [pool-2-thread-1]
2023-09-12 00:07:53,751 INFO 10.244.0.3 - - [12/Sep/2023:00:07:36 +0000] "PUT /connectors/snowflake-connector/config HTTP/1.1" 400 300 "-" "-" 17321 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp1585347198-37]
2023-09-12 00:07:53,906 INFO 10.244.0.1 - - [12/Sep/2023:00:07:53 +0000] "GET / HTTP/1.1" 200 91 "-" "kube-probe/1.27" 109 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp1585347198-41]
2023-09-12 00:07:53,917 INFO 10.244.0.3 - - [12/Sep/2023:00:07:53 +0000] "GET /connectors/snowflake-connector/config HTTP/1.1" 404 70 "-" "-" 24 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp1585347198-58]
2023-09-12 00:07:54,208 ERROR [SF_KAFKA_CONNECTOR] Validate: Error connecting to snowflake:[SF_KAFKA_CONNECTOR] Exception: Invalid encrypted private key or passphrase
Error Code: 0018
Detail: failed to decrypt private key. Please verify input private key and passphrase. Snowflake Kafka Connector only supports encryption algorithms in FIPS 140-2
Message: unable to decode base64 string: invalid characters encountered in base64 data
org.bouncycastle.util.encoders.Base64.decode(Unknown Source)
org.bouncycastle.util.io.pem.PemReader.loadObject(Unknown Source)
org.bouncycastle.util.io.pem.PemReader.readPemObject(Unknown Source)
org.bouncycastle.openssl.PEMParser.readObject(PEMParser.java:92)
com.snowflake.kafka.connector.internal.EncryptionUtils.parseEncryptedPrivateKey(EncryptionUtils.java:34)
com.snowflake.kafka.connector.internal.InternalUtils.createProperties(InternalUtils.java:167)
com.snowflake.kafka.connector.internal.SnowflakeConnectionServiceFactory$SnowflakeConnectionServiceBuilder.setProperties(SnowflakeConnectionServiceFactory.java:75)
com.snowflake.kafka.connector.SnowflakeSinkConnector.validate(SnowflakeSinkConnector.java:223)
org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:509)
org.apache.kafka.connect.runtime.AbstractHerder.lambda$validateConnectorConfig$2(AbstractHerder.java:390)
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base/java.lang.Thread.run(Thread.java:833), errorCode:0018 (com.snowflake.kafka.connector.SnowflakeSinkConnector) [pool-2-thread-1]
2023-09-12 00:07:54,210 INFO AbstractConfig values: 
 (org.apache.kafka.common.config.AbstractConfig) [pool-2-thread-1]
2023-09-12 00:07:54,221 INFO 10.244.0.3 - - [12/Sep/2023:00:07:53 +0000] "PUT /connectors/snowflake-connector/config HTTP/1.1" 400 300 "-" "-" 234 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp1585347198-59]
2023-09-12 00:07:54,582 INFO 10.244.0.3 - - [12/Sep/2023:00:07:54 +0000] "GET /connectors HTTP/1.1" 200 2 "-" "-" 93 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp1585347198-60]
2023-09-12 00:07:54,599 INFO 10.244.0.3 - - [12/Sep/2023:00:07:54 +0000] "GET /admin/loggers/ HTTP/1.1" 200 142 "-" "-" 94 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp1585347198-39]
2023-09-12 00:07:54,602 INFO 10.244.0.3 - - [12/Sep/2023:00:07:54 +0000] "GET /connector-plugins HTTP/1.1" 200 405 "-" "-" 114 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp1585347198-44]
2023-09-12 00:07:54,684 INFO 10.244.0.3 - - [12/Sep/2023:00:07:54 +0000] "GET /connectors/snowflake-connector/config HTTP/1.1" 404 70 "-" "-" 75 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp1585347198-40]

I even recreated my private/public keys according to this doc https://docs.snowflake.com/en/user-guide/key-pair-auth and still get the same error. I also removed line breaks from my private/public keys as they are in PEM format, that is recommended by the snowflake.

snowflake.private.key

    The private key to authenticate the user. Include only the key, not the header or footer. If the key is split across multiple lines, remove the line breaks. 
You can provide an unencrypted key, or you can provide an encrypted key and provide the snowflake.private.key.passphrase parameter to enable Snowflake to decrypt the key. 
Use this parameter if and only if the snowflake.private.key parameter value is encrypted. This decrypts private keys that were encrypted according to the instructions in [Using Key Pair Authentication & Key Rotation](https://docs.snowflake.com/en/user-guide/kafka-connector-install#using-key-pair-authentication-key-rotation) (in this topic).

I'm going to sleep on this and find other ways. Would like to hear any suggestions. Thank you.

[scholzj]

I think that is a question on Snowflake.

[hanlin1412sp]

I actually got it working after tweaking around my k8s secret values for my private key. I had to remove the line-breakers and double quotes in my secret.

Really appreciate your quick responses and suggestions.

[abhishekkh]

I am running into the decryption error as well. What was the tweak? The secret in the mounted file (opt/kafka/external-configuration/connector-config/snowflake-kafka-credentials.properties:snowflake_privatekey) has a header, footer and line breaks.