Exception during establishing a SSL connection

[ibrahimjelliti]

Describe the bug
I deploy Opensearch on EKS with helm chart with SSL configs, I can access the opensearch endpoint and Dashboard via HTTPS ,
in cluster logs there is a recurrent error log for Exception during establishing a SSL connection.
this is my opensearch.yaml config.

  ######## Start OpenSearch Security Demo Configuration ########
  # WARNING: revise all the lines below before you go into production
  plugins:
    security:
      ssl:
        transport:
          pemcert_filepath: certs/tls.crt
          pemkey_filepath: certs/tls.key
          pemtrustedcas_filepath: certs/tls.crt
          enforce_hostname_verification: false
        http:
          enabled: true
          pemcert_filepath: certs/tls.crt
          pemkey_filepath: certs/tls.key
          pemtrustedcas_filepath: certs/tls.crt
      allow_unsafe_democertificates: false
      allow_default_init_securityindex: true

{"type": "server", "timestamp": "2023-07-29T06:28:16,896Z", "level": "ERROR", "component": "o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport", "cluster.name": "opensearch-cluster", "node.name": "opensearch-cluster-master-0", "message": "Exception during establishing a SSL connection: java.net.SocketException: Connection reset", "cluster.uuid": "ybe8lbVIQfqVOuvUcFXCKA", "node.id": "sN04VXeURROEG9pLhKos3g" ,
"stacktrace": ["java.net.SocketException: Connection reset",
"at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]",
"at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]",
"at org.opensearch.transport.CopyBytesSocketChannel.readFromSocketChannel(CopyBytesSocketChannel.java:155) ~[transport-netty4-client-2.8.0.jar:2.8.0]",
"at org.opensearch.transport.CopyBytesSocketChannel.doReadBytes(CopyBytesSocketChannel.java:140) ~[transport-netty4-client-2.8.0.jar:2.8.0]",
"at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
"at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.91.Final.jar:4.1.91.Final]",
"at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.91.Final.jar:4.1.91.Final]",
"at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.91.Final.jar:4.1.91.Final]",
"at java.lang.Thread.run(Thread.java:833) [?:?]"] }

any how to further debug this log and resolve this issue?

[rishabh6788]

The issue seems to be related to security plugin than helm/EKS.
Can you please also add your cluster configuration, like OpenSearch version?

Tagging @opensearch-project/security-non-maintainers to advise on the above mentioned issue.

[ibrahimjelliti]

@rishabh6788 I am using version 2.8, I believe its more related to Security plugin as I added running Opensearch with Lets Encrypt certs. this is my opensearch.yaml

 opensearch.yml: |
   cluster.name: opensearch-cluster

   # Bind to all interfaces because we don't know what IP address Docker will assign to us.
   network.host: 0.0.0.0

   # # minimum_master_nodes need to be explicitly set when bound on a public IP
   # # set to 1 to allow single node clusters
   # discovery.zen.minimum_master_nodes: 1

   # Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
   #discovery.type: single-node

   # discovery.seed_hosts: [ 'opensearch-cluster-master-0', 'opensearch-cluster-master-1', 'opensearch-cluster-master-2' ]
   # cluster.initial_master_nodes: [ 'opensearch-cluster-master-0', 'opensearch-cluster-master-1', 'opensearch-cluster-master-2' ]

   ######## Start OpenSearch Security Demo Configuration ########
   # WARNING: revise all the lines below before you go into production
   plugins:
     security:
       nodes_dn:
          - 'CN=os.dev.example.com'
       ssl:
         transport:
           pemcert_filepath: certs/tls.crt
           pemkey_filepath: certs/tls.key
           pemtrustedcas_filepath: certs/tls.crt
           enforce_hostname_verification: false
         http:
           enabled: true
           pemcert_filepath: certs/tls.crt
           pemkey_filepath: certs/tls.key
           pemtrustedcas_filepath: certs/tls.crt
       allow_unsafe_democertificates: false
       allow_default_init_securityindex: true
       roles_mapping_resolution: BOTH
       authcz:
         admin_dn:
           - CN=kirk,OU=client,O=client,L=test, C=de
       
       enable_snapshot_restore_privilege: true
       check_snapshot_restore_write_privileges: true
       restapi:
         roles_enabled: ["all_access", "security_rest_api_access"]
       system_indices:
         enabled: true
         indices:
           [
             ".opendistro-alerting-config",
             ".opendistro-alerting-alert*",
             ".opendistro-anomaly-results*",
             ".opendistro-anomaly-detector*",
             ".opendistro-anomaly-checkpoints",
             ".opendistro-anomaly-detection-state",
             ".opendistro-reports-*",
             ".opendistro-notifications-*",
             ".opendistro-notebooks",
             ".opendistro-asynchronous-search-response*",
           ]
       compliance:
         salt: bYuCRu89h7fnmsT5eo6DEsQ57cn734ycUpBw
   ######## End OpenSearch Security Demo Configuration ########

   ####### S3 SETTINGS #######
   s3.client.default.endpoint: s3.amazonaws.com # S3 has alternate endpoints, but you probably don't need to change this value.
   s3.client.default.max_retries: 3 # number of retries if a request fails
   s3.client.default.path_style_access: false # whether to use the deprecated path-style bucket URLs.
   # You probably don't need to change this value, but for more information, see https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html#path-style-access.
   s3.client.default.protocol: https # http or https
   s3.client.default.read_timeout: 60s # the S3 connection timeout
   s3.client.default.use_throttle_retries: true # whether the client should wait a progressively longer amount of time (exponential backoff) between each successive retry
   s3.client.default.region: us-east-1 # AWS region to use
   s3.client.default.identity_token_file: aws-web-identity-token-file
   ###### END S3 SETTINGS #####
 

[dzanto]

I have same issue.
Cluster is green, but one node has repeated errorss.

[2023-08-09T16:12:01,601][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [elastic2] Exception during establishing a SSL connection: 
...
[2023-08-09T16:12:21,603][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [elastic2] Exception during establishing a SSL connection:
...
[2023-08-09T16:16:11,604][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [elastic2] Exception during establishing a SSL connection:
...
[2023-08-09T16:18:41,606][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [elastic2] Exception during establishing a SSL connection: java.net.SocketException: Connection reset
java.net.SocketException: Connection reset
	at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
	at org.opensearch.transport.CopyBytesSocketChannel.readFromSocketChannel(CopyBytesSocketChannel.java:155) ~[transport-netty4-client-2.7.0.jar:2.7.0]
	at org.opensearch.transport.CopyBytesSocketChannel.doReadBytes(CopyBytesSocketChannel.java:140) ~[transport-netty4-client-2.7.0.jar:2.7.0]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.91.Final.jar:4.1.91.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]

---

curl https://opensearch-1.cloud.ex.com:9200 --key admin.key --cert admin.pem --cacert root-ca.pem 
{
  "name" : "opensearch-1",
  "cluster_name" : "prod-cluster",
  "cluster_uuid" : "*************",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.7.0",
    "build_type" : "rpm",
    "build_hash" : "b7a6e09e492b1e965d827525f7863b366ef0e304",
    "build_date" : "2023-04-27T21:43:23.577631441Z",
    "build_snapshot" : false,
    "lucene_version" : "9.5.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}