[BUG][opensearch] How can I disable SSL and keep the authentication?

[campidelli-wcq]

Describe the bug
It is more like a lack of documentation I guess. It is a common scenario, I want to spin-up this Helm chart locally, for testing purposes and I don't want to deal with certificates. However, I wouldn't like to disable security completely, as I would like to test aspects related to users, roles, etc.

However, I can't find a precise documentation about this. I thought that there would be a flag like ssl.enabled=false for the opensearch chart, but there isn't.

The problem about using SSL is that when I run my Java code (Spring Boot) that is deployed in the same Kubernetes I get the following error:

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

And I don't want to implement any workaround in the Java side because in production this app actually is going to use valid certificates.

To Reproduce
Steps to reproduce the behavior:

1. Go to the documentation and search "how to disable SSL"
2. 😢

Expected behavior
A section describing how to do this in the docs.

Chart Name
opensearch

Screenshots
N/A

Host/Environment (please complete the following information):

• Helm Version: 2.12.2
• Kubernetes Version: v1.27.5+k3s1

Additional context
Just thank you for reading this and helping a distressed developer 😄

[smlx]

Take a look at the options here: https://opensearch.org/docs/2.9/security/configuration/tls/#rest-layer-tls

[campidelli-wcq]

Thanks @smlx , I have read this, and I tried something like this (by the way, I am using tilt here)

load('ext://helm_resource', 'helm_resource', 'helm_repo')

helm_repo(
    'opensearch-charts',
    'https://opensearch-project.github.io/helm-charts/',
    labels=['2-SUPPORT']
)

helm_resource(
    'opensearch',
    'opensearch-charts/opensearch',
    flags=['--version=2.12.2', '--set=singleNode=true,sysctlInit.enabled=true,plugins.security.ssl.http.enabled=false'],
    port_forwards=['9200:9200', '9300:9300', '9600:9600'],
    labels=['2-SUPPORT']
)

The initial arguments singleNode=true and sysctlInit.enabled=true work. The last does nothing though...

[campidelli-wcq]

I changed the approach to use a values.yaml file. I think I am getting there.

singleNode: true
sysctlInit:
  enabled: true
config:
  opensearch.yml: |
    plugins:
      security:
        ssl:
          transport:
            pemcert_filepath: esnode.pem
            pemkey_filepath: esnode-key.pem
            pemtrustedcas_filepath: root-ca.pem
            enforce_hostname_verification: false
          http:
            enabled: false
            pemcert_filepath: esnode.pem
            pemkey_filepath: esnode-key.pem
            pemtrustedcas_filepath: root-ca.pem
        allow_unsafe_democertificates: true
        allow_default_init_securityindex: true
        authcz:
          admin_dn:
            - CN=kirk,OU=client,O=client,L=test,C=de
        audit.type: internal_opensearch
        enable_snapshot_restore_privilege: true
        check_snapshot_restore_write_privileges: true
        restapi:
          roles_enabled: ["all_access", "security_rest_api_access"]
        system_indices:
          enabled: true
          indices:
            [
              ".opendistro-alerting-config",
              ".opendistro-alerting-alert*",
              ".opendistro-anomaly-results*",
              ".opendistro-anomaly-detector*",
              ".opendistro-anomaly-checkpoints",
              ".opendistro-anomaly-detection-state",
              ".opendistro-reports-*",
              ".opendistro-notifications-*",
              ".opendistro-notebooks",
              ".opendistro-asynchronous-search-response*",
            ]

Now if I hit http://localhost:9200, it asks for user and password master, which is the behaviour I want.

However, when I try to connect from my application using:

            final CredentialsProvider credentialsProvider =
                new BasicCredentialsProvider();
            credentialsProvider.setCredentials(AuthScope.ANY,
                new UsernamePasswordCredentials(username, password));

I get this error:

Caused by: org.opensearch.client.ResponseException: method [POST], host [http://localhost:9200], URI [/_aliases], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"},"status":403}
	... 12 common frames omitted

[campidelli-wcq]

And I am trying to use the other chart (opensearch-dashboards) with this configuration:

config:
  opensearch_dashboards.yml: |
    opensearch:
      hosts: ["http://opensearch-cluster-master:9200"]
      username: admin
      password: admin
      ssl:
        verificationMode: none
    server:
      ssl:
        enabled: false

But it fails:

{"type":"log","@timestamp":"2023-09-14T05:34:06Z","tags":["error","opensearch","data"],"pid":453,"message":"[ConnectionError]: write EPROTO 281473744556224:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:\n"}

[prudhvigodithi]

[Untriage]
Adding @tranngocsongtruc @SkollRyu @premkirank can you please take a look?

[tranngocsongtruc]

[Untriage] Adding @tranngocsongtruc @SkollRyu @premkirank can you please take a look?

Thank you for tagging me. May I ask if I can work on this issue? Thank you @prudhvigodithi

[prudhvigodithi]

Thanks @tranngocsongtruc I have assigned this issue to you.

[tranngocsongtruc]

Thanks @tranngocsongtruc I have assigned this issue to you.

Thank you!

[martinjt]

Did you ever manage to get a singlenode cluster working with the helm chart?

[random1st]

Guys, is it so big deal to fix it?

[woodsaj]

I managed to get this working, so thought i would share for anyone else that runs into this problem.

The problem here is due to how the DEMO_CONFIG is initialised for the security plugin. Unless the environment variable "DISABLE_INSTALL_DEMO_CONFIG" is set to 'true', the install_demo_configuration.sh script is run. This script looks at the opensearch.yaml config file for any lines that start with plugins.security and if none are found the demo configuration is applied, which includes defining config values in opensearch.yaml, creating the internal_users.yaml file (HTTP auth user using the OPENSEARCH_INITIAL_ADMIN_PASSWORD) and adding all the required TLS certs for both HTTP and Transport (communication between nodes).

The problem here is that the demo_config scripts seem to expect the opensearch.yml to have a flat structure, eg

######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true

However the helm chart encourages a hierarchal structure, eg

    plugins:
      security:
        ssl:
          transport:
            pemcert_filepath: esnode.pem
            pemkey_filepath: esnode-key.pem
            pemtrustedcas_filepath: root-ca.pem
            enforce_hostname_verification: false
          http:
            enabled: false
            pemcert_filepath: esnode.pem
            pemkey_filepath: esnode-key.pem
            pemtrustedcas_filepath: root-ca.pem
        allow_unsafe_democertificates: true

So if you configure the security settings via a hierarchal layout in the config file, the settings are overridden by the demo_config scripts which append their values to the end of the opensearch.yaml file.

Disabling the demo_config script from running is not an option as it is still needed for adding the demo certificates for TRANSPORT (ssl is required) and populating the internal_users.yml file.

The solution to all of this is to override the plugins.security.ssl.http.enabled via an environment variable which will take precedence over the value placed in opensearch.yml by the demo scripts. This can be done by simply setting the followning in your helm values

extraEnvs: 
  - name: plugins.security.ssl.http.enabled
    value: false

[ms-semarchy]

This workaround does not work for me with the latest version of the chart (2.27.1), I get the error:

StatefulSet in version "v1" cannot be handled as a StatefulSet: json: cannot unmarshal bool into Go struct field EnvVar.spec.template.spec.containers.env.value of type string
failed to create resource

[jack-sylvere]

This workaround does not work for me with the latest version of the chart (2.27.1), I get the error:

StatefulSet in version "v1" cannot be handled as a StatefulSet: json: cannot unmarshal bool into Go struct field EnvVar.spec.template.spec.containers.env.value of type string
failed to create resource

This works for me with latest chart version :

extraEnvs: 
  - name: plugins.security.ssl.http.enabled
    value: "false"

[ms-semarchy]

This works for me with latest chart version :

extraEnvs: 
  - name: plugins.security.ssl.http.enabled
    value: "false"

I confirm it works with the quotes around false.
Wonderful ✨

[prudhvigodithi]

Thanks everyone, closing this issue, please feel free to re-open or add comments if required.
@getsaurabh02