pandaproxy_client and schemaregistry_client should honnor trustStore settings

[fboismenu]

What happened?

Configuring mTLS with Kubernetes Secrets, following this guide: https://docs.redpanda.com/current/manage/kubernetes/security/tls/k-secrets/

We are using an internal PKI ultimately used via cert-manager generating the secrets used in the setup.
Our PKI returns the certificate chains in ca.crt up to last intermediate signing authority.

The root certificate of our company needs provided in the trustStore configuration (https://docs.redpanda.com/current/manage/kubernetes/security/tls/k-secrets/#configure-a-truststore)

Then, we intend to enable mTLS based authorisation using a specific kafka_mtls_principal_mapping_rules: https://docs.redpanda.com/current/manage/security/authentication/#mtls

This requires us to be in full control of the "admin" client cert (using the same PKI), therefore we inject it via the undocumented (!) usage of .Values.tls.certs.clientSecretRef, which is also handed over to the console.

Resulting in following values.yaml (only relevant -- hopefully) parts included:

auth:
  sasl:
    enabled: false

external:
  enabled: true
  type: LoadBalancer
  domain: <our domain>
  addresses:
    - redpanda-0
    - redpanda-1
    - redpanda-2
  externalDns:
    enabled: true
  service:
    enabled: true

tls:
  enabled: true
  certs:
    default:
      caEnabled: true
      secretRef:
        name: redpanda-tls-cert
      clientSecretRef:
        name: redpanda-admin-cert

listeners:
  admin:
    port: 9643
    tls:
      enabled: true
      cert: default
      requireClientAuth: true
      trustStore:
        configMapKeyRef:
          name: redpanda-company-cacrt
          key: ca.crt
    external:
      default:
        port: 9644
        tls:
          enabled: true
          cert: default
          requireClientAuth: true
          trustStore:
            configMapKeyRef:
              name: redpanda-company-cacrt
              key: ca.crt
        advertisedPorts:
          - 9644
  kafka:
    port: 9093
    tls:
      enabled: true
      cert: default
      requireClientAuth: true
      trustStore:
        configMapKeyRef:
          name: redpanda-company-cacrt
          key: ca.crt
    external:
      default:
        port: 9094
        tls:
          enabled: true
          cert: default
          requireClientAuth: true
          trustStore:
            configMapKeyRef:
              name: redpanda-company-cacrt
              key: ca.crt
        advertisedPorts:
          - 9094
  http:
    enabled: true
    port: 8082
    kafkaEndpoint: default
    tls:
      enabled: true
      cert: default
      requireClientAuth: true
      trustStore:
        configMapKeyRef:
          name: redpanda-company-cacrt
          key: ca.crt
    external:
      default:
        port: 8083
        tls:
          enabled: true
          cert: default
          requireClientAuth: true
          trustStore:
            configMapKeyRef:
              name: redpanda-company-cacrt
              key: ca.crt
        advertisedPorts:
          - 8083
  rpc:
    port: 33145
    tls:
      enabled: true
      cert: default
      requireClientAuth: true
      trustStore:
        configMapKeyRef:
          name: redpanda-company-cacrt
          key: ca.crt
schemaRegistry:
    enabled: true
    port: 8081
    kafkaEndpoint: default
    tls:
      enabled: true
      cert: default
      requireClientAuth: true
      trustStore:
        configMapKeyRef:
          name: redpanda-company-cacrt
          key: ca.crt
    external:
      default:
        port: 8084
        tls:
          enabled: true
          cert: default
          requireClientAuth: true
          trustStore:
            configMapKeyRef:
              name: redpanda-company-cacrt
              key: ca.crt
        advertisedPorts:
          - 8084

console:
  enabled: true
  analytics:
    enabled: false
  image:
    tag: v2.7.0
  config:
    kafka:
      tls:
        enabled: true
        caFilepath: /mnt/cert/admin/ca.crt
        certFilepath: /mnt/cert/admin/tls.crt
        keyFilepath: /mnt/cert/admin/tls.key
        insecureSkipTlsVerify: false
      schemaRegistry:
        tls:
          enabled: true
          caFilepath: /mnt/cert/admin/ca.crt
          certFilepath: /mnt/cert/admin/tls.crt
          keyFilepath: /mnt/cert/admin/tls.key
          insecureSkipTlsVerify: false
    redpanda:
      adminApi:
        tls:
          enabled: true
          caFilepath: /mnt/cert/admin/ca.crt
          certFilepath: /mnt/cert/admin/tls.crt
          keyFilepath: /mnt/cert/admin/tls.key
          insecureSkipTlsVerify: false
  secretMounts:
    - name: tls-admin-certs
      secretName: redpanda-admin-cert
      path: /mnt/cert/admin
      defaultMode: 272

Resulting into a running cluster and functional cluster:

$ kcat -b redpanda-2.<our domain>:9094 -X security.protocol=SSL -X ssl.ca.location=ca.crt -X ssl.key.location=tls.key -X ssl.certificate.location=tls.crt -C -t fred-test
bonjour
hello
toot
% Reached end of topic fred-test [0] at offset 3
^C%

but the pandaproxy and schemaregistry are not functional:
Logs in RedPanda:

INFO  2024-10-17 19:20:08,444 [shard 0:admi] kafka/client - broker.cc:69 - schema_registry_client: connected to broker:-1 - redpanda-2.redpanda.platform-redpanda-test.svc.cluster.local.:9093
ERROR 2024-10-17 19:20:08,449 [shard 0:admi] pandaproxy - service.cc:408 - Schema registry failed to initialize: kafka::client::broker_error ({ node: -1 }, { error_code: broker_not_available [8] })
WARN  2024-10-17 19:20:08,449 [shard 0:admi] pandaproxy - server.cc:125 - Request: GET /subjects failed: kafka::client::broker_error ({ node: -1 }, { error_code: broker_not_available [8] })                       WARN  2024-10-17 19:21:09,261 [shard 0:admi] pandaproxy - service.cc:468 - mitigate_error: kafka::client::broker_error ({ node: -1 }, { error_code: broker_not_available [8] })
WARN  2024-10-17 19:21:09,261 [shard 0:admi] kafka/client - client.cc:167 - schema_registry_client: broker_error: kafka::client::broker_error ({ node: -1 }, { error_code: broker_not_available [8] })
INFO  2024-10-17 19:21:09,264 [shard 0:admi] kafka/client - broker.cc:69 - schema_registry_client: connected to broker:-1 - redpanda-0.redpanda.platform-redpanda-test.svc.cluster.local.:9093
INFO  2024-10-17 19:21:09,270 [shard 2:main] kafka - server.cc:167 - Disconnected XXX.XXX.XXX.XXX:49176 (applying protocol, Broken pipe)
WARN  2024-10-17 19:21:09,468 [shard 0:admi] pandaproxy - service.cc:468 - mitigate_error: seastar::tls::verification_error (unable to get issuer certificate (Issuer=["CN=Root-CA"], Subject=["
CN=Intermediate-CA"]))

Looking at redpanda.yaml :

pandaproxy_client:
    brokers:
        - address: redpanda-0.redpanda.platform-redpanda-test.svc.cluster.local.
          port: 9093
        - address: redpanda-1.redpanda.platform-redpanda-test.svc.cluster.local.
          port: 9093
        - address: redpanda-2.redpanda.platform-redpanda-test.svc.cluster.local.
          port: 9093
    broker_tls:
        key_file: /etc/tls/certs/redpanda-client/tls.key
        cert_file: /etc/tls/certs/redpanda-client/tls.crt
        truststore_file: /etc/tls/certs/default/ca.crt
        enabled: true
        require_client_auth: true

Schema_registry_client looks the same.
We can see the truststore_file points to the tls.default ca, but should actually support the trustStore settings.

Console pod is not starting properly because pandaproxy and schemaregistries are non-functional because they cannot connect the internal cluster api.

What did you expect to happen?

A properly configured pandaproxy and schemaregistry able to authenticate against the cluster api, and validate with its certificated with a provided truststore file (as a Kubernetes secrets) just like for the listeners.

How can we reproduce it (as minimally and precisely as possible)?. Please include values file.

See above, relevant values.yaml provided

Anything else we need to know?

https://github.com/redpanda-data/helm-charts/blob/main/charts/redpanda/configmap.tpl.go#L408

should probably read:

"truststore_file": kafkaTLS.TrustStoreFilePath(&values.TLS),

Which are the affected charts?

No response

Chart Version(s)

$ helm -n <redpanda-release-namespace> list 
NAME            NAMESPACE               REVISION        UPDATED                                 STATUS          CHART           APP VERSION
redpanda        platform-redpanda-test  26              2024-10-18 22:24:53.084061583 +0000 UTC deployed        redpanda-5.9.6  v24.2.5

Cloud provider

None, on-prem setup.

JIRA Link: K8S-395