Predictable k8s loadbalancer service config for new brokers in nodepool

[mumlawski]

Hi guys,

Looking for some guidance on how to approach this problem we have.

Basically at the moment we are running 3 brokers , 1 in each AWS AZ.

Kafka CR:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka  
  namespace: kafka  
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  entityOperator:
    topicOperator: {}
    userOperator: {}
  cruiseControl:
    config:
      webserver.http.cors.enabled: true
      webserver.http.cors.origin: "*"
      webserver.http.cors.exposeheaders: "User-Task-ID,Content-Type"
      webserver.security.enable: false
    resources:
      requests:
        cpu: 500m
        memory: 2Gi
    logging:
      type: inline
      loggers:
        rootLogger.level: INFO
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
  kafkaExporter:
    groupRegex: .*
    logging: debug
    template:
      pod:
        metadata:
          annotations:
            prometheus.io/port: "9404"
            prometheus.io/scrape: "true"
    topicRegex: .*
  kafka:
    authorization:
      type: simple
    version: 3.7.0
    metadataVersion: 3.7.0
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yml
          name: kafka-metrics
    config:
      num.partitions: 3
      default.replication.factor: 3
      min.insync.replicas: 2
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
      log.retention.ms: 604800000
      log.retention.bytes: 5368709120
    listeners:
    - name: internaltls
      port: 9092
      type: internal
      tls: true
      authentication:
        type: tls
    - name: externaltls
      port: 9093
      type: loadbalancer
      tls: true
      authentication:
        type: tls
      configuration:
        bootstrap:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-internal: 10.0.0.0/8
            external-dns.alpha.kubernetes.io/hostname:  kafka-bootstrap.internal.domain.
            external-dns.alpha.kubernetes.io/ttl: "60"
          alternativeNames:
          - kafka-bootstrap.internal.domain
        brokers:
        - broker: 0
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-internal: 10.0.0.0/8
            external-dns.alpha.kubernetes.io/hostname: kafka-node-0.internal.domain.
            external-dns.alpha.kubernetes.io/ttl: "60"
          advertisedHost: kafka-node-0.internal.domain
        - broker: 1
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-internal: 10.0.0.0/8
            external-dns.alpha.kubernetes.io/hostname: kafka-node-1.internal.domain.
            external-dns.alpha.kubernetes.io/ttl: "60"
          advertisedHost: kafka-node-1.internal.domain.
        - broker: 2
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-internal: 10.0.0.0/8
            external-dns.alpha.kubernetes.io/hostname: kafka-node-2.internal.domain.
            external-dns.alpha.kubernetes.io/ttl: "60"
          advertisedHost: kafka-node-2.internal.domain.

Using advertisedHost and externalDNS to have predicable DNS names of LBs and not to have SSL issues.

KafkaNodePool CR for one of the zones (a) : (Config is pretty much identical between NodePools, only changing zone)

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: controller-broker-zone-a
  namespace: kafka
  labels:
    strimzi.io/cluster: kafka
spec:
  replicas: 1
  roles:
  - controller
  - broker
  storage:
    type: jbod
    volumes:
    - id: 0
      type: persistent-claim
      size: 1000Gi
      kraftMetadata: shared
      deleteClaim: false
  template:
    pod:
      metadata:
        annotations:
          prometheus.io/port: "9404"
          prometheus.io/scrape: "true"
      tolerations:
      - key: node-role.kubernetes.io/kafka
        effect: NoSchedule
      - key: kubernetes.io/arch
        operator: Equal
        value: arm64
        effect: NoSchedule
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: strimzi.io/component-type
                operator: In
                values:
                - kafka
              - key: app.kubernetes.io/instance
                operator: In
                values:
                - kafka
            topologyKey: kubernetes.io/hostname
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node-role.kubernetes.io/kafka
                operator: Exists
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                - ${CLUSTER_REGION}a

What we have noticed is that if we scale the kafkanodepool to 2 replicas, next broker (nr 3) is being deployed in public subnet (instead of private as brokers from Kafka CR) without any additional annotations which we need.

What's the best way to have additional config for each extra broker in each AZ/NodePool for scaling purposes so we follow similar design for ExternalDNS records and not have SSL issues?

I was having a look at kafkanodepool.spec.template.perPodService but this would apply to all of the brokers I understand or only to new brokers?

Thanks for any help

[scholzj]

In the listener configuration, you have a configuration per-broker. With the broker: 0, broker: 1, etc. So if you add more brokers, you should just make sure it is there for the new broker(s) as well.

[scholzj]

On more note on your configuration ... The annotation shared for all loadbalancers ... service.beta.kubernetes.io/aws-load-balancer-internal ... you should be able to configure that one in the template section as well. And the per-broker DNS names ... external-dns.alpha.kubernetes.io/hostname: kafka-node-0.internal.domain. ... the question is whether you really care about the DNS names of the brokers. Normally, it is the bootstrap DNS name that you configure in the clients, so you want to have that nice, easy to remember and stable. But the broker addresses are auto-discovered by the clients. So normally, you don't really care if they use your DNS or the load balancer address given to it by AWS. So if needed, you might be able to remove those and then you do not need to add the records for the new brokers.

[mumlawski]

Based on my testing if I remove the the broker config with advertisedHost I'm starting to get SSL handshake errors. My understanding from my testing is that after initial bootstrap connection Kafka will reply with available brokers to connect to through advertisedHosts field. Is this expected?

Kafka CR stripped:

      configuration:
        bootstrap:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-internal: 10.0.0.0/8
            external-dns.alpha.kubernetes.io/hostname:  kafka-bootstrap.internal.domain.
          alternativeNames:
          - kafka-bootstrap.internal.domain
        brokers:
        - broker: 0
        - broker: 1
        - broker: 2

And NodePool as suggested:

  template:
    perPodService:
      metadata:
        annotations: 
          service.beta.kubernetes.io/aws-load-balancer-internal: 10.0.0.0/8
    pod:
      metadata:
        annotations:
          prometheus.io/port: "9404"
          prometheus.io/scrape: "true"

[scholzj]

Based on my testing if I remove the the broker config with advertisedHost I'm starting to get SSL handshake errors.

That should not be the case. If you don't override the advertised hosts, it will use the load balancer address that will be added to the server certificates. None of that should cause any TLS errors.

My understanding from my testing is that after initial bootstrap connection Kafka will reply with available brokers to connect to through advertisedHosts field.

Yes, this is correct.

---

The config you shared seems fine. You can just remove this part as it doesn't really do anything anymore:

        brokers:
        - broker: 0
        - broker: 1
        - broker: 2

[mumlawski]

Thanks for your help, I did refactor my config and retested and all is working fine. This allows us to scale brokers easily with predictable config on internal subnets which was important to us. 👍