[Bug]: Kafka Brokers metrics endpoint unresponsive on port 9094 #10134

marcogenualdo · 2024-05-20T03:26:20Z

marcogenualdo
May 20, 2024

Bug Description

I've installed the strimzi operator on an Azure Kubernetes cluster, and deployed a kafka instance from the metrics example.

When trying to reach the metrics endpoint from one pod inside the cluster, the brokers don't answer. Here's an example using the IP of broker 0. Trying telnet on port 9092 first, works fine, then the metrics port 9094, broken, then an http call to the metrics endpoint, also broken.

root@client:/# curl -v telnet://10.224.0.81:9092
*   Trying 10.224.0.81:9092...
* Connected to 10.224.0.81 (10.224.0.81) port 9092
^C
root@client:/# curl -v telnet://10.224.0.81:9094
*   Trying 10.224.0.81:9094...
^C
curl -v http://10.224.0.81:9094/metrics
*   Trying 10.224.0.81:9094...
^C

Everything is fine with zookeeper pods, answering the http://:9404/metrics call fine.

Steps to reproduce

No response

Expected behavior

No response

Strimzi version

3.7.0

Kubernetes version

1.27.9

Installation method

No response

Infrastructure

No response

Configuration files and logs

No response

Additional context

No response

scholzj · 2024-05-20T08:00:34Z

scholzj
May 20, 2024
Maintainer

Without knowing all the details, it is hard to really say what the cause it.

The Prometheus JMX Exporter has performance issues with big clusters. If you have a big cluster with many topics and consumers etc., you might need to customize the metrics configuration to expose less of them. There is some work going on to avoid relying on JMX Exporter. But it needs some improvements in Kafka itself and will take some time.
But this can be also caused by other things such as your brokers not having enough resources etc.

3 replies

marcogenualdo May 20, 2024
Author

Thank you for your answer. As I deployed this kafka cluster just to showcase this issue, the brokers sit unused. I also did not touch the configuration from the example you guys provided. So it can't be a resource/utilization issue.
I also deployed the service in two separate Azure Kubernetes clusters, having the same issue.
What kind of details would you need to debug the cause of the issue?

scholzj May 20, 2024
Maintainer

So it can't be a resource/utilization issue.

Why would it not be resource issue? If you haven't added any resource requests corresponding to your environment and use-case, it could be exactly that.

What kind of details would you need to debug the cause of the issue?

For example all configurations, Strimzi version, full logs from the operator, full logs from the Kafka brokers etc.

marcogenualdo May 20, 2024
Author

I've just given the brokers 2Gi of memory and 100m CPU requests. kubectl top pods shows each broker consuming about 380Mi and 23m. I've got the same issue.
As for the info you need. The strimzi version is 3.7.0
Broker image quay.io/strimzi/kafka:0.41.0-kafka-3.7.0
Here are the full logs from the broker-0 and the operator, respectively.

broker.log
op.log

And here's the full config:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: default
spec:
  kafka:
    version: 3.7.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.7"
    resources:
      limits:
        cpu: 1000m
        memory: 2Gi
      requests:
        cpu: 100m
        memory: 2Gi
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi
        deleteClaim: false
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: false
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yml
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kafka-metrics
  namespace: default
  labels:
    app: strimzi
data:
  kafka-metrics-config.yml: |
    # See https://github.com/prometheus/jmx_exporter for more info about JMX Prometheus Exporter metrics
    lowercaseOutputName: true
    rules:
    # Special cases and very specific rules
    - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
      name: kafka_server_$1_$2
      type: GAUGE
      labels:
        clientId: "$3"
        topic: "$4"
        partition: "$5"
    - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
      name: kafka_server_$1_$2
      type: GAUGE
      labels:
        clientId: "$3"
        broker: "$4:$5"
    - pattern: kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections
      name: kafka_server_$1_connections_tls_info
      type: GAUGE
      labels:
        cipher: "$2"
        protocol: "$3"
        listener: "$4"
        networkProcessor: "$5"
    - pattern: kafka.server<type=(.+), clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>connections
      name: kafka_server_$1_connections_software
      type: GAUGE
      labels:
        clientSoftwareName: "$2"
        clientSoftwareVersion: "$3"
        listener: "$4"
        networkProcessor: "$5"
    - pattern: "kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+):"
      name: kafka_server_$1_$4
      type: GAUGE
      labels:
        listener: "$2"
        networkProcessor: "$3"
    - pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+)
      name: kafka_server_$1_$4
      type: GAUGE
      labels:
        listener: "$2"
        networkProcessor: "$3"
    # Some percent metrics use MeanRate attribute
    # Ex) kafka.server<type=(KafkaRequestHandlerPool), name=(RequestHandlerAvgIdlePercent)><>MeanRate
    - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>MeanRate
      name: kafka_$1_$2_$3_percent
      type: GAUGE
    # Generic gauges for percents
    - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>Value
      name: kafka_$1_$2_$3_percent
      type: GAUGE
    - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*, (.+)=(.+)><>Value
      name: kafka_$1_$2_$3_percent
      type: GAUGE
      labels:
        "$4": "$5"
    # Generic per-second counters with 0-2 key/value pairs
    - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_total
      type: COUNTER
      labels:
        "$4": "$5"
        "$6": "$7"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_total
      type: COUNTER
      labels:
        "$4": "$5"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
      name: kafka_$1_$2_$3_total
      type: COUNTER
    # Generic gauges with 0-2 key/value pairs
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
        "$6": "$7"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
      name: kafka_$1_$2_$3
      type: GAUGE
    # Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
    # Note that these are missing the '_sum' metric!
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_count
      type: COUNTER
      labels:
        "$4": "$5"
        "$6": "$7"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
        "$6": "$7"
        quantile: "0.$8"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_count
      type: COUNTER
      labels:
        "$4": "$5"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
        quantile: "0.$6"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
      name: kafka_$1_$2_$3_count
      type: COUNTER
    - pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        quantile: "0.$4"
    # KRaft mode: uncomment the following lines to export KRaft related metrics
    # KRaft overall related metrics
    # distinguish between always increasing COUNTER (total and max) and variable GAUGE (all others) metrics
    #- pattern: "kafka.server<type=raft-metrics><>(.+-total|.+-max):"
    #  name: kafka_server_raftmetrics_$1
    #  type: COUNTER
    #- pattern: "kafka.server<type=raft-metrics><>(.+):"
    #  name: kafka_server_raftmetrics_$1
    #  type: GAUGE
    # KRaft "low level" channels related metrics
    # distinguish between always increasing COUNTER (total and max) and variable GAUGE (all others) metrics
    #- pattern: "kafka.server<type=raft-channel-metrics><>(.+-total|.+-max):"
    #  name: kafka_server_raftchannelmetrics_$1
    #  type: COUNTER
    #- pattern: "kafka.server<type=raft-channel-metrics><>(.+):"
    #  name: kafka_server_raftchannelmetrics_$1
    #  type: GAUGE
    # Broker metrics related to fetching metadata topic records in KRaft mode
    #- pattern: "kafka.server<type=broker-metadata-metrics><>(.+):"
    #  name: kafka_server_brokermetadatametrics_$1
    #  type: GAUGE
  zookeeper-metrics-config.yml: |
    # See https://github.com/prometheus/jmx_exporter for more info about JMX Prometheus Exporter metrics
    lowercaseOutputName: true
    rules:
    # replicated Zookeeper
    - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+)><>(\\w+)"
      name: "zookeeper_$2"
      type: GAUGE
    - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+)><>(\\w+)"
      name: "zookeeper_$3"
      type: GAUGE
      labels:
        replicaId: "$2"
    - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(Packets\\w+)"
      name: "zookeeper_$4"
      type: COUNTER
      labels:
        replicaId: "$2"
        memberType: "$3"
    - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(\\w+)"
      name: "zookeeper_$4"
      type: GAUGE
      labels:
        replicaId: "$2"
        memberType: "$3"
    - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+), name3=(\\w+)><>(\\w+)"
      name: "zookeeper_$4_$5"
      type: GAUGE
      labels:
        replicaId: "$2"
        memberType: "$3"

Thank you :)

marcogenualdo · 2024-05-20T09:27:47Z

marcogenualdo
May 20, 2024
Author

So has it turns out, I had misread the port. It's 9404, not 9094. I think there's some tutorial out there saying it's 9094. I got confused.
Sorry for wasting your time. Thanks again.

1 reply

scholzj May 20, 2024
Maintainer

Ups, sorry ... I missed that as well in your description 🤦 😄

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

[Bug]: Kafka Brokers metrics endpoint unresponsive on port 9094 #10134

{{title}}

Replies: 2 comments 4 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Strimzi

[Bug]: Kafka Brokers metrics endpoint unresponsive on port 9094 #10134

marcogenualdo May 20, 2024

Bug Description

Steps to reproduce

Expected behavior

Strimzi version

Kubernetes version

Installation method

Infrastructure

Configuration files and logs

Additional context

Replies: 2 comments · 4 replies

scholzj May 20, 2024 Maintainer

marcogenualdo May 20, 2024 Author

scholzj May 20, 2024 Maintainer

marcogenualdo May 20, 2024 Author

marcogenualdo May 20, 2024 Author

scholzj May 20, 2024 Maintainer

marcogenualdo
May 20, 2024

Replies: 2 comments 4 replies

scholzj
May 20, 2024
Maintainer

marcogenualdo May 20, 2024
Author

scholzj May 20, 2024
Maintainer

marcogenualdo May 20, 2024
Author

marcogenualdo
May 20, 2024
Author

scholzj May 20, 2024
Maintainer