Access JMX metrics from outside K8s

[alonpr]

Hi,

When setting jmxOptions: {} it exposes JMX metrics (on port 9999) which are accessible only from other pods but not from outside k8s (due to ephemeral rmi port...) - Is that correct?

[scholzj]

I don't know JMX much. If you would want to expose it to outside your cluster, you will probably need to create some load balancer, node port or similar services for it to expose it. Strimzi does not expose it to the outside on its own and IMHO the JMX security is relatively weak so I would not do it.

[alonpr]

To follow up on this, does jmxOptions: {} set both JMX_PORT and KAFKA_JMX_OPTS ?

[scholzj]

I don't know. You would need to check the code or try it out. It does what is needed to enable the JMX.

[alonpr]

It's set in set_kafka_jmx_options.sh ->
KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Dcom.sun.management.jmxremote=true -Djava.rmi.server.hostname=$(hostname -i) -Djava.net.preferIPv4Stack=true"

Is it possible to override KAFKA_JMX_OPTS somehow that instead of this value it'd be something that I provide?

[scholzj]

I don't know - I don' think so.

[omersiar]

This would also help troubleshoot JVM further deeply with JMX RMI tools.

Currently default behaviour for strimzi to set java.rmi.server.hostname JMX configuration to Pod's IP address, which makes impossible to connect JMX RMI interface from outside world (behind NAT or Port-Forwarded proxy like kubectl)

If this is intended (to be only routable within the kubernetes network) then jmxOptions should expose a configuration for those who want change this behaviour enabling us to start a troubleshooting session.
https://strimzi.io/docs/operators/latest/configuring.html#type-KafkaJmxOptions

If it does not advertise Pod IP or advertise 127.0.0.1 or localhost then this would not be a problem. -javaagent would not be affected since it is also another local process.

[omersiar]

Putting it behind a Loadbalancer would not work because once initial connection is made, RMI advertise the IP address of the Pod, unless modified by the loadbalancer (I could not able to find a way to do it other than https://github.com/marcin-wolak/SimpleRmiDiscoverer?tab=readme-ov-file#simplermidiscoverer) client would try to connect that IP.

[omersiar]

Hi,

When setting jmxOptions: {} it exposes JMX metrics (on port 9999) which are accessible only from other pods but not from outside k8s (due to ephemeral rmi port...) - Is that correct?

Correct, but not because it is ephemeral, Strimzi explicitly set RMI port to as same as JMX > 9999, it is not ephemeral.