Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kube-state-metrics] Readiness probe not working with rbac proxy #4992

Open
plnordquist opened this issue Nov 11, 2024 · 3 comments · May be fixed by #5234
Open

[kube-state-metrics] Readiness probe not working with rbac proxy #4992

plnordquist opened this issue Nov 11, 2024 · 3 comments · May be fixed by #5234
Labels
bug Something isn't working

Comments

@plnordquist
Copy link

plnordquist commented Nov 11, 2024

Describe the bug a clear and concise description of what the bug is.

After upgrading to chart version v5.27.0, the kube-state-metrics pod is never marked as ready if the kube-rbac-proxy feature is enabled. The deployment configures the telemetry host to 127.0.0.1 when kubeRBACProxy.enabled is true and the readiness probe uses the pod IP to attempt to reach the telemetry host so it is not reachable.

What's your helm version?

version.BuildInfo{Version:"v3.16.2", GitCommit:"13654a52f7c70a143b1dd51416d633e1071faffb", GitTreeState:"clean", GoVersion:"go1.22.7"}

What's your kubectl version?

Client Version: v1.31.2 Kustomize Version: v5.4.2 Server Version: v1.31.2

Which chart?

kube-state-metrics

What's the chart version?

5.27.0

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

kubeRBACProxy:
  enabled: true

Enter the command that you execute and failing/misfunctioning.

helm install -n monitoring ksm prometheus-community/kube-state-metrics --set "kubeRBACProxy.enabled=true"

Anything else we need to know?

No response

@plnordquist plnordquist added the bug Something isn't working label Nov 11, 2024
@HaveFun83
Copy link

HaveFun83 commented Nov 18, 2024

same here

e5dcb61#diff-964657bf9c31e2d1338046dc10aff7a7d28dc34813c6cd09d84228512e966132L203

Broke the readiness probe with rbacproxy enabled

@fahedouch
Copy link

fahedouch commented Jan 1, 2025

same here. The issue is the following:
when rbacproxyenabled, the telemetry server port is binded to the loopback IP adresse (127.0.0.1) which prevents the kubelet from checking the readiness of the pod, the solution is to bind the socket to 0.0.0.0 or to vary the host via the values

@r0bj
Copy link
Contributor

r0bj commented Jan 19, 2025

The purpose of using kube-rbac-proxy is to prevent unauthorized access to the kube-state-metrics endpoints. However, binding it to 0.0.0.0 so that the kubelet can use it for probes defeats this purpose because it allows direct access to the kube-state-metrics port, bypassing kube-rbac-proxy entirely. So, we should bind both ports of kube-state-metrics to 127.0.0.1, which unfortunately would cause the kubelet probes to stop working. One solution would be to disable probes when using kube-rbac-proxy, similar to how it is done in kube-prometheus: https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/kubeStateMetrics-deployment.yaml
We could make livenessProbe and readinessProbe optional by introducing an enabled field, in a manner similar to the startupProbe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
4 participants