Error Enabling Prometheus Integration

[prasadkris]

Greetings,

I am running the latest icinga-kubernetes stack, and it works fine overall. However, I attempted to enable Prometheus integration by adding the following into the configuration:

  prometheus:
     url: http://prometheus-operator-kube-p-prometheus.ops-monitoring.svc.cluster.local:9090

Unfortunately, it fails with the error message below.

I1014 06:55:26.085874       1 main.go:69] Starting Icinga for Kubernetes (0.2.0)
I1014 06:55:26.087175       1 database.go:286] "Connecting to database" logger="database"
F1014 06:55:30.725815       1 main.go:385] error querying Prometheus: can't retry: execution: found duplicate series for the match group {} on the right hand-side of the operation: [{__name__="node_memory_MemTotal_bytes", container="node-exporter", endpoint="http-metrics", job="node-exporter", namespace="ops-monitoring", pod="prometheus-operator-prometheus-node-exporter-gdjwk", service="prometheus-operator-prometheus-node-exporter"}, {__name__="node_memory_MemTotal_bytes", container="node-exporter", endpoint="http-metrics", job="node-exporter", namespace="ops-monitoring", pod="prometheus-operator-prometheus-node-exporter-j4tq5", service="prometheus-operator-prometheus-node-exporter"}];many-to-many matching not allowed: matching labels must be unique on one side

Any idea how we can resolve this? Thanks! 🙏🏻

[jrauh01]

Hi @prasadkris,

I suspect the error in following query:

• sum by (instance, namespace, pod) (container_memory_usage_bytes) / on () group_left () label_replace(node_memory_MemTotal_bytes, "instance", "$1", "node", "(.*)")

A solution for that could be the following query:

• sum by (instance, namespace, pod) (container_memory_usage_bytes) / on (instance) group_left () label_replace(sum by (node) (node_memory_MemTotal_bytes), "instance", "$1", "node", "(.*)")

As I don't have the data from your system it would be nice if you could execute both of the queries in your Prometheus web interface and tell me if the first one is failing and the second one is returning data.

Best regards,
Johannes