Alert assignment to required opsgenie team based on alert label is not working

[ankitdh7]

Problem

Based on the Alertmanager configuration and the PrometheusRule CRD, I expected that alerts could be routed to specific Opsgenie teams based on the opsgenie_team label in the alert. However, the alerts are always assigned to the team associated with the Opsgenie integration. When using a global integration key (not tied to any specific team), the alerts remain unassigned.

If I set up a new Opsgenie outbound webhook with an integration key linked to a different Opsgenie team, the alerts do get routed correctly. But with potentially hundreds of Opsgenie teams, this manual approach is neither scalable nor secure. Creating individual Opsgenie integration keys for each team and configuring them in the Coralogix platform every time is not a practical solution.

Steps to reproduce

• Create an opsgenie global(not assigned to any team) integration key
  

• Create an alert rule in the above integration to dynamically assign the responders from the payload to responders team in the opsgenie
  

• Clone the repo

git clone https://github.com/coralogix/coralogix-operator.git && cd coralogix-operator

• Install the operator

helm install coralogix-operator charts/coralogix-operator -n observability --set secret.data.apiKey="<coralogix_api_key>" --set coralogixOperator.image.tag="0.2.3" --set coralogixOperator.region="EUROPE2"

• Update the opsgenie URL(received in step 1 ) in this alertmanager config and apply

kubectl apply -f config/samples/alertmanager/example-alertmanager.yaml -n obserbability

• Apply the below test alerts to validate alerting and routing

kubectl apply -n observability -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: observability-test-alerts
  labels:
    role: alert-rules
    app.coralogix.com/track-recording-rules: "true"
    app.coralogix.com/track-alerting-rules: "true"
    app.coralogix.com/managed-by-alertmanger-config: "true"
spec:
  groups:
  - name: example
    rules:
    - alert: exampleAlert
      expr: vector(1)
      for: 1m
      labels:
        priority: P5
        slack_channel: "#tm_coralogix_alert_test"
      annotations:
        summary: "Example Alert Triggered"
        description: "This is an example alert that triggers when the expression vector(1) is true for 30 minutes."
  - name: example2
    rules:
    - alert: exampleAlert2
      expr: vector(1)
      for: 1m
      labels:
        priority: P5
        opsgenie_team: "demo_team"
      annotations:
        summary: "Example Alert Triggered"
        description: "This is an example alert that triggers when the expression vector(1) is true for 30 minutes."
        cxMinNonNullValuesPercentage: "20"
  - name: example3
    rules:
    - alert: exampleAlert3
      expr: vector(1)
      for: 1m
      labels:
        priority: P5
        slack_channel: "#tm_coralogix_alert_test2"
        opsgenie_team: "demo_team"
      annotations:
        summary: "Third Example Alert Triggered"
        description: "This is the third example alert that triggers when the expression vector(1) is true for 30 minutes."
  - name: recording-rules
    rules:
    - record: example:recording:rule
      expr: vector(1)
EOF

Actual Output:

• The alert exampleAlert2, which has the label opsgenie_team set to demo_team, is received in the opsgenie but remains unassigned.

Expected output

• Alerts should be dynamically assigned to opsgenie team - demo_team based on the opsgenie_team label