Alert not triggering?

[Danferno]

Am I doing something wrong? I have set up a trace event with event_name failure and want to get Slack notifications whenever such an event occurs. I see these events in the uptrace overview and set up following monitor:

monitors:
  - name: failure
    type: metric
    notify_everyone_by_email: true
    metrics:
      - uptrace_tracing_events as $events
    query:
      - perMin(sum($events))
      - where _event_name = "failure"
    column: perMin(sum($events))
    grouping_interval: 1m0s
    check_num_point: 1
    null_points: convert
    max_allowed_value: 0
    flapping: null

And as you can see in the attached screenshot, it is correctly identifying these events. I played around with grouping interval and check_num_point but that doesn't seem to do anything. What am I missing and/or is there a better way to do this?

[Danferno]

So it finally triggered after half an hour for some reason. Any ideas why this is so delayed?

:fire: Firing: 3 is greater than 0
[sequenza] failure: perMin(sum($events))
You specified that the value should be between -Inf and 0.
The actual value of 3 (3) has been greater than this range for at least 1m.

[vmihailenco]

Any ideas why this is so delayed?

When you create an alert, Uptrace only checks the very last point. Can it be that you did not have any events after you created the alert?

[Danferno]

Any ideas why this is so delayed?

When you create an alert, Uptrace only checks the very last point. Can it be that you did not have any events after you created the alert?

I did make new events after, so it's really weird. Now it's working consistently, so I guess we can close this issue but if there's anything I can provide for internal debugging then let me know!

[Danferno]

There seems to be a wider issue. E.g. now I'm using error logs rather than traces and they arrive in uptrace just fine, but no alerts are being sent out. As you can see in the screenshot, a bunch of errors are getting logged by the standard error alerter ("Notify on all errors") but it doesn't actually lead to any alerts being sent out

[vmihailenco]

It is not clear whether all those errors are new or those are new occurrences for existing errors. From documentation:

[Danferno]

I haven't received a single notification yet from any of those errors, so I think that doesn't explain it?

[vmihailenco]

so I think that doesn't explain it?

It does, because Uptrace always creates alerts for such errors, but it only sends you a notification after you've configured one.

You can check the Alerting to see when alerts were created.

[Danferno]

I have to admit I still don't get it, but maybe it's just me. Errors are being raised continuously, the monitoring rule is enabled by default by uptrade, the errors are showing up in the panel and alerting is set for email and slack notifications (see screenshot), yet I've never received a single notification? If this is expected behaviour I feel like that should be documented explicitly in the relevant panel?