api queue message_stats returns wrong numbers occasionally

[HemeraOne]

Describe the bug

From the interface the message rates when sorted from high to low sometimes show a absurt high number which is no where close to the real number (sometimes millions):

During this time the Message rates graph of the specified queue show an accurate number:

meanwhile I have a curl request running in a loop requesting /api/queues/[vhost]/[queue] and extract the .message_stats which return:
"ack": 59407377, "ack_details": { "rate": 667538.2 }, "deliver": 62354194, "deliver_details": { "rate": 701618.2 }, "deliver_get": 62354194, "deliver_get_details": { "rate": 701618.2 },

Reproduction steps

1. running normal production load
2. wait for it to show up

...

Expected behavior

I expect the api returning a more accurate number like the specific queue page in the UI does

Additional context

No response

[michaelklishin]

You do not mention how you define "correct" but management UI uses a data source that is updated every 5 seconds by default and rates are computed using a sliding window, which means every now and then they can be different for different metrics. In addition, while a metric is collected or serve, other things continue happening in parallel.

This "absurdly high" number is likely a side effect of the sliding window calculation.
We have never been able to reproduce it, and I don't remember seeing it recently. The rate is reported "correctly" on the next calculation.

Relying on point-in-time rates for making decisions is not a great idea because they can quickly change. Using multiple measurements in a row and removing outliers may work for monitoring.

The recommendation (for many different reasons) is to disable metrics in the management UI and use Prometheus for monitoring. Management UI is fine as a way of getting started and during development. It has a number of limitations that external monitoring systems do not have.

[HemeraOne]

I'm aware that the stats should be updated every 5 seconds(15 in our case), however the stats from the api endpoint are updated every time I request them, so seemingly it's coming from another source as the statistics should only be collected every 5 seconds.

My current solution is now using prometheus as I'm using KEDA to scale out consumers, but starting KEDA it made me aware of the issue as it was scaling out the deployment to maximum every time.

If there's anything I can additionally provide so it might be more clear as what is happening and might help resolving this issue I'm happy to help.

[michaelklishin]

This is exactly what I mean by using a single returned value for important decisions such as growing the number of application instances: using just one sample means that you will trip up on outliers, including natural spikes that do not last for long.

Why does KEDA and some other similar tools assume that trusting a single sample is a good idea, I don't know.