Latency in trace search

[RonanQuigley]

When I use trace search to return the latest queries, they are always n minutes behind. I'd like to understand why that is.

If I specify no service name, the latency is ~5 minutes (even though I only want to review results within the last 5 minutes). The screenshot was taken at 16:29

If I specify a specific service, the start time lag decreases.

If I decrease the limit e.g. use the default of 20, the number of minutes we are behind increases.

Helm chart info:

Chart: tempo-distributed-1.8.2
App Version: 2.3.1

Edit:

Okay, if we increase these settings plus have a very high limit the latency decreases:

  grpc_server_max_recv_msg_size: 29000000
  grpc_server_max_send_msg_size: 29000000

So could someone please explain what they're doing? As it's not clear to me. What I'd expect to happen is if I set a limit of 20 traces, it always returns the last 20 traces within the last N minutes.

[mdisibio]

Hi, I think this is an effect of how Tempo searches in a non-deterministic way. When searching it fans out horizontally and returns the first N results. Deterministic search is on our radar (see a few issues), and there is ongoing work (see this PR ) but there's no defined timeline.

[RonanQuigley]

Thanks, I'll keep an eye on those issues. I have a similar sentiment to some of the comments I read: it definitely threw me off when I was looking through the traces, so if the plan is to move towards deterministic search then that's a ➕ from me.