-
|
Hi! After upgrading Tempo from 1.5 to 2.0, I noticed that the data for this dropdown is loading much longer. When previously it felt almost instantaneous (seconds), now it loads in 30s+ (sometimes a couple of minutes). I am unsure whether this is connected to the new storage format or something else. Also, I noticed that searching by trace id is now taking longer. The number of spans we store is not small, something like 3k spans a second.
Do you have any advice on how to investigate the root cause? What metrics to look at, what configuration parameters to tweak? UPD: In logs we see "error querying ingesters in Querier.SearchTags: failed to get response from ingesters: context deadline exceeded" |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 3 replies
-
|
We tinkered with config a little, and the problem disappeared. FYI The candidate for the culprit is the |
Beta Was this translation helpful? Give feedback.
-
|
it seems very odd that |
Beta Was this translation helpful? Give feedback.
-
|
Today I saw 10s+ times for this dropdown population too. TBH, upon thinking about it, I am thinking that this (tag search time improvement with minor config changes) was coincidental/transient, and the root problem is the same as in #2118 - just search for tag values being slow-ish (as far as I understand, this dropdown just issues a search for service name tag values). |
Beta Was this translation helpful? Give feedback.
-
|
yup, we are definitely seeing tag search times take too long in certain circumstances. we are working to improve. |
Beta Was this translation helpful? Give feedback.
We tinkered with config a little, and the problem disappeared.
FYI The candidate for the culprit is the
reportingEnabledsetting that was accidentally enabled with the upgrade, but we are not sure at all.