Distributor Ring is not removing Unhealthy Pods automatically

[Shindek77]

Hello,

We have deployed tempo on our cluster in one namespace. On distributor, HPA is enabled and it's of type deployment.

We saw our traces/data was getting dropped. So while investigation we found out that on the distributor ring we can see the few distributor pods with state UNHEALTH and few other pods with state HEALTHY

When we checked on cluster the pods with state HEALTHY are currently running on cluster but the pods with state UNHEALTHY re not currently running on our cluster.
Also on memberlist of tempo, we can see only HEALTHY distributor instances.

Then as per our understanding the ring to unregister them automatically, but its not doing why????? and how to get rid of this issue???

[joe-elliott]

By default the ring does not remove unhealthy instances as it considers this to be an operational issue that must handled manually. Unhealthy distributors should not impact ingestion. The distributor ring is only used to count distributors for global rate limit enforcement.

To prevent unhealthy distributors you need to make sure they are given time to shut down cleanly and unregister themselves from the ring. An improvement was made here recently.