Introduce heap balancer for the gRPC connection pool #589
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mrahs
force-pushed
the
master
branch
4 times, most recently
from
502427d
to
bc3a965
Compare
We observed increased latencies in builds that use the round-robin balancer. It is possible that the load is not uniformly spread across the pool, particularly around streams. This could cause client-side queuing on hot connections which may explain the increased latencies. The heap-based balancer keeps track of the number of invocation and streaming requests per connection and selects the least used connection. One big caveat is that the pool is not aware of active connections. The number of stream requests is not necessarily the number of active streams on the connection. Without that knowledge, it is still possible for some connections to be used more than others. E.g. if long streaming requests accumulate on the same connection.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We observed increased latencies in builds that use the round-robin balancer. It is possible that the load is not uniformly spread across the pool, particularly around streams. This could cause client-side queuing on hot connections which may explain the increased latencies.
The heap-based balancer keeps track of the number of invocation and streaming requests per connection and selects the least used connection. One big caveat is that the pool is not aware of active connections. The number of stream requests is not necessarily the number of active streams on the connection. Without that knowledge, it is still possible for some connections to be used more than others. E.g. if long streaming requests accumulate on the same connection.