-
Notifications
You must be signed in to change notification settings - Fork 159
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: Pin redis-py to 4.5.5 and enhance logging for Redis reconnection and failover #1620
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
achimnol
added
type:bug
Reports about that are not working
type:enhance
Enhance component, behavior, internals without user-facing features
impact:invisible
This change is invisible to users (internal changes).
labels
Oct 15, 2023
github-actions
bot
added
comp:manager
Related to Manager component
comp:webserver
Related to Web Server component
size:L
100~500 LoC
labels
Oct 15, 2023
.fix.md -> 1620.fix.md
- This includes critical bug fixes related to Redis connection leaks...
- Implicit "CLIENT SETINFO" command in redis-py 5.0 with Redis 7.x seems to interfere with our reconnection/retry test cases... :(
- In this case, we can distinguish whether the conection failure is caused by either the full connection pool or actual network/server failures.
Made an upstream report: redis/redis-py#3008 |
achimnol
changed the title
fix: Enhance logging for Redis reconnection and failover
fix: Pin redis-py to 4.5.5 and enhance logging for Redis reconnection and failover
Oct 16, 2023
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
Oct 16, 2023
achimnol
added a commit
that referenced
this pull request
Oct 16, 2023
… and failover (#1620) Backported-from: main Backported-to: 23.09
achimnol
added a commit
that referenced
this pull request
Oct 16, 2023
… and failover (#1620) This commit only includes redis-py 4.5.5 pinning, connection pool mis-creation in `AgentRegistry.handle_kernel_log()`, and limitation of `max_connections` for the default connection pool. Added an explicit warning if the native sentinel client is used, because the rework on the sentinel connection pool (#1586) targets 23.09 only. Backported-from: main Backported-to: 23.03
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
comp:manager
Related to Manager component
comp:webserver
Related to Web Server component
impact:invisible
This change is invisible to users (internal changes).
size:XL
500~ LoC
type:bug
Reports about that are not working
type:enhance
Enhance component, behavior, internals without user-facing features
urgency:5
It is imperative that action be taken right away.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
refs #1443
This PR includes several critical fixes:
Upgrade redis-py (4.6.0 → 5.0.1) to get the latest bug fixes related with garbage collection of socket connections and timeout detection on pipelined commandsBlockingConnectionPool
. We need to wait until redis-py stabilizes........Connection.__del__()
.auto_close_connection_pool=True
(NOTE: Missing of this parameter is fixed in redis-py 5.0.0 and deprecated in 5.0.1, but the whole upstream package is broken now.)max_connections
to 16 per connection pool instead of the default 2**31.... (!!)manager.registry.handle_kernel_log()
to avoid creating a new Redis connection pool upon every container destruction requestRelated upstream issues
timeout
cannot exceed client'ssocket_timeout
redis/redis-py#2807Checklist: (if applicable)