Increase RequestTimout to fix overlappingIP context deadline error #478
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: smoshiur1237 <[email protected]>
|
/cc @dougbtv @manuelbuil |
|
/cc @mlguerrero12 |
|
We have a customer reporting this issue with 100 nodes and 30k pods. What you're proposing might work for 500 pods but not for 30k. |
Here the error and podref issue is coming up because of the timeout. Yes 100s timeout will work with 500pods incase of listing and deletion of overlappingIP. Yes it may not work for 30k pods. Do you have any suggestion how to handle this? |
|
New fix PR is up #480, so closing this PR |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
ListOverlappingIPs function fails with error :
failed to list all OverLappingIPs: client rate limiter Wait returned an error: context deadline exceeded.Also, DeleteOverlappingIP function also unable to delete unused overlapping after scale down of pods.
Here with the test case we are having 500 pods with overlapping IP feature enabled, during scale down of pods from 500 to 1, we are getting the error with
context deadline exceedand also see undeleted pod reference. So basically it is an issue with the RequestTimeout where it was having 10s timeout. The client have default 5qps and for 500 pods, it needs 100s to send all the query. Because of 10s request timeout used in overlapping ip list and deletion, it gets timed out and giving the following error. The modification of this timeout from 10s to 100s will not change the basic functionality but adding more time to process the query and deletion.Which issue(s) this PR fixes:
Fixes #389