Limit per-host concurrency and add retries #217
+4,539
−269
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Configuration minimizes impact of retries
theunrepentantgeek
force-pushed
the
feature/http-resiliency
branch
from
63c5f1b to
503aa92
Compare
| @@ -280,6 +275,39 @@ func (hT *HTMLTest) checkExternal(ref *htmldoc.Reference) { | |||
| // TODO check a hash id exists in external page if present in reference (URL.Fragment) | |||
| } | |||
|
|
|||
| var hostChannelsLock sync.Mutex // Lock used to prevent concurrent updates to hostChannels | |||
There was a problem hiding this comment.
I initially added this as a member of HTMLTest but that in turn required setRedirectLimitCheck() to be modified by passing a reference instead of a value; this broke redirection tests by increasing the number of issues found from 1 to 2.
Making this a global is slightly suboptimal, but seemed to be a reasonable compromise. Happy to move it back into HTMLTest if that's a better spot for it to live.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Limits the number of concurrent requests to a single host in order to reduce (hopefully to eliminate) errors where the server objects. Also adds automatic retries (with exponential backoff) to further mitigate transient errors.
Partial mitigation of #152.
Closes #103
Closes #25
The introduction of retries has required update of the VCR recordings; the actual changes are fairly limited in scope - I deliberately chose
retryablehttpbecause it was a near seamless drop in replacement.Marked as draft because too many commits have been pulled in by Github.Fixed by rebasing this PR. Might have conflicts if #215 is merged first; I'll remedy those if that happens.