Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GitHubAbuseLimitHandler#isError fails to detect some secondary rate limit errors #2009

Open
yrodiere opened this issue Jan 15, 2025 · 1 comment

Comments

@yrodiere
Copy link
Contributor

Describe the bug

#1895 attempted to detect more errors related to secondary rate limits, but it seems there are more.

It appears GitHub APIs -- https://api.github.com/search/issues in particular, maybe more -- can hit a secondary limit and just return a 403 error, with no particular header indicating that a limit was reached. No Retry-After, no gh-limited-by, nothing. Only the body response explains "You have exceeded a secondary rate limit. Please wait a few minutes before you try again". (and in fact waiting 30s/1min is enough).

This, understandably, doesn't get caught by GitHubAbuseLimitHandler#isError, and results in the request simply failing.

To Reproduce

Send the following request a few dozen times (or just a dozen when unauthenticated):

https://api.github.com/search/issues?sort=updated&order=desc&q=repo%3Ayrodiere%2Fquarkus-github-playground+is%3Aissue+is%3Aopen+label%3Atriage%2Fneeds-feedback%2Ctriage%2Fneeds-reproducer+label%3Aarea%2Fhibernate-search+updated%3A%3C2025-01-15T12%3A03%3A47.243460581

Obviously my actual use case is not to send the same request over and over, but it's enough to reproduce the problem.

Interestingly, you can just click that URL in your browser, refresh quickly a couple times, and you'll be able to observe the exact error I'm running into: HTTP 403 with no particular retry header.

See bottom of this messsage for more logs.

Expected behavior

Ideally, I'd expect such requests to be detected as secondary rate limit errors, and the client to just wait and retry. For my use case it's fine to wait a minute.

Failing that, I'd settle for a way to override GitHubAbuseLimitHandler#isError and implement some dirty logic based on the response body in there. But this method is currently package-protected.

Desktop (please complete the following information):

  • OS: Fedora 41
  • Browser [e.g. chrome, safari]: no browser involved
  • Version [e.g. 22]: no browser involved

Additional context

Here is a failing request/response with all details/headers. I got it using -Djdk.httpclient.HttpClient.log=all.

2025-01-15 13:04:53,295 INFO  [jdk.htt.HttpClient] (executor-thread-1) REQUEST: https://api.github.com/search/issues?sort=updated&order=desc&q=repo%3Ayrodiere%2Fquarkus-github-playground+is%3Aissue+is%3Aopen+label%3Atriage%2Fneeds-feedback%2Ctriage%2Fneeds-reproducer+label%3Aarea%2Fhibernate-search+updated%3A%3C2025-01-15T12%3A03%3A47.243460581 GET
2025-01-15 13:04:53,295 INFO  [jdk.htt.HttpClient] (executor-thread-1) HEADERS: REQUEST HEADERS:
GET /search/issues?sort=updated&order=desc&q=repo%3Ayrodiere%2Fquarkus-github-playground+is%3Aissue+is%3Aopen+label%3Atriage%2Fneeds-feedback%2Ctriage%2Fneeds-reproducer+label%3Aarea%2Fhibernate-search+updated%3A%3C2025-01-15T12%3A03%3A47.243460581 HTTP/1.1
Content-Length: 0
Host: api.github.com
User-Agent: Java-http-client/17.0.13
Accept: application/vnd.github+json
Accept-Encoding: gzip
Authorization: token <NOPE>
X-GitHub-Api-Version: 2022-11-28


2025-01-15 13:04:53,422 INFO  [jdk.htt.HttpClient] (HttpClient-3-Worker-0) HEADERS: RESPONSE HEADERS:
    access-control-allow-origin: *
    access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
    content-encoding: gzip
    content-security-policy: default-src 'none'
    content-type: application/json; charset=utf-8
    date: Wed, 15 Jan 2025 12:04:53 GMT
    referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
    server: github.com
    strict-transport-security: max-age=31536000; includeSubdomains; preload
    transfer-encoding: chunked
    vary: Accept-Encoding, Accept, X-Requested-With
    x-content-type-options: nosniff
    x-frame-options: deny
    x-github-api-version-selected: 2022-11-28
    x-github-media-type: github.v3; format=json
    x-github-request-id: <NOPE>
    x-ratelimit-limit: 30
    x-ratelimit-remaining: 21
    x-ratelimit-reset: 1736942747
    x-ratelimit-resource: search
    x-ratelimit-used: 9
    x-xss-protection: 0


2025-01-15 13:04:53,423 INFO  [jdk.htt.HttpClient] (HttpClient-3-Worker-0) RESPONSE: (GET https://api.github.com/search/issues?sort=updated&order=desc&q=repo%3Ayrodiere%2Fquarkus-github-playground+is%3Aissue+is%3Aopen+label%3Atriage%2Fneeds-feedback%2Ctriage%2Fneeds-reproducer+label%3Aarea%2Fhibernate-search+updated%3A%3C2025-01-15T12%3A03%3A47.243460581) 403 HTTP_1_1 Local port:  57158

I end up with this error, proving the secondary limit error wasn't detected:

org.kohsuke.github.GHException: Failed to retrieve https://api.github.com/search/issues?sort=updated&order=desc&q=repo%3Ayrodiere%2Fquarkus-github-playground+is%3Aissue+is%3Aopen+label%3Atriage%2Fneeds-feedback%2Ctriage%2Fneeds-reproducer+label%3Aarea%2Fhibernate-search+updated%3A%3C2025-01-15T12%3A03%3A47.243460581
        at org.kohsuke.github.GitHubPageIterator.fetch(GitHubPageIterator.java:157)
        at org.kohsuke.github.GitHubPageIterator.hasNext(GitHubPageIterator.java:93)
        at org.kohsuke.github.PagedSearchIterable$1.hasNext(PagedSearchIterable.java:111)
        at org.kohsuke.github.PagedIterator.fetch(PagedIterator.java:116)
        at org.kohsuke.github.PagedIterator.hasNext(PagedIterator.java:84)
        at java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1855)
        at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292)
        at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
        at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
        at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298)
        at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
        at io.quarkus.github.lottery.draw.Lottery$Draw.runSingleRound(Lottery.java:286)
        at io.quarkus.github.lottery.draw.Lottery.draw(Lottery.java:80)
        at io.quarkus.github.lottery.LotteryService.doDrawForRepository(LotteryService.java:109)
        at io.quarkus.github.lottery.LotteryService.drawForRepository(LotteryService.java:82)
        at io.quarkus.github.lottery.LotteryService.draw(LotteryService.java:66)
        at io.quarkus.github.lottery.LotteryService_ClientProxy.draw(Unknown Source)
        at io.quarkus.github.lottery.LotteryCli$DrawCommand.run(LotteryCli.java:27)
        at io.quarkus.github.lottery.LotteryCliCommandDispatcherImpl.dispatch(Unknown Source)
        at io.quarkus.github.lottery.LotteryCliCommandDispatcherImpl_Multiplexer.dispatch_9c21b751bd1c3c76c9e9f8d4cbcacbc05e9e7ed7(Unknown Source)
        at io.quarkus.github.lottery.LotteryCliCommandDispatcherImpl_Multiplexer_Observer_dispatch_9c21b751bd1c3c76c9e9f8d4cbcacbc05e9e7ed7_usg3oRfZFhFtsdK_zfgxrYi5m6Q.notify(Unknown Source)
        at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:351)
        at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:333)
        at io.quarkus.arc.impl.EventImpl$1.get(EventImpl.java:110)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:637)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654)
        at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.kohsuke.github.HttpException: {
  "documentation_url": "https://docs.github.com/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits",
  "message": "You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 6F00:24CFC5:<NOPE>."
}
        at org.kohsuke.github.GitHubConnectorResponseErrorHandler$1.onError(GitHubConnectorResponseErrorHandler.java:83)
        at org.kohsuke.github.GitHubClient.detectKnownErrors(GitHubClient.java:504)
        at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:464)
        at org.kohsuke.github.GitHubPageIterator.fetch(GitHubPageIterator.java:146)
        ... 33 more
@yrodiere
Copy link
Contributor Author

FWIW I submitted a ticket to GitHub support to see if they would consider it a bug that there's no useful header in the response. We'll see.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant