Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build in handling of HTTP errors and/or rate-limiting for NOMADS, especially in FastHerbie #371

Open
alexmerm opened this issue Sep 23, 2024 · 0 comments

Comments

@alexmerm
Copy link

alexmerm commented Sep 23, 2024

Would be nice to be able to be able to be able to handle 403 errors and do some sort of rate limiting for NOMADs requests. Ran into an issue where I got blocked by the NWS's abusive Behavior policy when requesting data from NOMADs.
Abusive Behavior Policy can be found here
Info on how to handle retries on error codes can be found here
and rate limiting in requests can be done using requests-ratelimiter python package.

At the minimum the ability to pass in a requests.session variable would be helpful because then we could set up our own rate limiting/error handling

EDIT: Just wanted to add the issue is especially prevalent when requesting a forecast that hasn't yet been generated, as it causes Herbie to mass-ignore 403 error codes for files that don't yet exist and quickly results in a ban

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant