New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade RAWDIR save and load from s3 #222

Open

peterdudfield opened this issue Jan 10, 2025 · 1 comment

Contributor

peterdudfield commented Jan 10, 2025

Detailed Description

Currently RAWDIR has to be local

Context

for gfs, its good to download the raw files to s3, so the next time the consumer runs, it doesnt need to re download these

Possible Implementation

use ffspec, or similar, instead of pathlib in https://github.com/openclimatefix/nwp-consumer/blob/main/src/nwp_consumer/internal/repositories/raw_repositories/noaa_s3.py#L131

The text was updated successfully, but these errors were encountered:

Contributor Author

peterdudfield commented Jan 10, 2025

Actually this might not be needed right now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment