ECMWF Open Data download and conversion functions #2975
Conversation
|
|
||
| all_parameters <- c("2t", "tp", "10u", "10v", "q", "r", "sp") | ||
|
|
||
| if (tolower(parameters) == "all") { |
There was a problem hiding this comment.
if this condition is not met there will be no parameter_types, then line 61 will most likely throw a 'parameter_types' not found error, should there be an else to this if?
There was a problem hiding this comment.
Added else to show what can be given as an input, here all.
There was a problem hiding this comment.
I believe this is remains to be the case, please add an else to make sure parameter_types are assigned something in case tolower(parameters) != "all"
There was a problem hiding this comment.
If tolower(parameters) != "all" should the parameter_types still be "all"?
*modified datasets acc to CF standard *updated documentation *resolved PR comments
swarnaleem
changed the title
There was a problem hiding this comment.
I'll continue to review the revised code but in the meantime could you please make sure this PR would make ECMWF pipeline run successfully through the workflow. I still don't see a registration file, which should look something like this.
I also suspect that fchour will never be passed to download.ECMWF by upstream functions, i.e. see here. You can either change that fcn arg to product (logically it's not that far off I'd be fine with it) or you need to add fchour to that convert.input call
| } | ||
|
|
||
| # Removes any already existing files with base name same as `latest_filedate`` | ||
| if ((length(list.files(path = outfolder, pattern = as.character(latest_filedate)) ) > 0) == TRUE){ |
There was a problem hiding this comment.
do we really want to remove or if the global files are already there can we skip the download?
Co-authored-by: istfer <[email protected]>
* updated documentation
|
|
||
| ##### 3h CF -> 0 - 141h at 3h timestep | ||
| cf_3h_out_fname <- paste0(latest_filedate, time, stream, "cf_3h", sep = "_") | ||
| cf_3h_nc <- combine_files_cf(fnames_3h, lat_in, lon_in, out_filename= cf_3h_out_fname) |
There was a problem hiding this comment.
I thought we agreed to not write these intermediate 3hrly and 6 hrly netcdf files, are these still being written?
| script.path = file.path(system.file("ECMWF/download_ecmwf.py", package = "PEcAn.data.atmosphere")) | ||
| reticulate::source_python(script.path) | ||
|
|
||
| date_latestdata <- ecmwflatest(time, step, stream, type, all_parameters) |
There was a problem hiding this comment.
| date_latestdata <- ecmwflatest(time, step, stream, type, all_parameters) | |
| date_latestdata <- ecmwflatest(time, step_15day, stream, type, all_parameters) |
there is no step variable, was this supposed to be step_15day?
|
|
||
| in_fname <- paste(latest_filedate, time, stream, sep = "_") | ||
|
|
||
| data_download <- ecmwfdownload(date, time, stream, type, all_parameters, in_fname) |
There was a problem hiding this comment.
I haven't thoroughly tested, but this function seems to be downloading the grib files to the working directory whereas it probably should use outfolder
|
@swarnalee-m we'd love to pull in this code. Any chance you can address the few remaining issues? |
|
@swarnalee-m pinging you again to hopefully finish up this PR |
|
@istfer any chance you can finish up this PR? @swarnaleem hasn't touched it in a year and doesn't seem to be responding to inquiries about addressing the changes you requested. |
|
I can give it a try |
ECMWF Open Data 15-day forecast download and conversion to NetCDF (from Grib2)
Description
The download.ECMWF.R function downloads ECMWF Opendata 15-day perturbed forecast (
pf) consisting of 50 ensemble members usingecmwf-opendatapython package. Currently, it is downloading 50 ensemble members usingdownload_ecmwf_opendata.py. The parameters being downloaded are as follows:After downloading,
ecmwf_grib2nc.pyfunction converts thegrib2file consisting of 50 ensemble members into 50 separatenetCDFfiles based on their ensemble numbers.Motivation and Context
PEcAn currently has 17 meteorological drivers available, some of which are Ameriflux, ERA5, and CMIP5 which are used as input data to various models. ECMWF Open Data provides globally available ensemble forecasts that can be used to drive various models (available in PEcAn).
Creating an ECMWF Open Data pipeline in PEcAn would help in adding various meteorological forecast data (weather data) as input variables to the ecosystem models which would help in running them in the future. In this proposal, the interest is around 15-day ensemble forecast that originates from the day of the forecast and goes 360h into the future.
Review Time Estimate
Types of changes
Checklist: