Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve reproducibility of "deltares_data" catalog sources #537

Closed
2 of 4 tasks
DirkEilander opened this issue Sep 29, 2023 · 1 comment · Fixed by #833
Closed
2 of 4 tasks

Improve reproducibility of "deltares_data" catalog sources #537

DirkEilander opened this issue Sep 29, 2023 · 1 comment · Fixed by #833
Assignees
Labels
Enhancement New feature or request Spillover Issues that were planned but not completed last quarter
Milestone

Comments

@DirkEilander
Copy link
Contributor

DirkEilander commented Sep 29, 2023

Enhancement Description

Updating the meta data section in deltares_data.yml & documentation according to:

  meta:
    source_url: zenodo.org/my_dataset # should point to processed data OR original in combi with processing_notes/script
    source_license: CC-BY-3.0
    source_version: vX.X
    paper_ref: Author et al. (year)
    paper_doi: doi
    processing_notes:  <description of process in script OR simple processing steps (e.g. filter / gdalbuildvrt)>
    processing_script: <url to script>
    category: category

It should be checked case by case what is required for reproducibility. there are several options:

  • publish pre-processed data together with the script on Zenodo (e.g. MODIS_LAI/ MERIT Hydro basins map) and point to this data in source_url
  • point to scripts in processing_script to download and/or process (e.g. ERA5)
  • add processing_notes for simple processing to filter data (e.g. hydro_lakes) or create a vrt (merit)
  • documentation of required data (e.g. bounds is required in the hydrographic region argument unless the basin map and index are present)
  • check used data sources in examples (e.g. replace merit_hydro with merit_hydro_ihu).

ToDo

  • identify data sources in deltares_data.yml that need updating
  • update meta section
  • if the dataset requires publishing / new scripts (more work) -> create separate issues
  • update docs and examples (also in plugins)

Additional Context

this is a continuation of #356

@DirkEilander DirkEilander added Enhancement New feature or request Needs refinement issue still needs refinement labels Sep 29, 2023
@DirkEilander DirkEilander added this to the Q1 milestone Sep 29, 2023
@savente93 savente93 modified the milestones: Q1, Q4 Oct 20, 2023
@savente93 savente93 removed the Needs refinement issue still needs refinement label Nov 2, 2023
@Tjalling-dejong
Copy link
Contributor

The following datasets need to be updated:

  • CHELSA 1.2 -> 2.1
  • COPDEM 2021_1 -> 2022_1
  • E-OBS 25.0e -> 28.0e
  • Esa world cover v100 -> v200
  • FABDEM v1.0 -> v1.2
  • GADM 3.6 -> 4.1
  • GEBCO 2020 -> 2023
  • GHS_POP R2019A_v1.0 -> R2023A
  • GHS_SMOD R2016A_v1.0 -> R2023A
  • GLOFAS v31? -> v4.0
  • GLW v3 -> v4
  • GSWO v1_1_2019 -> v2021
  • Mdt_cnes_cls18 -> cls22
  • MODIS LAI MCD15A3H V006- > MCD15A3H V061
  • SM2RAIN_ASCAT v1.4 -> v2.1
  • Worldclim 2.0 -> worldclim 2.1
  • World settlement footprint 2015 -> 2019 version

@savente93 savente93 added the Spillover Issues that were planned but not completed last quarter label Jan 8, 2024
@savente93 savente93 modified the milestones: 2023 - Q4, 2024 - Q1 Jan 8, 2024
@savente93 savente93 modified the milestones: 2024 - Q1, 2024 - Q2 Apr 8, 2024
@savente93 savente93 linked a pull request May 16, 2024 that will close this issue
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement New feature or request Spillover Issues that were planned but not completed last quarter
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants