Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update data catalog #718

Merged
merged 76 commits into from
Mar 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
3ed4bd3
Updated first couple of datasets in deltares_data.yml datacatalog
Tjalling-dejong Nov 9, 2023
d93f9a5
Added processing scripts/notes tags where necessary #356
Tjalling-dejong Nov 10, 2023
1afa979
Merge branch 'main' into improve_reproducibility_dd
Tjalling-dejong Nov 30, 2023
4a9d791
Updated kwargs to driver_kwargs
Tjalling-dejong Nov 30, 2023
4c215db
Updated deltares datacatalog with data version and added variants whe…
Tjalling-dejong Nov 30, 2023
94ba815
updated data catalog sources with version and where applicable variants
Tjalling-dejong Nov 30, 2023
818002c
Added code for updating meta from variants
Tjalling-dejong Nov 30, 2023
66259fe
updated tests for checking if meta is preserved for variants that hav…
Tjalling-dejong Dec 1, 2023
9d6b346
removed source_version
Tjalling-dejong Dec 1, 2023
77c3055
Updated predefined data catalog changelog
Tjalling-dejong Dec 1, 2023
8d37459
updated changelog
Tjalling-dejong Dec 1, 2023
88ad4f8
Merge branch 'main' into improve_reproducibility_dd
Tjalling-dejong Dec 1, 2023
efe622a
removed '*' era5 path in deltares data catalog and added check in dat…
Tjalling-dejong Dec 7, 2023
fe8ea5a
Added data_type to glofas_uparea
Tjalling-dejong Dec 8, 2023
30aa581
merged dd_era5_quick_fix in branch
Tjalling-dejong Dec 8, 2023
47f7fdc
Set mollweide crs to ESRI:54009 otherwise will error
Tjalling-dejong Dec 15, 2023
a7af219
merged main into branch
Tjalling-dejong Dec 21, 2023
252dab3
fixed failing datacatalog validation tests
Tjalling-dejong Dec 21, 2023
1817fbc
added temporal extent to deltares data
Tjalling-dejong Dec 21, 2023
2bab26b
updated changelog
Tjalling-dejong Dec 21, 2023
c79cc31
Merge branch 'main' into improve_reproducibility_dd
savente93 Jan 2, 2024
9a85eb3
added hydromt version to deltares_data
Tjalling-dejong Jan 10, 2024
2dc66f1
added hydromt versions to aws_data and gcs_cmip6_data data catalogs
Tjalling-dejong Jan 10, 2024
237579f
Updated data catalog version
Tjalling-dejong Jan 10, 2024
dd80683
updated changelog
Tjalling-dejong Jan 10, 2024
cac2cf3
Added spatial extent for raster datasets
Tjalling-dejong Jan 15, 2024
c53b5c9
Added spatial extents to geodataframe
Tjalling-dejong Jan 16, 2024
9a9b6a9
updated data catalog changelog
Tjalling-dejong Jan 16, 2024
0403192
fixed merge conflict
Tjalling-dejong Jan 16, 2024
9ad2aad
removed gtsm_codec_reanalysis dataset from deltares data catalog and …
Tjalling-dejong Jan 16, 2024
cb16e3c
updated data catalog change log
Tjalling-dejong Jan 16, 2024
c4c9239
Merge branch 'main' into improve_reproducibility_dd
Tjalling-dejong Jan 23, 2024
552c142
updated changelog
Tjalling-dejong Jan 23, 2024
9b2ea1a
Merge branch 'main' into improve_reproducibility_dd
Tjalling-dejong Jan 25, 2024
d0a31f3
Merge branch 'main' into improve_reproducibility_dd
Tjalling-dejong Jan 29, 2024
e4acd67
update data catalog readme
Tjalling-dejong Jan 29, 2024
d3323f1
Merge branch 'main' into improve_reproducibility_dd
Tjalling-dejong Feb 1, 2024
7d46830
removed merge conflict statements
Tjalling-dejong Feb 7, 2024
0b56622
Merge branch 'main' into improve_reproducibility_dd
Tjalling-dejong Feb 7, 2024
a356974
Added mapping table for old to new names of datasets in datacatalog
Tjalling-dejong Feb 7, 2024
b4ee491
removed versions from dataset names and added hydro prefix where nece…
Tjalling-dejong Feb 7, 2024
5944ecf
removed source version and unit from meta
Tjalling-dejong Feb 7, 2024
4397c35
removed archived gtsm dataset data catalog
Tjalling-dejong Feb 7, 2024
0174234
trying to add table to rst
Tjalling-dejong Feb 7, 2024
86f1a25
trying to add table to rst
Tjalling-dejong Feb 7, 2024
13d2f31
trying to add table to rst
Tjalling-dejong Feb 7, 2024
0c8e306
test changelog in markdown
Tjalling-dejong Feb 8, 2024
8b2c0ef
added table to changelog
Tjalling-dejong Feb 8, 2024
5d928eb
transefered rst changelog to md changelog
Tjalling-dejong Feb 8, 2024
93b360b
added versions to datasets where possible
Tjalling-dejong Feb 8, 2024
b93ca08
removed source version
Tjalling-dejong Feb 8, 2024
1c4c204
added table with default versions for datasets with variants
Tjalling-dejong Feb 8, 2024
1559bc7
Merge branch 'main' into improve_reproducibility_dd
Tjalling-dejong Feb 8, 2024
4fe7c4d
Merge branch 'main' into improve_reproducibility_dd
Tjalling-dejong Mar 1, 2024
710e378
moved soilgrids_v2 from deltares_data catalog to its own data catalog
Tjalling-dejong Mar 1, 2024
cef7a69
updated artifact_data catalog
Tjalling-dejong Mar 1, 2024
f384955
updated artifact_data in predefined_data_catalog.yml
Tjalling-dejong Mar 1, 2024
166af6e
Update data/catalogs/deltares_data.yml
Tjalling-dejong Mar 1, 2024
8622059
Update data/catalogs/deltares_data.yml
Tjalling-dejong Mar 1, 2024
58d0c41
renamed vito
Tjalling-dejong Mar 1, 2024
07e6be9
Merge branch 'improve_reproducibility_dd' of https://github.com/Delta…
Tjalling-dejong Mar 1, 2024
67c0a44
Update data/catalogs/deltares_data.yml
Tjalling-dejong Mar 1, 2024
284a056
Merge branch 'improve_reproducibility_dd' of https://github.com/Delta…
Tjalling-dejong Mar 1, 2024
fe6a942
added temporal extents for each eobs version
Tjalling-dejong Mar 1, 2024
e630cb9
added nodata to worldclim
Tjalling-dejong Mar 1, 2024
94632b3
updated artifact versions
Tjalling-dejong Mar 1, 2024
b4914e8
review changes and add consistency with epoch in naming
DirkEilander Mar 1, 2024
d138d4c
revert move artifact_data catalog to core
DirkEilander Mar 1, 2024
76ea696
Update data/catalogs/deltares_data.yml
Tjalling-dejong Mar 4, 2024
06170f1
Merge branch 'main' into improve_reproducibility_dd
Tjalling-dejong Mar 4, 2024
6a582a7
fixed typo
Tjalling-dejong Mar 4, 2024
9fa6713
re-revert move artifact data cataloc to core
DirkEilander Mar 4, 2024
555ccb8
fix artifact version in example and test; add artifact data changelog
DirkEilander Mar 4, 2024
e34e355
Merge branch 'main' into improve_reproducibility_dd
DirkEilander Mar 4, 2024
a80cb33
Merge branch 'main' into improve_reproducibility_dd
DirkEilander Mar 6, 2024
64c1c16
update artifact data v0.0.8 git hash
DirkEilander Mar 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 13 additions & 10 deletions data/catalogs/README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -80,21 +80,24 @@ structure per data set:
- order the components of each data set alphabetically
- for adding meta data use the following optional keys:

category:
notes:
paper_doi:
paper_ref:
source_author: (if different from paper_ref)
source_license:
source_url:
source_version:
unit:
meta:
category:
notes:
paper_doi:
paper_ref:
source_author: (if different from paper_ref)
source_license:
source_url:
source_spatial_extent:
source_temporal_extent:
processing_script:
processing_notes:

updates
-------

- create new branch on github
- make changes and bump the version in the global meta section using `calendar versioning <https://calver.org/>`
- test your yml file (Can the added/changed data sources be read through HydroMT?)
- test your yml file (Can the added/changed data sources be read through HydroMT?), this can be done by running the test_data_catalog.py file located in this folder. Read the docstring for more information.
- create pull request
- add new version to hydromt\data\predefined_catalogs.yml
Loading
Loading