Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update data catalog #718

Merged
merged 76 commits into from
Mar 6, 2024
Merged

Update data catalog #718

merged 76 commits into from
Mar 6, 2024

Conversation

Tjalling-dejong
Copy link
Contributor

@Tjalling-dejong Tjalling-dejong commented Jan 10, 2024 •
edited
Loading

Issue addressed

Reopens PR #667
Addresses issues #537 #521 #708 #701

Explanation

Explain how you addressed the bug/feature request, what choices you made and why.

Checklist

  • Updated tests or added new tests
  • Branch is up to date with main
  • Tests & pre-commit hooks pass
  • Updated documentation if needed
  • Updated changelog.rst if needed
  • For predefined catalogs: update the catalog version in the file itself, the references in data/predefined_catalogs.yml, and the changelog in data/changelog.rst

Additional Notes (optional)

Add any additional notes or information that may be helpful.

Tjalling-dejong and others added 30 commits November 9, 2023 17:08
@Tjalling-dejong
Updated first couple of datasets in deltares_data.yml datacatalog
3ed4bd3
@Tjalling-dejong
Added processing scripts/notes tags where necessary #356
d93f9a5
@Tjalling-dejong
Merge branch 'main' into improve_reproducibility_dd
1afa979
@Tjalling-dejong
Updated kwargs to driver_kwargs
4a9d791
@Tjalling-dejong
Updated deltares datacatalog with data version and added variants whe…
4c215db 
…re possible
@Tjalling-dejong
updated data catalog sources with version and where applicable variants
94ba815
@Tjalling-dejong
Added code for updating meta from variants
818002c
@Tjalling-dejong
updated tests for checking if meta is preserved for variants that hav…
66259fe 
…e their own meta
@Tjalling-dejong
removed source_version
9d6b346
@Tjalling-dejong
Updated predefined data catalog changelog
77c3055
@Tjalling-dejong
updated changelog
8d37459
@Tjalling-dejong
Merge branch 'main' into improve_reproducibility_dd
88ad4f8
@Tjalling-dejong
removed '*' era5 path in deltares data catalog and added check in dat…
efe622a 
…a adapter
@Tjalling-dejong
Added data_type to glofas_uparea
fe8ea5a
@Tjalling-dejong
merged dd_era5_quick_fix in branch
30aa581
@Tjalling-dejong
Set mollweide crs to ESRI:54009 otherwise will error
47f7fdc
@Tjalling-dejong
merged main into branch
a7af219
@Tjalling-dejong
fixed failing datacatalog validation tests
252dab3
@Tjalling-dejong
added temporal extent to deltares data
1817fbc
@Tjalling-dejong
updated changelog
2bab26b
@savente93
Merge branch 'main' into improve_reproducibility_dd
c79cc31
@Tjalling-dejong
added hydromt version to deltares_data
9a85eb3
@Tjalling-dejong
added hydromt versions to aws_data and gcs_cmip6_data data catalogs
2dc66f1
@Tjalling-dejong
Updated data catalog version
237579f
@Tjalling-dejong
updated changelog
dd80683
@Tjalling-dejong
Added spatial extent for raster datasets
cac2cf3
@Tjalling-dejong
Added spatial extents to geodataframe
c53b5c9
@Tjalling-dejong
updated data catalog changelog
9a9b6a9
@Tjalling-dejong
fixed merge conflict
0403192
@Tjalling-dejong
removed gtsm_codec_reanalysis dataset from deltares data catalog and …
9ad2aad 
…created a separate gtsm data catalog
Tjalling-dejong and others added 13 commits March 1, 2024 13:58
@Tjalling-dejong
updated artifact_data catalog
cef7a69
@Tjalling-dejong
updated artifact_data in predefined_data_catalog.yml
f384955
@Tjalling-dejong @hboisgon
Update data/catalogs/deltares_data.yml
166af6e 
Co-authored-by: hboisgon <[email protected]>
@Tjalling-dejong @hboisgon
Update data/catalogs/deltares_data.yml
8622059 
Co-authored-by: hboisgon <[email protected]>
@Tjalling-dejong
renamed vito
58d0c41
@Tjalling-dejong
Merge branch 'improve_reproducibility_dd' of https://github.com/Delta…
07e6be9 
…res/hydromt into improve_reproducibility_dd
@Tjalling-dejong @hboisgon
Update data/catalogs/deltares_data.yml
67c0a44 
Co-authored-by: hboisgon <[email protected]>
@Tjalling-dejong
Merge branch 'improve_reproducibility_dd' of https://github.com/Delta…
284a056 
…res/hydromt into improve_reproducibility_dd
@Tjalling-dejong
added temporal extents for each eobs version
fe6a942
@Tjalling-dejong
added nodata to worldclim
e630cb9
@Tjalling-dejong
updated artifact versions
94632b3
@DirkEilander
review changes and add consistency with epoch in naming
b4914e8
@DirkEilander
revert move artifact_data catalog to core
d138d4c
@DirkEilander DirkEilander mentioned this pull request Mar 1, 2024
Closed
2 tasks
@DirkEilander
Copy link
Contributor

thanks for your work on this one @Tjalling-dejong!
I've addressed all remaining open issues by @hboisgon and checked the catalogs. Generally these seems ready.
For the artifact_data catalog I switched to the old behaviour, including the data_catalog.yml in the tar archive due to a bug in the code I think, see #811. I suggest to leave this for another PR, also in relation to #737
To merge this PR we just need to fix the tests, I haven't yet looked into that. Would be great if you can follow up on that next week.

@Tjalling-dejong
Copy link
Contributor Author

@DirkEilander failing tests have been fixed. Can you approve?

DirkEilander
DirkEilander approved these changes Mar 6, 2024
View reviewed changes
Copy link
Contributor

@DirkEilander DirkEilander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Green light, finally! Thanks for pushing this one forward @Tjalling-dejong

@Tjalling-dejong Tjalling-dejong merged commit 9f4d3f2 into main Mar 6, 2024
11 checks passed
@Tjalling-dejong Tjalling-dejong deleted the improve_reproducibility_dd branch March 6, 2024 15:07
Tjalling-dejong added a commit that referenced this pull request Mar 6, 2024
@Tjalling-dejong
Revert "Update data catalog (#718)"
d0f336f 
This reverts commit 9f4d3f2.
@Tjalling-dejong Tjalling-dejong mentioned this pull request Mar 6, 2024
Merged
@DirkEilander DirkEilander restored the improve_reproducibility_dd branch March 6, 2024 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hboisgon hboisgon hboisgon left review comments

@DirkEilander DirkEilander DirkEilander approved these changes

Assignees

@Tjalling-dejong Tjalling-dejong

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Tjalling-dejong @hboisgon @DirkEilander @savente93