Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change the sequence of renaming and the unit_add/unit_mult/astype preprocessing steps in data adapter #59

Closed
hboisgon opened this issue Jul 23, 2021 · 2 comments
Assignees
Labels
Enhancement New feature or request

Comments

@hboisgon
Copy link
Contributor

Currently in the data_adapter, renaming of variables happens before unit conversion or type checks.

It would be more consistent to first do:

  • astype
  • unit_add
  • unit_mult
  • renmaing last

This has consequences on the data catalog format.
The dictionnary would then always have the variables names as in the original data as keys (instead of already the renamed variable for unit_mult and unit_add).
This impacts the deltares_data.yml file stored in data-artifacts but not the data extract for Piave itself.

@DirkEilander
Copy link
Contributor

The current sequence (first renaming) is because the .raster and .vector extensions for Datasets/Arrays require correct x, y, time and index dimension names. Some minimal renaming of dimensions should therefore be the first step after reading. Note that nodata values can also be set per variable. My suggested order of the pre-processing would therefore be:

  • renaming dimensions
  • nodata
  • astype
  • unit_add
  • unit_mult
  • renaming variables

@DirkEilander DirkEilander added the Enhancement New feature or request label Dec 3, 2021
@DirkEilander
Copy link
Contributor

This has been fixed in #481

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants