Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Force MaskedColumns to regular Columns #696

Open
DinoBektesevic opened this issue Sep 6, 2024 · 0 comments
Open

Force MaskedColumns to regular Columns #696

DinoBektesevic opened this issue Sep 6, 2024 · 0 comments

Comments

@DinoBektesevic
Copy link
Member

See #695.

We should return the test for falsy values, otherwise we risk Astropy autocasting the column into a MaskedColumn without asking us. This is a symptom of the fact that the ButlerStandardizer is version dependent and we have data in our datastore that was processed, exported and then re-ingested, from different stack versions.

When the stack processing the data is of a later version than the one we use with ButlerStandardizer, and has removed one or more properties from objects like visit_info or summary_stats (which we use to extract metadata in the Butler), the returned value is sometimes ambiguous and is interpreted as a MaskedConstant (by either pybind or astropy). This auto-casts the entire column into a masked column which causes unexpected behavior.

The solution is either:

  • set the appropriate obvious, unambiguous, falsy value in the ButlerStandardizer itself (probably the best)
  • force the masked columns in the ImageCollection itself into a regular column by calling col.filled(default_value) and replacing the masked column
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant