Modis l2 available datasets

[BENR0]

Adds the available_datasets method to the modis level 2 reader to get datasets of arbitrary modis level 2 product files which are not specified in the readers yaml file as suggested in #812 (review).

Currently only datasets which have a 2D shape are added and bit encoded fields are not decoded.

There are lots of level two products out there so this might not yet work in every case. I tested a couple of products which worked fine. I am not sure how to implement mock tests which cover all these different files because I am not very fit with writing tests with mock. If someone can help me out/ give me some hints I can implement those so that this can eventually be merged.

Also there is a problem with the output of the available_dataset_ids of Scene when this is used with the cloud_mask dataset of the mod35_l2 files which are already specified in the yaml file. While the dataset can be loaded it is not shown as available when available_dataset_ids is envoked.
The reason for this as fas as I can see is that this method returns the available_ids keys in the FileYAMLReader here

satpy/satpy/readers/yaml_reader.py

Line 282 in 35739e4

def available_dataset_ids(self):

and not the all_ids attribute where all of them are included. My guess is that available_ids should also be updated with the contents of all_ids in

satpy/satpy/readers/yaml_reader.py

Line 540 in 35739e4

def update_ds_ids_from_file_handlers(self):

Not directly related to the main purpose of this PR I also added add_offset to the scale_factor conversion in the hdfeos_base which was missing. Currently data is read with pyhdf, maybe the reading could be changed to xarray which would do this conversion automatically and also reads the other attributes which might be desireable?

• Tests added and test suite added to parent suite
• Tests passed
• Passes flake8 satpy
• Fully documented
• Add your name to AUTHORS.md if not there already

[djhoese]

Currently data is read with pyhdf, maybe the reading could be changed to xarray which would do this conversion automatically and also reads the other attributes which might be desireable?

Can xarray use pyhdf?

[djhoese]

As for the all versus available, the available_dataset_ids are only the datasets that are available in the files provided so they should not be merged with the "all" datasets (which are all of the possible datasets we know about regardless of the files provided).

Let me check your implementation.

[djhoese]

Wow, great job! Nice start to getting this reader to be more flexible. There are few things that need to be changed, but it's close.

As for the tests, if you add one additional 2D variable (one not configured in the YAML) and one 3D variable to the existing tests and make sure that the 2D variable is added to the available_dataset_ids and the 3D one isn't then I think that'd be good enough.

[djhoese]

One problem with this is that it overlaps with the mod35_hdf pattern so a file could be matched to either file type depending on which one was checked first. Either we have a file pattern for each possible level 2 filename or one single generic one. There are two problems with the single generic one:

• We'd have to handle cloud_mask specially in available_datasets so that it is only shown as available for the mod35 file. Not a huge deal, but could be confusing to some.
• Bigger issue is that if someone provides multiple level 2 files there is no way for Satpy to know which files are additional granules of the same file type (all MOD35 files) or if they are different level 2 product files. The reader ends up asking every file handler for a product even though only some of them have the dataset. I ran in to this issue with the ABI L2 data files even though those aren't granules.

In the future we could make the base reader check the file handler after it is created to see if it changed it's file type. For example the ModisL2HDFFileHandler could say "I see that 'product' in the filename is XX so my file type is actually 'modis_l2_product_XX'". The base reader then knows how to organize the files. @mraspaud did you have to do anything like this for the VIIRS SDR reader when you updated it for the various file naming schemes?

[BENR0]

Yes I guessed that this overlap might pose a problem. But I after experimenting with the order of the patterns I thought that the reader took the first match which would work in the current situation but might be difficult in the future if more datasets are added to the yaml file. I think it would be nice if the reader could also identify if any dataset in the file needs special treatment like bit decoding (then there would be no need for specifying additional datasets in the yaml file) . But this might not be easy, if doable at all.

The problem with different level 2 product files given to the reader at the same time also occurred to me but I thought that most users might be smart enough to keep product files in different directories but I guess that is a hopeful wish and sooner or later somebody will discover this "bug". I think I don't understand enough of the inner workings of how the readers are initialized (interested though to get a better understanding of what is done when and the idea behind it) to judge what would be the best way.

[djhoese]

Understood. I'm 99% sure that there is no guarantee about the order of which file pattern/file type is checked first so it might come down to luck. Maybe let's continue the file type discussion on slack to nail down what I'm thinking. I could make a separate PR for the functionality I'm thinking of and then we can incorporate it in to this PR after it is merged.

[BENR0]

Currently data is read with pyhdf, maybe the reading could be changed to xarray which would do this conversion automatically and also reads the other attributes which might be desireable?

Can xarray use pyhdf?

I guess it uses pyhdf in the background because I can open modis hdf files with it.

[djhoese]

I wouldn't be surprised if it is using NetCDF to read pyhdf by using the NetCDF C library compiled with the HDF4 C library.

[ghost]

DeepCode's analysis on #c6332c found:

• ⚠️ 1 warning, ℹ️ 38 minor issues. 👇
• ✔️ 31 issues were fixed.

Top issues

Description	Example fixes
Defining only __eq__ but not __ne__ will result in a Python2 error if objects are compared with inequality. Occurrences: dataid.py:485	🔧 Example fixes
Unused CRS imported from pyproj Occurrences: test_scene.py:507 test_cf.py:957 nucaps.py:36	🔧 Example fixes
Statement seems to have no effect Occurrences: anc_vars.py:31	🔧 Example fixes

👉 View analysis in DeepCode’s Dashboard | Configure the bot

[djhoese]

It looks like git got the best of this pull request with 2558 commits being shown. I can't even get github to load all the file changes to show me what changed to the hdfeos_base.py file. @BENR0 How do you feel about this PR? Do you want me to try to extract the old history before the merge with main corrupted things?

[BENR0]

@djhoese yeah I could not remember what I did that messed things up (didn't even notice until your comment).

I reset to the appropriate commit and merged main and force pushed which seemed to fix the weird amount of number of commits.

[djhoese]

Nice job. I'm still a little concerned by the file pattern that was added, but otherwise quickly glancing at the available datasets method, it seems pretty good. However, I think it needs to be updated to make sure the variable that it found in the file isn't one that was already configured in the YAML, right?

[BENR0]

However, I think it needs to be updated to make sure the variable that it found in the file isn't one that was already configured in the YAML, right?

That rings a bell. But I can't remember. To be honest I just fixed the commit issue. I have to look at his a little closer again.

[BENR0]

I don't know why netcdf writer tests are failing. Didn't touch anything related.

[pnuu]

Merge with main and they'll work again, there was another XArray release without compression kwarg fixes. See #2674

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.10%. Comparing base (fd2cec6) to head (bb5f538).
Report is 6 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #913   +/-   ##
=======================================
  Coverage   96.10%   96.10%           
=======================================
  Files         377      377           
  Lines       55147    55193   +46     
=======================================
+ Hits        52997    53043   +46     
  Misses       2150     2150           

Flag	Coverage Δ
behaviourtests	3.94% <0.00%> (-0.01%)	⬇️
unittests	96.19% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[coveralls]

Pull Request Test Coverage Report for Build 7248787685

• 48 of 48 (100.0%) changed or added relevant lines in 3 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.003%) to 95.958%

---

Totals
Change from base Build 7246279156:	0.003%
Covered Lines:	50427
Relevant Lines:	52551

---

💛 - Coveralls

[pnuu]

One inline suggestion. I think there should be a test, but it might be pretty hard to write...

[BENR0]

Darn I am pretty sure I did write tests at some point but can't find/recover them. I guess they got lost in the git mixup when I re-merged main and force pushed :-/. I will add some tests again.