ADD: MRR2 backend reattempt

[rcjackson]

I have restarted the pull request for the MRR2 reader using the current version as a base. I'm hoping that I've made changes that fix the PR.

• Closes #xxxx
• Tests added
• Changes are documented in history.md

[rcjackson]

Ruff is failing on notebooks unrelated to this PR. I wonder if there was a recent update for it to support Jupyter notebooks.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[codecov]

Codecov Report

Attention: Patch coverage is 96.50794% with 11 lines in your changes missing coverage. Please review.

Project coverage is 92.26%. Comparing base (cd13dc1) to head (d4feaee).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
xradar/io/backends/metek.py	96.49%	11 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #198      +/-   ##
==========================================
+ Coverage   91.97%   92.26%   +0.29%     
==========================================
  Files          23       24       +1     
  Lines        4573     4888     +315     
==========================================
+ Hits         4206     4510     +304     
- Misses        367      378      +11     

Flag	Coverage Δ
notebooktests	78.70% <85.39%> (+0.46%)	⬆️
unittests	90.60% <95.87%> (+0.36%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kmuehlbauer]

@rcjackson Any idea on the bus error at the one nexrad test?

[rcjackson]

Memory leak somewhere? I haven’t been able to find it. From: Kai Mühlbauer ***@***.***> Date: Tuesday, August 27, 2024 at 1:30 AM To: openradar/xradar ***@***.***> Cc: Jackson, Robert ***@***.***>, Mention ***@***.***> Subject: Re: [openradar/xradar] ADD: MRR2 backend reattempt (PR #198) @ rcjackson Any idea on the bus error at the one nexrad test? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned. Message ID: <openradar/xradar/pull/198/c2311678517@ github. com> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd @rcjackson<https://urldefense.us/v3/__https:/github.com/rcjackson__;!!G_uCfscf7eWS!Y_ThD4cL6cHVVZFproDciAN6Zhh3zHbZjGjkBh00rdXNdZifRZTblHs3Olk6cj_ElibCJCXw1HbCk4qgArYj4Sko$> Any idea on the bus error at the one nexrad test? — Reply to this email directly, view it on GitHub<https://urldefense.us/v3/__https:/github.com/openradar/xradar/pull/198*issuecomment-2311678517__;Iw!!G_uCfscf7eWS!Y_ThD4cL6cHVVZFproDciAN6Zhh3zHbZjGjkBh00rdXNdZifRZTblHs3Olk6cj_ElibCJCXw1HbCk4qgAuQyKY9d$>, or unsubscribe<https://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/AFIQA5AQZNYZPYRKM4XHGPDZTQMIFAVCNFSM6AAAAABNAOJKR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJRGY3TQNJRG4__;!!G_uCfscf7eWS!Y_ThD4cL6cHVVZFproDciAN6Zhh3zHbZjGjkBh00rdXNdZifRZTblHs3Olk6cj_ElibCJCXw1HbCk4qgAkmwPjZP$>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[kmuehlbauer]

@rcjackson I've made a couple comments, how we might tackle the issue with the bus error.

I've checked the tests on main here:
https://github.com/openradar/xradar/actions/runs/10578167450
and they are working perfectly fine.

So I'm assuming that the ominous bus error is somehow introduced with the changes in this PR, or are revealed by the changes in this PR.

If you have any other ideas how to tackle this, please comment.

[rcjackson]

@kmuehlbauer None of the suggestions worked. I even explicitly close the datatree's Dataset to make sure the references clear, but that still doesn't seem to resolve the issue. I'll have to think of more solutions for this later, but for now I'm at a loss.

[kmuehlbauer]

@rcjackson This seems like a hard nut to crack, thanks for checking this out.

Maybe those open references are some hint as to why the error occurred in the first place? I'll have a look at our mrr reader (which is extremely fragile while being based on numpy.genfromtxt).

[kmuehlbauer]

@rcjackson We might try to run this sequentially.

Please change here:

xradar/.github/workflows/ci.yml

Line 69 in 1b4a535

pytest -n auto --dist loadfile --verbose --durations=15 --cov-report xml:coverage_unit.xml --cov=xradar --pyargs tests

and remove -n auto from the call.

Another thing to look after is this error:

  Traceback (most recent call last):
    File "/home/runner/work/xradar/xradar/xradar/io/backends/metek.py", line 447, in close
      for k in self._data.keys():
               ^^^^^^^^^^
  AttributeError: 'MRR2File' object has no attribute '_data'. Did you mean: 'data'?

which was issued here: https://github.com/openradar/xradar/actions/runs/10600167596/job/29376980199?pr=198

It seems, that one CI run finishes and all other raise with this error. Strange!

[rcjackson]

@rcjackson We might try to run this sequentially.

Please change here:

xradar/.github/workflows/ci.yml

Line 69 in 1b4a535

pytest -n auto --dist loadfile --verbose --durations=15 --cov-report xml:coverage_unit.xml --cov=xradar --pyargs tests

and remove -n auto from the call.

Another thing to look after is this error:

  Traceback (most recent call last):
    File "/home/runner/work/xradar/xradar/xradar/io/backends/metek.py", line 447, in close
      for k in self._data.keys():
               ^^^^^^^^^^
  AttributeError: 'MRR2File' object has no attribute '_data'. Did you mean: 'data'?

which was issued here: https://github.com/openradar/xradar/actions/runs/10600167596/job/29376980199?pr=198

It seems, that one CI run finishes and all other raise with this error. Strange!

Running it sequentially seems to get it working, though slows down the testing a bit. Azure stores are typically memory limited at 1 GB, so I bet it was running in parallel and going up against the memory limit.

[kmuehlbauer]

Ok, so this looks like a memory leak somewhere.

This will need some profiling, I guess.

[kmuehlbauer]

It seems, that finally there is a solution to the bus error issue.

Also this PR seems good enough to get in. Thanks @rcjackson, please feel free to make any additional changes in follow-up PR's.