ADD: Auto Read Function #205

syedhamidali · 2024-09-02T23:38:36Z

Closes #xxxx
Tests added
Changes are documented in history.md

Description: Enhanced Radar File Reading Functionality with Timeout Support

Summary:

This PR introduces the auto_read.py module, which provides an enhanced read function designed to improve the radar file reading process in the xradar library. The read function iterates over all available file-opening functions within xradar.io to attempt to read radar files, with added support for georeferencing and verbose output. A key feature of this function is its ability to handle timeouts, ensuring that long-running file reads do not block execution indefinitely.

Key Features:

Automatic Format Detection: The read function attempts to open the provided radar file using all available open_ functions in the xradar.io module, automatically selecting the correct file format.
Timeout Support: The function now includes an optional timeout parameter. If specified, the file read operation is wrapped in a timeout handler, raising a TimeoutException if the read operation exceeds the allowed time.
Georeferencing: The function can automatically georeference the radar data by adding x, y, z coordinates if the georeference parameter is set to True.
Verbose Mode: A verbose option is available to print detailed processing information, aiding in debugging and providing insights into which file-opening method was successful.
Custom Exception Handling: The function raises a ValueError if the file cannot be opened by any supported format, providing clear feedback in cases of unsupported file types or corrupted files.

Usage:

Basic Usage:
```
dtree = xd.io.read(file_path)
```

With Timeout and Verbose Output:

dtree =  xd.io.read(file_path, verbose=True, timeout=10)

Other tests

import os
from open_radar_data import DATASETS

# Dictionary to track which prefixes have been processed
prefix_processed = {}

# Iterate through the files in the registry
for file in DATASETS.registry_files:
#     print(file)

    # Skip files with .gz extension as they are not supported by xradar
    if file.endswith('.gz'):
        continue

    # Extract the prefix of the file (up to the first underscore, dot, or space)
    prefix = file.split('_')[0].split('.')[0].split()[0]

    # Check if a file from this prefix group has already been processed
    if prefix in prefix_processed:
        continue  # Skip the file if a file from this group has already been processed

    try:
        # Attempt to fetch the file path
        filepath = DATASETS.fetch(file)
        print(f"Reading: {os.path.basename(file)}")
        # Attempt to read the radar data
        radar = xd.io.read(filepath, timeout=10)
        
        # If no exception is raised, print success
        print('+++++++++++')
        print('| SUCCESS |')
        print('+++++++++++\n')

        # Mark this prefix as processed
        prefix_processed[prefix] = True

        # Delete the radar object after processing
        del radar

    except Exception as e:
        # Print the error and continue with the next file
        print(f"Error processing file {os.path.basename(file)}: \n{e}")
        print('+++++++++++')
        print('| FAILED |')
        print('+++++++++++\n')
        continue

Reading: 110635.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: 2013051000000600dBZ.vol
+++++++++++
| SUCCESS |
+++++++++++

Reading: 20220628072500_savevol_COSMO_LOOKUP_TEMP.nc
Error processing file 20220628072500_savevol_COSMO_LOOKUP_TEMP.nc: 
File could not be opened by any supported format in xradar.io.
+++++++++++
| FAILED |
+++++++++++

Reading: 71_20181220_060628.pvol.h5
+++++++++++
| SUCCESS |
+++++++++++

Reading: DES_VOL_RAW_20240522_1600.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: DWD-Vol-2_99999_20180601054047_00.h5
+++++++++++
| SUCCESS |
+++++++++++

Reading: KATX20130717_195021_V06
+++++++++++
| SUCCESS |
+++++++++++

Reading: KLBB20160601_150025_V06
+++++++++++
| SUCCESS |
+++++++++++

Reading: MLA2119412050U.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: MLL2217907250U.003.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: RAW_NA_000_125_20080411181219
+++++++++++
| SUCCESS |
+++++++++++

Reading: SUR210819000227.RAWKPJV
+++++++++++
| SUCCESS |
+++++++++++

Reading: T_PAGZ35_C_ENMI_20170421090837.hdf
+++++++++++
| SUCCESS |
+++++++++++

Reading: User1_184_20240601_013257.hpl
+++++++++++
| SUCCESS |
+++++++++++

Reading: Z__C_RJTD_20230801200000_RDR_JMAGPV_RS47937_Gar0p250km0p35deg_PRkdp_N11_ANAL_cfrad.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: cfrad.20080604_002217_000_SPOL_v36_SUR.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: convsf.19990811.221202.cdf
Error processing file convsf.19990811.221202.cdf: 
File could not be opened by any supported format in xradar.io.
+++++++++++
| FAILED |
+++++++++++

Reading: cor-main131125105503.RAW2049
+++++++++++
| SUCCESS |
+++++++++++

Reading: corcsapr2cmacppiM1.c1.20181111.030003.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: csapr.nc
Error processing file csapr.nc: 
File could not be opened by any supported format in xradar.io.
+++++++++++
| FAILED |
+++++++++++

Reading: ddop.910815.212303.cdf
Error processing file ddop.910815.212303.cdf: 
File could not be opened by any supported format in xradar.io.
+++++++++++
| FAILED |
+++++++++++

Reading: ddop.910815.213931.cdf
Error processing file ddop.910815.213931.cdf: 
File could not be opened by any supported format in xradar.io.
+++++++++++
| FAILED |
+++++++++++

Reading: ddop.910815.214454.cdf
Error processing file ddop.910815.214454.cdf: 
File could not be opened by any supported format in xradar.io.
+++++++++++
| FAILED |
+++++++++++

Reading: ddop.910815.215545.cdf
Error processing file ddop.910815.215545.cdf: 
File could not be opened by any supported format in xradar.io.
+++++++++++
| FAILED |
+++++++++++

Reading: example_plot_ppi_single_sweep.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: gucxprecipradarcmacppiS2.c1.20220314.021559.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: houkasacrcfrM1.a1.20210922.150006.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: sample_rainbow_5_59.vol
+++++++++++
| SUCCESS |
+++++++++++

Reading: sgpkasacrcrrasterC1.a1.20130419.012153.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: sgpkazrgeC1.a1.20190529.000002.cdf
Error processing file sgpkazrgeC1.a1.20190529.000002.cdf: 
File could not be opened by any supported format in xradar.io.
+++++++++++
| FAILED |
+++++++++++

Reading: sgpxsaprcfrvptI4.a1.20200205.100827.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: swx_20120520_0641.nc
+++++++++++
| SUCCESS |
+++++++++++

codecov · 2024-09-03T02:29:49Z

Codecov Report

Attention: Patch coverage is 93.33333% with 5 lines in your changes missing coverage. Please review.

Project coverage is 92.10%. Comparing base (c998f25) to head (6d878bd).

Files with missing lines	Patch %	Lines
xradar/io/auto_read.py	93.24%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #205      +/-   ##
==========================================
+ Coverage   91.97%   92.10%   +0.12%     
==========================================
  Files          23       24       +1     
  Lines        4573     4648      +75     
==========================================
+ Hits         4206     4281      +75     
  Misses        367      367

Flag	Coverage Δ
notebooktests	`78.98% <49.33%> (+0.73%)`	⬆️
unittests	`90.27% <92.00%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

review-notebook-app · 2024-09-04T06:14:48Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

kmuehlbauer · 2024-10-17T09:53:56Z

@syedhamidali Checking if a file can be opened with one of our backends and return the dataset/datatree is indeed a nice idea and a useful feature. Thanks for tackling this to make xradar more user-friendly.

For the implementation we might think about using the guess_can_open-method of each BackendEntrypoint.

We would have to implement such function for our backends first, but this would be a great opportunity to do this finally.

In the long run we should think about implementing xradar.open_dataset() and xradar.open_datatree() which do essentially the same as the xarray counterparts, but solely iterates over xradar backends (when engine=None). This would also remove the need of a new read-function.

@openradar/xradar More thoughts here?

syedhamidali · 2024-10-17T18:20:16Z

@kmuehlbauer That’s a great idea, especially in terms of aligning with the xarray model. I do have a few questions, though. What would you typically open as a dataset—would it be a single sweep? In our case, we often work with datatree as the file structure. I was thinking of implementing something like open_datatree and open_mfdatatree or read_volume, where the node names would follow a /volume_{}/sweep_{} pattern.

I’ve done something similar with the imd_data model in radarx. You can check out an example at radarx-rtd. Moving forward, the read_imd function will be integrated into the xradar package, and the radarx package itself will be contributed to the openradar community. I’ve been focusing on radarx primarily for visualization (both traditional and interactive) and potentially some microphysics retrievals, and I’d be happy to contribute these efforts.

kmuehlbauer · 2024-10-18T07:58:56Z

@syedhamidali Thanks for sharing your efforts here. Great to hear that the IMD backend will find it's way into xradar.

I like the idea behind radarx. Did you conceive the package more as companion to xradar being capable of things mentioned in #174? Or should it play in the same league as Py-ART, wradlib etc implementing algorithms for some specific purposes? Maybe split this out to it's own issue for better visibility and discussion?

syedhamidali · 2024-10-22T18:00:20Z

@kmuehlbauer Sorry for the late response, as I am currently at the AMS SLS conference. radarx will indeed act as a companion to xradar, addressing most of the features mentioned in that comment. However, I’m also considering adding additional functionality such as retrieving DSDs and performing thermodynamic calculations like the evaporative cooling rate etc. For this, we would need temperature profiles, which could be sourced from ERA5 data or nearby soundings.

ADD: Auto Read Function

a4d69b9

syedhamidali added the enhancement New feature or request label Sep 3, 2024

syedhamidali self-assigned this Sep 3, 2024

FIX: Linting issues

2a77967

syedhamidali added 5 commits September 2, 2024 23:08

FIX: auto_read tests

0599ec7

FIX: Test Coverage

6ba967c

FIX: Test 2 with cov

1e136ac

FIX: Coverge Test 3

2de6ac7

ADD: Example notebook for auto read

01b270b

syedhamidali added 3 commits September 4, 2024 02:18

ADD: Example notebook for auto read + Linting

fdcb75e

FIX: Linting issue with notebook

b7412b4

FIX: Fix doc of auto_read

fdca6ab

syedhamidali requested a review from kmuehlbauer September 4, 2024 07:24

syedhamidali and others added 2 commits September 8, 2024 17:51

ADD: Added Notebook to docs/usage.md

e9a10c6

Merge branch 'main' into auto-read

4ebe4bc

syedhamidali requested a review from mgrover1 September 19, 2024 15:31

syedhamidali added 2 commits September 20, 2024 10:03

Merge branch 'main' into auto-read

4c4616c

Merge branch 'main' into auto-read

6d878bd

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ADD: Auto Read Function #205

ADD: Auto Read Function #205

syedhamidali commented Sep 2, 2024 •

edited

Loading

codecov bot commented Sep 3, 2024 •

edited

Loading

review-notebook-app bot commented Sep 4, 2024

kmuehlbauer commented Oct 17, 2024

syedhamidali commented Oct 17, 2024

kmuehlbauer commented Oct 18, 2024

syedhamidali commented Oct 22, 2024

ADD: Auto Read Function #205

Are you sure you want to change the base?

ADD: Auto Read Function #205

Conversation

syedhamidali commented Sep 2, 2024 • edited Loading

Description: Enhanced Radar File Reading Functionality with Timeout Support

Summary:

Key Features:

Usage:

Other tests

codecov bot commented Sep 3, 2024 • edited Loading

Codecov Report

review-notebook-app bot commented Sep 4, 2024

kmuehlbauer commented Oct 17, 2024

syedhamidali commented Oct 17, 2024

kmuehlbauer commented Oct 18, 2024

syedhamidali commented Oct 22, 2024

syedhamidali commented Sep 2, 2024 •

edited

Loading

codecov bot commented Sep 3, 2024 •

edited

Loading