Merge pull request #670 from dbekaert/dev

v0.5.2

.github/workflows/build.yml

@@ -12,13 +12,13 @@ on:
 
 jobs:
   call-version-info-workflow:
-    uses: ASFHyP3/actions/.github/workflows/[email protected].0
+    uses: ASFHyP3/actions/.github/workflows/[email protected].2
     with:
       python_version: '3.10'
 
   call-docker-ghcr-workflow:
     needs: call-version-info-workflow
-    uses: ASFHyP3/actions/.github/workflows/[email protected].0
+    uses: ASFHyP3/actions/.github/workflows/[email protected].2
     with:
       version_tag: ${{ needs.call-version-info-workflow.outputs.version_tag }}
       release_branch: main

.github/workflows/changelog.yml

@@ -13,6 +13,4 @@ on:
 
 jobs:
   call-changelog-check-workflow:
-    uses: ASFHyP3/actions/.github/workflows/[email protected]
-    secrets:
-      USER_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+    uses: ASFHyP3/actions/.github/workflows/[email protected]

.github/workflows/labeled-pr.yml

@@ -12,4 +12,4 @@ on:
 
 jobs:
   call-labeled-pr-check-workflow:
-    uses: ASFHyP3/actions/.github/workflows/[email protected].0
+    uses: ASFHyP3/actions/.github/workflows/[email protected].2

.github/workflows/release.yml

@@ -7,7 +7,7 @@ on:
 
 jobs:
   call-release-workflow:
-    uses: ASFHyP3/actions/.github/workflows/[email protected].0
+    uses: ASFHyP3/actions/.github/workflows/[email protected].2
     with:
       release_prefix: RAiDER
       develop_branch: dev

.github/workflows/tag.yml

@@ -7,7 +7,7 @@ on:
 
 jobs:
   call-bump-version-workflow:
-    uses: ASFHyP3/actions/.github/workflows/[email protected].0
+    uses: ASFHyP3/actions/.github/workflows/[email protected].2
     with:
       user: dbekaert
       email: [email protected]

CHANGELOG.md

@@ -6,6 +6,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/)
 and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.5.2]
+### Changed
+* [627](https://github.com/dbekaert/RAiDER/pull/627) - Made Python datetimes timezone-aware and added unit tests and bug fixes.
+* [651](https://github.com/dbekaert/RAiDER/pull/651) - Removed use of deprecated argument to `pandas.read_csv`.
+* [652](https://github.com/dbekaert/RAiDER/pull/652) - Changed the behavior of `RAiDER.models.credentials.check_api` to not overwrite the user's API credential files.
+* [656](https://github.com/dbekaert/RAiDER/pull/656) - Example run configuration files available through `raider.py --generate_config <example_name>`.
+* [657](https://github.com/dbekaert/RAiDER/pull/657) - Fixed a few typos in `README.md`.
+* [658](https://github.com/dbekaert/RAiDER/pull/658) - Fixed opaque error message if a GUNW file is not produced while HyP3 independently against a previous INSAR_ISCE.
+* [661](https://github.com/dbekaert/RAiDER/pull/661) - Fixed bug in raiderDownloadGNSS, removed call to scipy.sum, and added unit tests.
+* [662](https://github.com/dbekaert/RAiDER/pull/662) - Bumped dem-stitcher to >= v2.5.6, which updates the URL for reading the Geoid EGM 2008.
+
 ## [0.5.1]
 ### Changed
 * Use hyp3-lib v3* to download orbits to be able to distribute load across ESA and ASF. Can be easily swapped out for `sentineleof` in future release.
@@ -28,6 +39,7 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 ## [0.4.6]
 
 ### Added
+* Added check for intermediate grid creation in _get_delays_on_cube
 * Adds an `s1_orbits.py` module which includes:
   * `get_orbits_from_slc_ids` to download the associated orbit files for a list of Sentinel-1 SLC IDs
   * `ensure_orbit_credentials` to ensure ESA CSDE credentials have been provides to download orbit files. This should be called before `sentineleof` is used to download orbits.

README.md

@@ -36,7 +36,7 @@ RAiDER does **not** currently run on arm64 processors on Mac. We will update thi
 
 ### Installing With Conda
 
-RAiDER is available on [conda-forge](https://anaconda.org/conda-forge/raider). __[Conda](https://docs.conda.io/en/latest/index.html)__ is a cross-platform way to use Python that allows you to setup and use "virtual environments." These can help to keep dependencies for different sets of code separate. We recommend using [Miniforge](https://github.com/conda-forge/miniforge), a conda environment manager that uses conda-forge as its default code repo. Alternatively,see __[here](https://docs.anaconda.com/anaconda/install/)__ for help installing Anaconda and __[here](https://docs.conda.io/en/latest/miniconda.html)__ for installing Miniconda.
+RAiDER is available on [conda-forge](https://anaconda.org/conda-forge/raider). __[Conda](https://docs.conda.io/en/latest/index.html)__ is a cross-platform way to use Python that allows you to setup and use "virtual environments." These can help to keep dependencies for different sets of code separate. We recommend using [Miniforge](https://github.com/conda-forge/miniforge), a conda environment manager that uses conda-forge as its default code repo. Alternatively, see __[here](https://docs.anaconda.com/anaconda/install/)__ for help installing Anaconda and __[here](https://docs.conda.io/en/latest/miniconda.html)__ for installing Miniconda.
 
 Installing RAiDER:
 ```
@@ -85,8 +85,8 @@ RAiDER has the ability to download weather models from third-parties; some of wh
 ## 3. Running RAiDER and Documentation
 
 For detailed documentation, examples, and Jupyter notebooks see the [RAiDER-docs repository](https://github.com/dbekaert/RAiDER-docs).
-We welcome contributions of other examples on how to leverage the RAiDER  (see [here](https://github.com/dbekaert/RAiDER/blob/dev/CONTRIBUTING.md) for instructions).
-``` raiderDelay.py -h ``` provides a help menu and list of example commands to get started.
+We welcome contributions of other examples on how to leverage the RAiDER (see [here](https://github.com/dbekaert/RAiDER/blob/dev/CONTRIBUTING.md) for instructions).  
+``` raider.py -h ``` provides a help menu and set of example run configurations to get started.
 The RAiDER scripts are highly modularized in Python and allows for building your own processing workflow.
 
 ------

environment.yml

@@ -17,7 +17,7 @@ dependencies:
  - cdsapi
  - cfgrib
  - dask
- - dem_stitcher>=2.3.1
+ - dem_stitcher>=2.5.6
  - ecmwf-api-client
  - h5netcdf
  - h5py
@@ -67,3 +67,5 @@ dependencies:
  - jupyter_contrib_nbextensions
  - jupyterlab
  - wand
+ # For development
+ - boto3-stubs

test/__init__.py

@@ -2,6 +2,8 @@
 import pytest
 import subprocess
 import shutil
+import string
+import random
 from contextlib import contextmanager
 from pathlib import Path
 
@@ -32,25 +34,30 @@ def pushd(dir):
 def update_yaml(dct_cfg:dict, dst:str='temp.yaml'):
     """ Write a new yaml file from a dictionary.
 
-    Updates parameters in the default 'raider.yaml' file.
+    Updates parameters in the default 'template.yaml' file.
     Each key:value pair will in 'dct_cfg' will overwrite that in the default
     """
     import RAiDER, yaml
 
-    template_file = os.path.join(
-                    os.path.dirname(RAiDER.__file__), 'cli', 'raider.yaml')
+    run_config_path = os.path.join(
+        os.path.dirname(RAiDER.__file__),
+        'cli',
+        'examples',
+        'template',
+        'template.yaml'
+    )
 
-    with open(template_file, 'r') as f:
+    with open(run_config_path, 'r') as f:
         try:
             params = yaml.safe_load(f)
         except yaml.YAMLError as exc:
             print(exc)
-            raise ValueError(f'Something is wrong with the yaml file {template_file}')
+            raise ValueError(f'Something is wrong with the yaml file {run_config_path}')
 
     params = {**params, **dct_cfg}
 
     with open(dst, 'w') as fh:
-        yaml.safe_dump(params, fh,  default_flow_style=False)
+        yaml.safe_dump(params, fh, default_flow_style=False)
 
     return dst
 
@@ -77,4 +84,11 @@ def makeLatLonGrid(bbox, reg, out_dir, spacing=0.1):
 
 def make_delay_name(weather_model_name, date, time, kind='ztd'):
     assert kind in 'ztd std ray'.split(), 'Incorrect type of delays.'
-    return f'{weather_model_name}_tropo_{date}T{time.replace(":", "")}_{kind}.nc'
+    return f'{weather_model_name}_tropo_{date}T{time.replace(":", "")}_{kind}.nc'
+
+
+def random_string(
+    length: int = 32,
+    alphabet: str = string.ascii_letters + string.digits
+) -> str:
+    return ''.join(random.choices(alphabet, k=length))

test/credentials/test_createFile.py

@@ -0,0 +1,75 @@
+'''
+When update_rc_file is either True or False, the relevant API RC file should be
+created if it doesn't exist.
+'''
+from typing import Tuple
+
+import pytest
+import os
+from pathlib import Path
+from platform import system
+from RAiDER.models import credentials
+from test import random_string
+
+
+def get_creds_cds(rc_path: Path) -> Tuple[str, str]:
+    import cdsapi
+    cds_credentials = cdsapi.api.read_config(rc_path)
+    uid, key = cds_credentials['key'].split(':')
+    return uid, key
+
+
+def get_creds_ecmwf(rc_path: Path) -> Tuple[str, str]:
+    import ecmwfapi
+    # Get current ECMWF API RC file path
+    old_rc_path = os.getenv("ECMWF_API_RC_FILE", ecmwfapi.api.DEFAULT_RCFILE_PATH)
+
+    # Point ecmwfapi to current dir to avoid overwriting ~/.ecmwfapirc
+    os.environ["ECMWF_API_RC_FILE"] = str(rc_path)
+    key, _, uid = ecmwfapi.api.get_apikey_values()
+
+    # Point ecmwfapi back to previous value and remove local API file
+    os.environ["ECMWF_API_RC_FILE"] = old_rc_path
+    return uid, key
+
+
+def get_creds_netrc(rc_path: Path) -> Tuple[str, str]:
+    import netrc
+    host = 'urs.earthdata.nasa.gov'
+    netrc_credentials = netrc.netrc(rc_path)
+    uid, _, key = netrc_credentials.authenticators(host)
+    return uid, key
+
+
+@pytest.mark.parametrize(
+    'model_name,get_creds',
+    (
+        ('ERA5', get_creds_cds),
+        ('ERA5T', get_creds_cds),
+        ('HRES', get_creds_ecmwf),
+        ('GMAO', get_creds_netrc),
+        ('MERRA2', get_creds_netrc)
+    )
+)
+def test_createFile(model_name, get_creds):
+    # Get the rc file's path
+    hidden_ext = '_' if system() == "Windows" else '.'
+    rc_filename = credentials.RC_FILENAMES[model_name]
+    if rc_filename is None:
+        return
+    rc_path = Path('./') / (hidden_ext + rc_filename)
+    rc_path = rc_path.expanduser()
+    rc_path.unlink(missing_ok=True)
+
+    test_uid = random_string()
+    test_key = random_string()
+
+    # Test creation of the rc file
+    credentials.check_api(model_name, test_uid, test_key, './', update_rc_file=False)
+    assert rc_path.exists(), f'{rc_path} does not exist'
+
+    # Check if API is written correctly
+    uid, key = get_creds(rc_path)
+    rc_path.unlink()
+    assert uid == test_uid, f'{rc_path}: UID was not written correctly'
+    assert key == test_key, f'{rc_path}: KEY was not written correctly'

test/credentials/test_envVars.py

@@ -0,0 +1,108 @@
+'''
+Environment variables specific to each model are accepted iff uid and key
+arguments are None.
+'''
+
+import pytest
+from pathlib import Path
+from platform import system
+from RAiDER.models import credentials
+from test import random_string
+
+@pytest.mark.parametrize(
+    'model_name,template,env_var_name_uid,env_var_name_key',
+    [
+        (
+            'ERA5', (
+                'url: https://cds.climate.copernicus.eu/api/v2\n'
+                'key: {uid}:{key}\n'
+            ),
+            'RAIDER_ECMWF_ERA5_UID',
+            'RAIDER_ECMWF_ERA5_API_KEY'
+        ),
+        (
+            'ERA5T', (
+                'url: https://cds.climate.copernicus.eu/api/v2\n'
+                'key: {uid}:{key}\n'
+            ),
+            'RAIDER_ECMWF_ERA5_UID',
+            'RAIDER_ECMWF_ERA5_API_KEY'
+        ),
+        (
+            'HRES', (
+                '{{\n'
+                '    "url"   : "https://api.ecmwf.int/v1",\n'
+                '    "key"   : "{key}",\n'
+                '    "email" : "{uid}"\n'
+                '}}\n'
+            ),
+            'RAIDER_HRES_EMAIL',
+            'RAIDER_HRES_API_KEY'
+        ),
+        (
+            # Simulate a .netrc file with multiple sets of credentials.
+            # The ones not for urs.earthdata.nasa.gov should NOT be touched.
+            # Indentation is done with TABS, as that is what the netrc package
+            # generates.
+            'GMAO', (
+                'machine example.com\n'
+                '	login johndoe\n'
+                '	password hunter2\n'
+                'machine urs.earthdata.nasa.gov\n'
+                '	login {uid}\n'
+                '	password {key}\n'
+                'machine 127.0.0.1\n'
+                '	login bobsmith\n'
+                '	password dolphins\n'
+            ),
+            'EARTHDATA_USERNAME',
+            'EARTHDATA_PASSWORD'
+        ),
+        (
+            'MERRA2', (
+                'machine example.com\n'
+                '	login johndoe\n'
+                '	password hunter2\n'
+                'machine urs.earthdata.nasa.gov\n'
+                '	login {uid}\n'
+                '	password {key}\n'
+                'machine 127.0.0.1\n'
+                '	login bobsmith\n'
+                '	password dolphins\n'
+            ),
+            'EARTHDATA_USERNAME',
+            'EARTHDATA_PASSWORD'
+        ),
+    ]
+)
+def test_envVars(
+    monkeypatch,
+    model_name,
+    template,
+    env_var_name_uid,
+    env_var_name_key
+):
+    hidden_ext = '_' if system() == "Windows" else '.'
+    rc_filename = credentials.RC_FILENAMES[model_name]
+    if rc_filename is None:
+        return
+    rc_path = Path('./') / (hidden_ext + rc_filename)
+    rc_path = rc_path.expanduser()
+    rc_path.unlink(missing_ok=True)
+
+    test_uid = random_string()
+    test_key = random_string()
+
+    with monkeypatch.context() as mp:
+        mp.setenv(env_var_name_uid, test_uid)
+        mp.setenv(env_var_name_key, test_key)
+        credentials.check_api(model_name, None, None, './', update_rc_file=False)
+
+    expected_content = template.format(uid=test_uid, key=test_key)
+    actual_content = rc_path.read_text()
+    rc_path.unlink()
+
+    assert (
+        expected_content == actual_content,
+        f'{rc_path} was not updated correctly'
+    )

test/credentials/test_updateFalse.py

@@ -0,0 +1,31 @@
+'''
+When update_rc_file is False, the RC file should NOT be modified if it already
+exists.
+'''
+import pytest
+
+from pathlib import Path
+from platform import system
+from RAiDER.models import credentials
+
+
+@pytest.mark.parametrize('model_name', 'ERA5 ERA5T HRES GMAO MERRA2'.split())
+def test_updateFalse(model_name):
+    # Get the rc file's path
+    hidden_ext = '_' if system() == "Windows" else '.'
+    rc_filename = credentials.RC_FILENAMES[model_name]
+    if rc_filename is None:
+        return
+    rc_path = Path('./') / (hidden_ext + rc_filename)
+    rc_path = rc_path.expanduser()
+
+    # Write some example text to test for
+    rc_path.write_text('dummy')
+
+    # Test creation of this model's RC file in current dir
+    credentials.check_api(model_name, None, None, './', update_rc_file=False)
+
+    # Assert the content was unchanged
+    content = rc_path.read_text()
+    rc_path.unlink()
+    assert content == 'dummy', f'{rc_path} was modified'