Add GEFS regression test suite from EP5r2 configuration/case

[NickSzapiro-NOAA]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• Commit 'test_changes.list' from previous step

---

Description:

This PR updates the cpld_bmark_p8 tests to a prototype GEFS test case of fully coupled s2swa+IAU+stochastics in atmosphere and ocean, with configuration and warm starts from restarts of EP5r2 ensemble member 1 for 2021-03-25 06Z. The EP5r2 test case was kindly provided by @bingfu-NOAA via @junwang-noaa with aerosol input data and configurations from @lipan-NOAA.

A separate INPUTDATA_ROOT_BMIC is no longer needed and is removed.

The regression test suite samples basic reproducibility/quality checks, particularly:

• control reproduces itself
• restart reproduces control
• changing number of tasks reproduces control
• changing number of threads reproduces control
• Intel debug version reproduces itself
• GNU debug version reproduces itself

All tests do not pass across all platforms, summarized in this regression test suite matrix:

and summary slides on pre-test and common issues.

Some tests fail in common. This commit helps share reproducers to follow up on remaining issues. User needs to uncomment these tests to run. Failures may require library/platform support. Hopefully committing this test suite as work in progress facilitates collaborative development particularly in:

• Platforms with GNU support for coupled model
• Sensitivity to spack-stack updates (including dcp failure & splitting ESMF_MeshCreate in WW3 and intel-debug failure and HDF update
• Derecho test failures

Note that there are two intentional differences from GEFS workflow configuration (please inform if you see other differences): 1) aerosols are 1-way coupled in diagnostic mode and 2) wave element mask has been modified as discussed in NOAA-EMC/WW3#1328.

In the future, depending on aerosol coupling, GOCART .rc files and ExtData directory structure may be revised for consistency with global-workflow. This benchmark configuration and case may be updated as well, particularly with GEFS reforecast or UFS case study.

TODO: Scripts need finalizing once filepaths are in shared space.
Input data is currently in user space on hera:
/scratch1/NCEPDEV/nems/Nick.Szapiro/tasks/input_data/gefs.v13/RT_GEFS/

Commit Message:

* UFSWM - Add GEFS regression test suite from EP5r2 configuration/case

Priority:

• Normal

Git Tracking

UFSWM:

• Closes Update cpld_bmark_p8 with GEFSv13 EP5 configuration #1987
• Closes Add RT test for gocart_on, gccpp_on, nasa_on #1818

Sub component Pull Requests:

• None

UFSWM Blocking Dependencies:

• None

---

Changes

Regression Test Changes (Please commit test_changes.list):

• PR Adds New Tests/Baselines.

Input data Changes:

• New input data.

Library Changes/Upgrades:

• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • Jet
  • Gaea
  • Derecho
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[edwardhartnett]

Are the versions of PIO, netcdf-fortran, netcdf-c, and HDF5 all the same on hercules as on other platforms?

[NickSzapiro-NOAA]

Are the versions of PIO, netcdf-fortran, netcdf-c, and HDF5 all the same on hercules as on other platforms?

Hi @edwardhartnett. Hercules is maybe the only platform reliably running gnu tests right now. I see the same gnu and intel libraries via spack-stack-1.6.0 on hercules:
gcc/12.2.0: parallelio-2.5.10 parallel-netcdf-1.12.2 netcdf-fortran-4.6.1 netcdf-c-4.9.2 hdf5-1.14.0
intel/2021.9.0: parallelio-2.5.10 parallel-netcdf-1.12.2 netcdf-fortran-4.6.1 netcdf-c-4.9.2 hdf5-1.14.0

[edwardhartnett]

OK, I thought we had switched to hdf5-1.14.3?

Some parallel I/O bugs were fixed in HDF5-1.14.3.

[NickSzapiro-NOAA]

OK, I thought we had switched to hdf5-1.14.3?

Some parallel I/O bugs were fixed in HDF5-1.14.3.

@edwardhartnett We can test with updated packages. What versions would you suggest?

[edwardhartnett]

hdf5-1.14.3

[DeniseWorthen]

This is a pretty big departure from how tests are set up currently. The advantage of the current system is that you can just diff two tests and more easily see the differences from defaults. Here the non-default settings are one-step removed from the default_vars. The disadvantage w/ the current way we do is is that you have a lot of settings carried over. @BrianCurtis-NOAA do you have any comments?

[NickSzapiro-NOAA]

Sure. Maybe I should mention this more follows the ORTs:
https://github.com/ufs-community/ufs-weather-model/blob/develop/tests/opnReqTests/dcp.sh
but it's not clear current ORT setup is being/will be maintained