Port the GSI to Acorn

[DavidHuber-NOAA]

Description

This ports the GSI to Acorn to allow testing of new library installations before official installations go to Cactus and Dogwood. This is a prerequisite (but does not fix) for #642, #662, and #665.

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?
Ctests were run on Acorn. I do not currently have rstprod access on the system, so only the HAFS and RTMA tests passed.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• New and existing tests pass with my changes

[RussTreadon-NOAA]

Install and build DavidHuber-NOAA:feature/acorn at 542b6f7 on Acorn. Run ctests with the following results

Test project /lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/acorn/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_rdasens
    Start 4: hafs_4denvar_glbens
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  727.26 sec
2/6 Test #6: global_enkf ......................   Passed  851.52 sec
3/6 Test #2: rtma .............................   Passed  968.93 sec
4/6 Test #5: hafs_3denvar_hybens ..............   Passed  1153.76 sec
5/6 Test #4: hafs_4denvar_glbens ..............   Passed  1213.47 sec
6/6 Test #1: global_4denvar ...................   Passed  1682.33 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) = 1682.34 sec

All tests pass!

Note: Use DavidHuber-NOAA:feature/acorn as updat and contrl.

[aerorahul]

some requested changes in gsi_acornn.intel.lua and optional comments on sub_acorn.

[RussTreadon-NOAA]

@DavidHuber-NOAA , @aerorahul is on leave until 11/4. Is there anyone else who can complete the second peer review?

Today I will install DavidHuber-NOAA:feature/acorn on Cactus, Hera, Hercules, & Orion to run ctests.

[DavidHuber-NOAA]

@BinLiu-NOAA @ZhanZhang-NOAA Since you both had helped with the original HAFS GSI port to Acorn, would one of you mind reviewing this PR?

[RussTreadon-NOAA]

Cactus (WCOSS2) ctest results
Install develop at 665f9b4 and DavidHuber-NOAA:feature/acorn at b0aa72d on Orion. Run ctests with following results

Test project /lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/pr793/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_rdasens
    Start 4: hafs_4denvar_glbens
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  1332.26 sec
2/6 Test #6: global_enkf ......................   Passed  1337.77 sec
3/6 Test #5: hafs_3denvar_hybens ..............   Passed  1641.24 sec
4/6 Test #4: hafs_4denvar_glbens ..............   Passed  1882.29 sec
5/6 Test #2: rtma .............................   Passed  2532.07 sec
6/6 Test #1: global_4denvar ...................   Passed  2644.91 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) = 2644.92 sec

Hera ctest results
Repeat the above installation on Hera. Run ctests with the following results

Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/pr793/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_rdasens
    Start 4: hafs_4denvar_glbens
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  494.59 sec
2/6 Test #6: global_enkf ......................   Passed  760.73 sec
3/6 Test #2: rtma .............................   Passed  973.52 sec
4/6 Test #5: hafs_3denvar_hybens ..............   Passed  1108.23 sec
5/6 Test #4: hafs_4denvar_glbens ..............   Passed  1170.90 sec
6/6 Test #1: global_4denvar ...................   Passed  1804.89 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) = 1804.93 sec

Hercules ctest results
Repeat the above installation on Hercules. Run ctests with the following results

Test project /work/noaa/da/rtreadon/git/gsi/pr793/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_rdasens
    Start 4: hafs_4denvar_glbens
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  605.12 sec
2/6 Test #6: global_enkf ......................   Passed  905.35 sec
3/6 Test #2: rtma .............................   Passed  1325.60 sec
4/6 Test #5: hafs_3denvar_hybens ..............   Passed  1392.84 sec
5/6 Test #4: hafs_4denvar_glbens ..............   Passed  1522.91 sec
6/6 Test #1: global_4denvar ...................   Passed  1923.31 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) = 1923.32 sec

Orion ctest results
Repeat the above installation on Orion. Run ctests with the following results

Test project /work2/noaa/da/rtreadon/git/gsi/pr793/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_rdasens
    Start 4: hafs_4denvar_glbens
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  1094.18 sec
2/6 Test #6: global_enkf ......................   Passed  1279.48 sec
3/6 Test #2: rtma .............................   Passed  1749.78 sec
4/6 Test #5: hafs_3denvar_hybens ..............   Passed  3081.64 sec
5/6 Test #4: hafs_4denvar_glbens ..............***Failed  3383.25 sec
6/6 Test #1: global_4denvar ...................***Failed  4023.81 sec

67% tests passed, 2 tests failed out of 6

Total Test time (real) = 4023.82 sec

The following tests FAILED:
          1 - global_4denvar (Failed)
          4 - hafs_4denvar_glbens (Failed)
Errors while running CTest

global_4denvar failed due to

The runtime for global_4denvar_hiproc_updat is 1041.124062 seconds.  This has exceeded maximum allowable threshold time of 801.522912 seconds,
resulting in Failure of timethresh2 the regression test.

hafs_4denvar_glbens failed for the same reason

The runtime for hafs_4denvar_glbens_loproc_updat is 978.292937 seconds.  This has exceeded maximum allowable threshold time of 814.187132 seconds,
resulting in Failure time-thresh of the regression test.

Neither of these are fatal failures. A rerun of the ctests yields a Passed result for all tests

Test project /work/noaa/da/rtreadon/git/gsi/pr793/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_rdasens
    Start 4: hafs_4denvar_glbens
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
1/6 Test #6: global_enkf ......................   Passed  727.86 sec
2/6 Test #3: rrfs_3denvar_rdasens .............   Passed  966.84 sec
3/6 Test #2: rtma .............................   Passed  1747.13 sec
4/6 Test #5: hafs_3denvar_hybens ..............   Passed  3141.75 sec
5/6 Test #4: hafs_4denvar_glbens ..............   Passed  3201.12 sec
6/6 Test #1: global_4denvar ...................   Passed  3721.94 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) = 3721.95 sec

Summary
All ctests eventually Passed on Cactus, Hera, Hercules, and Orion. This is an expected results since this PR does not modify source code. Nor is the build on these machines altered by this PR.

[RussTreadon-NOAA]

Approve.

[RussTreadon-NOAA]

@DavidHuber-NOAA , if you have addressed @aerorahul 's review comments, please go ahead and resolve the conversations. Thanks.

ctests have been run on all platforms with expected results. Once we get a second peer review and approval we can schedule this PR for merger into develop.

[RussTreadon-NOAA]

@BinLiu-NOAA @ZhanZhang-NOAA : If you have time, your review of this PR is much appreciated. Closure of this PR is a prerequisite to making progress on several open GSI issues. Hence my reaching out to you.