Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Additional fixes for downstream jobs #3187

Merged
merged 37 commits into from
Jan 14, 2025
Merged

Additional fixes for downstream jobs #3187

merged 37 commits into from
Jan 14, 2025

Conversation

DavidHuber-NOAA
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA commented Dec 20, 2024 •
edited by KateFriedman-NOAA
Loading

Description

This fixes a number of issues with the postsnd and arch jobs discovered during testing of HR4.

Resolves #3019

Type of change

  • Bug fix (fixes something broken)

Change characteristics

How has this been tested?

  • Extended test on Hera
  • Extended test on WCOSS2

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
    • Portions of this change are covered by existing CI, but others are not (e.g. archiving, which is disabled on WCOSS2)
    • Updates will need to be made to run the AWIPS jobs in the existing C96_atm3DVar_extended test

@DavidHuber-NOAA
Only archive gempak if it was run
5974b96
@DavidHuber-NOAA
Remove literal quotes
2649a86
@DavidHuber-NOAA
Shellcheck stuff
a038cb9
@DavidHuber-NOAA
Remove duplicate marinebmat entry
1180627
@DavidHuber-NOAA
Add DO_GEMPAK to archive environment
1efe711
@DavidHuber-NOAA
Merge remote-tracking branch 'emc/develop' into fix/awips
4b7d39e 
* emc/develop:
  Change orog gravity wave drag scheme for grid sizes less than 10km (NOAA-EMC#3175)
  Switch snow DA to use 2DVar for deterministic and ensemble mean (NOAA-EMC#3163)
  Update compression options for GEFS history files (NOAA-EMC#3184)
@DavidHuber-NOAA
Copy link
Contributor Author

All post jobs now run for the C96_atm3dvar_extended test when DO_AWIPS=YES on WCOSS2. I have thus enabled AWIPS for this test to ensure the AWIPS jobs and linked downstream jobs are always run to avoid lapses in support.

The AWIPS jobs still cannot be enabled on other machines due to the fbwinds executable which currently only runs on WCOSS2. That will be an issue for another time.

Also, during this testing, I found that the MOS jobs, when enabled, do not run on WCOSS2. I opened issue #3196 to look into this, but I believe the MOS package will need an overhaul to work with the new COM structure.

Marking this PR ready for review.

@DavidHuber-NOAA DavidHuber-NOAA marked this pull request as ready for review December 30, 2024 19:01
@DavidHuber-NOAA DavidHuber-NOAA mentioned this pull request Dec 30, 2024
Merged
22 tasks
aerorahul
aerorahul reviewed Dec 31, 2024
View reviewed changes
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Just one observation. Approve.

workflow/applications/gfs_cycled.py Show resolved Hide resolved
aerorahul
aerorahul approved these changes Jan 2, 2025
View reviewed changes
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.

aerorahul
aerorahul previously approved these changes Jan 3, 2025
View reviewed changes
@aerorahul
Copy link
Contributor

@WalterKolczynski-NOAA
Can you please run the ci on wcoss2?

@DavidHuber-NOAA DavidHuber-NOAA added CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS labels Jan 6, 2025
@DavidHuber-NOAA DavidHuber-NOAA added CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS and removed CI-Wcoss2-Failed **Bot use only** CI testing on WCOSS for this PR has failed labels Jan 13, 2025
@emcbot emcbot added the CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS label Jan 13, 2025
@KateFriedman-NOAA KateFriedman-NOAA removed the CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS label Jan 13, 2025
@emcbot emcbot added CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS labels Jan 13, 2025
@emcbot
Copy link

emcbot commented Jan 13, 2025

CI Tests set up to run in /lfs/h2/emc/ptmp/emc.global/PR/PR_3187/RUNTESTS on WCOSS

@DavidHuber-NOAA
Copy link
Contributor Author

It looks like the gdas_gempakmetancdc job failed on WCOSS2. I'm investigating.

@DavidHuber-NOAA DavidHuber-NOAA added CI-Wcoss2-Failed **Bot use only** CI testing on WCOSS for this PR has failed and removed CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress labels Jan 14, 2025
@KateFriedman-NOAA KateFriedman-NOAA added CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Failed **Bot use only** CI testing on WCOSS for this PR has failed labels Jan 14, 2025
@KateFriedman-NOAA
Copy link
Member

KateFriedman-NOAA commented Jan 14, 2025 •
edited
Loading

Manually pulled in fixes to CI clone on WCOSS2 and reran failed jobs. Failing CI test completed successfully on WCOSS2, as did others.

Tue Jan 14 16:46:41 UTC 2025
******** C48_ATM_3187 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Jan 13 2025 19:55:31    Jan 13 2025 21:05:25
202103231800        Done    Jan 13 2025 19:55:31    Jan 13 2025 21:15:34

******** C48_S2SW_3187 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Jan 13 2025 19:55:34    Jan 13 2025 21:40:34
202103231800        Done    Jan 13 2025 19:55:34    Jan 13 2025 21:50:29

******** C48_S2SWA_gefs_3187 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Jan 13 2025 19:55:39    Jan 13 2025 22:50:36

******** C96_atm3DVar_extended_3187 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Jan 13 2025 19:55:43    Jan 13 2025 20:15:47
202112210000        Done    Jan 13 2025 19:55:43    Jan 14 2025 16:30:40
202112210600        Done    Jan 13 2025 19:55:43    Jan 14 2025 01:10:39
202112211200        Done    Jan 13 2025 20:20:54    Jan 14 2025 16:35:34
202112211800        Done    Jan 14 2025 01:15:37    Jan 14 2025 06:05:49

******** C96C48_hybatmDA_3187 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Jan 13 2025 19:55:46    Jan 13 2025 20:15:51
202112210000        Done    Jan 13 2025 19:55:46    Jan 13 2025 22:25:49
202112210600        Done    Jan 13 2025 19:55:46    Jan 13 2025 22:15:52

******** C96_S2SWA_gefs_replay_ics_3187 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202011010000        Done    Jan 13 2025 19:55:49    Jan 13 2025 20:35:54

@DavidHuber-NOAA DavidHuber-NOAA mentioned this pull request Jan 14, 2025
Open
@DavidHuber-NOAA
Copy link
Contributor Author

I added some fixes for the gfs_waveawipsbulls job, but I was unable to test the job completely due to missing buoy data that should be generated by the gfs_wavepostpnt job. This is documented in #3226. Once this issue is resolved in a future PR, the new C48_S2SW_extended test can be enabled on WCOSS2.

@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit aea82a8 into NOAA-EMC:develop Jan 14, 2025
5 checks passed
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this pull request Jan 15, 2025
@KateFriedman-NOAA
Merge remote-tracking branch 'upstream/develop' into feature/jgdas_en…
583e738 
…kf_sfc_update_com_in_out

* upstream/develop:
  Resolve bug with LMOD_TMOD_FIND_FIRST setting affecting build on WCOSS2 (NOAA-EMC#3229)
  Reinstate product groups (NOAA-EMC#3208)
  Additional fixes for downstream jobs (NOAA-EMC#3187)
  Turn IAU off during staging job for cold start experiments (NOAA-EMC#3215)
  Update the gdas.cd hash and enable GDASApp to run on WCOSS2 (NOAA-EMC#3220)
  Update upload-artifact to v4 (NOAA-EMC#3216)
  Prevent duplicate case generation in generate_workflows.sh (NOAA-EMC#3217)
  Update g-w to cycle with C1152 ATM (NOAA-EMC#3206)
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this pull request Jan 15, 2025
@KateFriedman-NOAA
Merge remote-tracking branch 'upstream/develop' into feature/jgdas_en…
3f7d555 
…kf_sfc_update_com_in_out

* upstream/develop:
  Resolve bug with LMOD_TMOD_FIND_FIRST setting affecting build on WCOSS2 (NOAA-EMC#3229)
  Reinstate product groups (NOAA-EMC#3208)
  Additional fixes for downstream jobs (NOAA-EMC#3187)
  Turn IAU off during staging job for cold start experiments (NOAA-EMC#3215)
  Update the gdas.cd hash and enable GDASApp to run on WCOSS2 (NOAA-EMC#3220)
  Update upload-artifact to v4 (NOAA-EMC#3216)
  Prevent duplicate case generation in generate_workflows.sh (NOAA-EMC#3217)
  Update g-w to cycle with C1152 ATM (NOAA-EMC#3206)
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this pull request Jan 15, 2025
@KateFriedman-NOAA
Merge remote-tracking branch 'upstream/develop' into feature/jgdas_en…
02e810e 
…kf_sfc_update_com_in_out

* upstream/develop:
  Resolve bug with LMOD_TMOD_FIND_FIRST setting affecting build on WCOSS2 (NOAA-EMC#3229)
  Reinstate product groups (NOAA-EMC#3208)
  Additional fixes for downstream jobs (NOAA-EMC#3187)
  Turn IAU off during staging job for cold start experiments (NOAA-EMC#3215)
  Update the gdas.cd hash and enable GDASApp to run on WCOSS2 (NOAA-EMC#3220)
  Update upload-artifact to v4 (NOAA-EMC#3216)
  Prevent duplicate case generation in generate_workflows.sh (NOAA-EMC#3217)
  Update g-w to cycle with C1152 ATM (NOAA-EMC#3206)
tsga added a commit to tsga/global-workflow that referenced this pull request Jan 22, 2025
@tsga
Merge branch 'develop' into feature/lnd_regrid_iau
dbebb2e 
* develop:
  Add echgres as a dependency only for RUN=enkfgdas, not enkfgfs (NOAA-EMC#3246)
  Add domain level to wave gridded COM path (NOAA-EMC#3137)
  CI JJOB Tests using CMake (NOAA-EMC#3214)
  Make assorted updates to waves (NOAA-EMC#3190)
  Move WCOSS2 LD_LIBRARY_PATH patches to load_ufsda_modules.sh (NOAA-EMC#3236)
  Adding a gefs_arch task to GEFS workflow (NOAA-EMC#3211)
  Add additional GEFS variables needed for AI/ML applications  (NOAA-EMC#3221)
  Add bmat task dependency to marine LETKF task (NOAA-EMC#3224)
  Resolve bug with LMOD_TMOD_FIND_FIRST setting affecting build on WCOSS2 (NOAA-EMC#3229)
  Reinstate product groups (NOAA-EMC#3208)
  Additional fixes for downstream jobs (NOAA-EMC#3187)
  Turn IAU off during staging job for cold start experiments (NOAA-EMC#3215)
  Update the gdas.cd hash and enable GDASApp to run on WCOSS2 (NOAA-EMC#3220)
  Update upload-artifact to v4 (NOAA-EMC#3216)
  Prevent duplicate case generation in generate_workflows.sh (NOAA-EMC#3217)
  Update g-w to cycle with C1152 ATM (NOAA-EMC#3206)
  Separate use of initial increment/perturbation file from REPLAY/+03 ICs  (NOAA-EMC#3119)
  Update gsi_enkf hash and gsi_ver (NOAA-EMC#3207)
  Remove cpus-per-task from APRUN_OCNANALECEN on WCOSS2 (NOAA-EMC#3212)
  Remove 5WAVH from AWIPS GRIB2 parm files (NOAA-EMC#3146)
  Remove multi-grid wave support (NOAA-EMC#3188)
  Add echgres as a dependency for earc (NOAA-EMC#3202)
danholdaway added a commit to danholdaway/global-workflow that referenced this pull request Jan 27, 2025
@danholdaway
Merge branch 'develop' into feature/jedi_vs_gsi
fd62a43 
* develop:
  Remove WAFS files and references from `develop` (NOAA-EMC#3263)
  fix intel stack version number on c5 (NOAA-EMC#3258)
  Update gsi_monitor and ufs_utils hashes to recent hashes for C5/C6 build and run (NOAA-EMC#3252)
  Enable DA cycling on gaea C5/C6 (NOAA-EMC#3255)
  Copy post-processed sea ice increment for diagnostics (NOAA-EMC#3235)
  Only run METplus in the 3Dvar tests (NOAA-EMC#3245)
  Clone, build, and run C48_ATM and C48_S2SW on Gaea C5 and C6 (NOAA-EMC#3106)
  Add echgres as a dependency only for RUN=enkfgdas, not enkfgfs (NOAA-EMC#3246)
  Add domain level to wave gridded COM path (NOAA-EMC#3137)
  CI JJOB Tests using CMake (NOAA-EMC#3214)
  Make assorted updates to waves (NOAA-EMC#3190)
  Move WCOSS2 LD_LIBRARY_PATH patches to load_ufsda_modules.sh (NOAA-EMC#3236)
  Adding a gefs_arch task to GEFS workflow (NOAA-EMC#3211)
  Add additional GEFS variables needed for AI/ML applications  (NOAA-EMC#3221)
  Add bmat task dependency to marine LETKF task (NOAA-EMC#3224)
  Resolve bug with LMOD_TMOD_FIND_FIRST setting affecting build on WCOSS2 (NOAA-EMC#3229)
  Reinstate product groups (NOAA-EMC#3208)
  Additional fixes for downstream jobs (NOAA-EMC#3187)
  Turn IAU off during staging job for cold start experiments (NOAA-EMC#3215)
  Update the gdas.cd hash and enable GDASApp to run on WCOSS2 (NOAA-EMC#3220)
  Update upload-artifact to v4 (NOAA-EMC#3216)
  Prevent duplicate case generation in generate_workflows.sh (NOAA-EMC#3217)
  Update g-w to cycle with C1152 ATM (NOAA-EMC#3206)
  Separate use of initial increment/perturbation file from REPLAY/+03 ICs  (NOAA-EMC#3119)
  Update gsi_enkf hash and gsi_ver (NOAA-EMC#3207)
  Remove cpus-per-task from APRUN_OCNANALECEN on WCOSS2 (NOAA-EMC#3212)
  Remove 5WAVH from AWIPS GRIB2 parm files (NOAA-EMC#3146)
  Remove multi-grid wave support (NOAA-EMC#3188)
  Add echgres as a dependency for earc (NOAA-EMC#3202)
  Ensure OCNRES and ICERES have 3 digits in the archive script (NOAA-EMC#3199)
  Set runtime shell requirements within Jenkins Pipeline (NOAA-EMC#3171)
  Add efcs and epos to ufs_hybatm xml (NOAA-EMC#3192) (NOAA-EMC#3193)
  Fix GEFS and SFS compile flags in build_all.sh (NOAA-EMC#3197)
  Remove early-cycle EnKF forecast (NOAA-EMC#3185)
  Fix mod_icec bug in atmos_prod (NOAA-EMC#3167)
  Create compute build option (NOAA-EMC#3186)
  Support global-workflow using Rocky 8 on CSPs (NOAA-EMC#2998)
  Change orog gravity wave drag scheme for grid sizes less than 10km (NOAA-EMC#3175)
  Switch snow DA to use 2DVar for deterministic and ensemble mean (NOAA-EMC#3163)
  Update compression options for GEFS history files (NOAA-EMC#3184)
  Update compression options for high res history files (NOAA-EMC#3178)
  Turn DO_TEST_MODE off (NOAA-EMC#3177)
  Hotfix for gdas_arch div/0 (NOAA-EMC#3169)
  Allow building of the ufs-weather-model, WW3 pre/post execs for GFS, GEFS, SFS in the same clone of global-workflow (NOAA-EMC#3098)
  Switch Aerosol DA to use JCB and Jedi class (NOAA-EMC#3125)
  Update ufs-weather-model to 2024-12-06 commit  (NOAA-EMC#3145)
  Enable traditional threading as an option (NOAA-EMC#3149)
  Update HPC_ACCOUNT on Hercules to fv3-cpu (NOAA-EMC#3164)
  Turn C96C48_ufs_hybatmDA and C48mx500_3DVarAOWCDA into a regression test (NOAA-EMC#3120)
  Update GSI analysis jobs to use COMIN/COMOUT (NOAA-EMC#3092)
  Update HPC Tier Definitions (NOAA-EMC#3138)
  Add marine hybrid envar (NOAA-EMC#3041)
  Archive the experiment directory along with git status/diff output (NOAA-EMC#3105)
  Use stochastic restart patterns on rerun (NOAA-EMC#3077)
  Point Jenkinsfile back to CI/ (NOAA-EMC#3139)
  Fix wave restart for cold start and add ic version file (NOAA-EMC#3112)
  Allow users to override the default account at setup time (NOAA-EMC#3127)
  Refactor gridded wave post (NOAA-EMC#3014)
  Update docs related to NOAA CSPs (NOAA-EMC#3043)
  Allow APP to differ between RUNs (NOAA-EMC#2943)
  Run one executable for soca2cice (instead of two) (NOAA-EMC#3118)
  Speed up GSI analysis jobs in CI testing (NOAA-EMC#3115)
  Make aerosol output frequency variable (NOAA-EMC#2982)
  Add new stations to GFS BUFR sounding products (NOAA-EMC#3107)
  JCB-based obs+bias staging, Jedi class updates, and marine B-matrix refactoring (NOAA-EMC#2992)
  Enable tapering of atm ens perts at the model top (NOAA-EMC#3097)
  Update JGDAS ENKF POST  job  (NOAA-EMC#3090)
  SFS Runs at C96mx100  (NOAA-EMC#2960)
  Move machine-based options from config.base to host files (NOAA-EMC#3053)
  Remove RUNDIRS before running CI cases to cover re-run events (NOAA-EMC#3076)
  CI GitHub pipeline (hotfix) update for fetching repo name (NOAA-EMC#3084)
  Update JGDAS ENKF ECEN job  (NOAA-EMC#3050)
  Update snow obs processing job (NOAA-EMC#3055)
  Update to action workflow pipeline in default repo for development  (NOAA-EMC#3062)
  Update to action workflow pipeline in default repo for development (NOAA-EMC#3061)
  Update workflow pipeline (NOAA-EMC#3060)
  PW CI pipeline update5 ready for review so it can be merged and tested (NOAA-EMC#3059)
  Revert "GitHub CI Pipeline update for debugging forked PR support" (NOAA-EMC#3057)
  GitHub CI Pipeline update for debugging forked PR support (NOAA-EMC#3056)
  Add more ocean variables for post-processing in GEFS (NOAA-EMC#2995)
  Auto provisioning of PW clusters from GitHub CI added (NOAA-EMC#3051)
  Fix the name of the TC tracker filenames in archive.py (NOAA-EMC#3030)
  Make wxflow links static instead of from link_workflow (NOAA-EMC#3008)
  Update global jdas enkf diag job with COMIN/COMOUT for COM prefix (NOAA-EMC#2959)
  Add run and finalize methods to marine LETKF task (NOAA-EMC#2944)
  Fix wave restarts and GEFS FHOUT/FHMAX (NOAA-EMC#3009)
  Disabling hyper-threading (NOAA-EMC#2965)
  GitHub Actions Pipeline Updates for Self-Hosted Runners on PW (NOAA-EMC#3018)
  CI jekninsfile update hotfix (NOAA-EMC#3038)
  Update gdas.cd (NOAA-EMC#2978)
  Add ability to add tag to pslots with generate_workflows (NOAA-EMC#3036)
  CI update to shell environment with HOMEgfs to HOME_GFS for systems that need the path (NOAA-EMC#3013)
  Quick updated to Jenkins (health check) launch script (NOAA-EMC#3033)
  Document the generate_workflows.sh script (NOAA-EMC#3028)
  Replace gfs_cyc with an interval (NOAA-EMC#2928)
  Hotfix: Fix generate_workflows.sh optional build flags (NOAA-EMC#3024)
  Add a tool to run multiple YAML cases locally (NOAA-EMC#3004)
  Hotfix: Correctly set overwrite option when specified (NOAA-EMC#3021)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WalterKolczynski-NOAA WalterKolczynski-NOAA WalterKolczynski-NOAA approved these changes

@KateFriedman-NOAA KateFriedman-NOAA KateFriedman-NOAA left review comments

@aerorahul aerorahul aerorahul left review comments

@GwenChen-NOAA GwenChen-NOAA Awaiting requested review from GwenChen-NOAA GwenChen-NOAA is a code owner

@BoCui-NOAA BoCui-NOAA Awaiting requested review from BoCui-NOAA BoCui-NOAA is a code owner

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA Awaiting requested review from TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA is a code owner

@JessicaMeixner-NOAA JessicaMeixner-NOAA Awaiting requested review from JessicaMeixner-NOAA JessicaMeixner-NOAA is a code owner

@sbanihash sbanihash Awaiting requested review from sbanihash sbanihash is a code owner

Assignees
No one assigned
Labels
CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Post processing errors resulting from GFS HR4 test run
10 participants
@DavidHuber-NOAA @aerorahul @ChristopherHill-NOAA @JessicaMeixner-NOAA @RuiyuSun @AminIlia-NOAA @WenMeng-NOAA @KateFriedman-NOAA @emcbot @WalterKolczynski-NOAA