I'm not exactly sure whats wrong with the test suite, but I think if I add the --generate flag they should be okay. Do we want the test suite to even test for examples because that would add another 30 minutes or more to them. Also I need to remove the -r flag from the example suites but it looks like it worked fine.

@okBrian we can add this as a separate GitHub workflow (Examples-test.yml, Example Smoke Test).

the -r --generate things I don't understand at the moment

@okBrian It's important that these cases run and produce output that isn't nonsense. So you can run the "example suite" with ./mfc.sh test —a (+ your flags, I guess?), where -a already exists. ' This runs the files through post_process and then checks for NaNs/Infinity.

If parallel_IO is not true then I'm not sure if -a works or not. You may need to turn it on via your toolchain tricks.

I'm not exactly sure whats wrong with the test suite, but I think if I add the --generate flag they should be okay. Do we want the test suite to even test for examples because that would add another 30 minutes or more to them. Also I need to remove the -r flag from the example suites but it looks like it worked fine.

If you specify --generate it will compare the pre-process and simulation results against itself (not with the reference values from this PR). So you shouldn't pass --generate.

Whether we should or should not use the -r (--relentless) flag is debatable but we have currently opted not to use it so we should probably keep it that way. The main advantage is that we don't spend CI resources running more tests if one already failed and the workflow is marked as failed right away when the first test case fails.

I am expecting the code to not work with --no-mpi so thats another thing to debug... I think everything with mpi should pass hopefully.

@okBrian you should be able to login to Frontier to see why your CI is failing. It is failing at build (link) time. It might be due to a change you made in the src/, or it could be due to the flags you added to CMake. It might be easier to debug Frontier by changing the flags and rerunning CI or just doing it on your own after login into Frontier and building there.

@okBrian, I also noticed that it failed in some cases with GNU and Intel compilers. This suggests that the flags might not be doing what they should (or you didn't add enough). One way to be sure you "have enough" is just using -O0 for all --strict builds and ensuring that passes. If that doesn't work, then there's something else awry.

With the most recent commits these are the problematic examples
MacOS, MPI, Debug, False
2D -> Example Viscous Error: 2.51E-02
1D -> Example -> sodHypo Error: 1.01E-04

Ubuntu MPI. no-Debug, True
1D -> Example -> hypo_2materials Error: 1.74E-04
2D -> Example -> ibm_cfl_dt Error: 1.51E-04
1D -> Example -> sodHypo: Error: 1.01E-04
2D -> Example -> viscous Error: 2.51E-02
2D -> 1 Fluid(s) -> IBM Error: 1.33E-10
2D -> 1 Fluid(s) -> Viscous -> IBM Error: 2.21E-10
2D -> 1 Fluid(s) -> Viscous -> IBM -> model_eqns=3 Error: 2.21E-10
2D -> 2 Fluid(s) -> IBM Error: 5.95E-10

Ubuntu Mpi, Debug, False
2D -> Example -> ibm_cfl_dt Error: 1.65E-04

GT CPU Pheonix
2D -> Example -> laplace_pressure_jump -> NaN's detected in Case

@okBrian can you visualize the output of a few of these cases (compared to the cases that are 'working')? I suspect that, given how diverse they are, some of them are just numerically unstable and eventually produce very large numbers that different compilers handle differently.

Remaking PR