Add missing additions from Frontier Merge (#389)

Co-authored-by: Spencer Bryngelson <[email protected]>

.github/pull_request_template.md

@@ -38,17 +38,21 @@ Please also list any relevant details for your test configuration
 - [ ] I added Doxygen docstrings to the new code
 - [ ] I have made corresponding changes to the documentation (`docs/`)
 - [ ] I have added regression tests to the test suite so that people can verify in the future that the feature is behaving as expected
-- [ ] I have added example cases in `examples/` that demonstrate my new feature performing as expected
+- [ ] I have added example cases in `examples/` that demonstrate my new feature performing as expected.
+They run to completion and demonstrate "interesting physics"
 - [ ] I ran `./mfc.sh format` before committing my code
-- [ ] New and existing tests pass locally with my changes, including with GPU capability enabled and disabled
+- [ ] New and existing tests pass locally with my changes, including with GPU capability enabled (both NVIDIA hardware with NVHPC compilers and AMD hardware with CRAY compilers) and disabled
 - [ ] This PR does not introduce any repeated code (it follows the [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) principle)
 - [ ] I cannot think of a way to condense this code and reduce any introduced additional line count
 
-### If your code changes any code source files (anything in `src/`)
+### If your code changes any code source files (anything in `src/simulation`)
 
 To make sure the code is performing as expected on GPU devices, I have:
 - [ ] Checked that the code compiles using NVHPC compilers
+- [ ] Checked that the code compiles using CRAY compilers
 - [ ] Ran the code on either V100, A100, or H100 GPUs and ensured the new feature performed as expected (the GPU results match the CPU results)
+- [ ] Ran the code on MI200+ GPUs and ensure the new features performed as expected (the GPU results match the CPU results)
 - [ ] Enclosed the new feature via `nvtx` ranges so that they can be identified in profiles
 - [ ] Ran a Nsight Systems profile using `./mfc.sh run XXXX --gpu -t simulation --nsys`, and have attached the output file (`.nsys-rep`) and plain text results to this PR
+- [ ] Ran an Omniperf profile using `./mfc.sh run XXXX --gpu -t simulation --omniperf`, and have attached the output file and plain text results to this PR.
 - [ ] Ran my code using various numbers of different GPUs (1, 2, and 8, for example) in parallel and made sure that the results scale similarly to what happens if you run without the new code/feature

docs/documentation/running.md

@@ -81,20 +81,35 @@ As an example, one might request GPUs on a SLURM system using the following:
 **Disclaimer**: IBM's JSRUN on LSF-managed computers does not use the traditional node-based approach to
 allocate resources. Therefore, the MFC constructs equivalent resource sets in the task and GPU count.
 
-### Profiling with NVIDIA Nsight
+### GPU Profiling 
+
+#### NVIDIA GPUs
 
 MFC provides two different arguments to facilitate profiling with NVIDIA Nsight.
 **Please ensure the used argument is placed at the end so their respective flags can be appended.**
-- Nsight Systems (Nsys): `./mfc.sh run ... --nsys [nsys flags]` allows one to visualize MFC's system-wide performance with [NVIDIA Nsight Systems](https://developer.nvidia.com/nsight-systems).
+- Nsight Systems (Nsys): `./mfc.sh run ... -t simulation --nsys [nsys flags]` allows one to visualize MFC's system-wide performance with [NVIDIA Nsight Systems](https://developer.nvidia.com/nsight-systems).
 NSys is best for understanding the order and execution times of major subroutines (WENO, Riemann, etc.) in MFC.
 When used, `--nsys` will run the simulation and generate `.nsys-rep` files in the case directory for all targets.
 These files can then be imported into Nsight System's GUI, which can be downloaded [here](https://developer.nvidia.com/nsight-systems/get-started#latest-Platforms). It is best to run case files with a few timesteps to keep the report files small. Learn more about NVIDIA Nsight Systems [here](https://docs.nvidia.com/nsight-systems/UserGuide/index.html).
-- Nsight Compute (NCU): `./mfc.sh run ... --ncu [ncu flags]` allows one to conduct kernel-level profiling with [NVIDIA Nsight Compute](https://developer.nvidia.com/nsight-compute).
+- Nsight Compute (NCU): `./mfc.sh run ... -t simulation --ncu [ncu flags]` allows one to conduct kernel-level profiling with [NVIDIA Nsight Compute](https://developer.nvidia.com/nsight-compute).
 NCU provides profiling information for every subroutine called and is more detailed than NSys.
 When used, `--ncu` will output profiling information for all subroutines, including elapsed clock cycles, memory used, and more after the simulation is run.
 Adding this argument will significantly slow the simulation and should only be used on case files with a few timesteps.
 Learn more about NVIDIA Nsight Compute [here](https://docs.nvidia.com/nsight-compute/NsightCompute/index.html).
 
+
+#### AMD GPUs
+- Rocprof (ROC): `./mfc.sh run ... -t simulation --roc --hip-trace [rocprof flags]` allows one to visualize MFC's system-wide performance with [Perfetto UI](https://ui.perfetto.dev/).
+When used, `--roc` will run the simulation and generate files in the case directory for all targets.
+`results.json` can then be imported in [Perfetto's UI](https://ui.perfetto.dev/).
+Learn more about AMD Rocprof [here](https://rocm.docs.amd.com/projects/rocprofiler/en/docs-5.5.1/rocprof.html)
+It is best to run case files with a few timesteps to keep the report files small.
+- Omniperf (OMNI): `./mfc.sh run ... -t simulation --omni [omniperf flags]`allows one to conduct kernel-level profiling with [AMD Omniperf](https://rocm.github.io/omniperf/introduction.html#what-is-omniperf).
+When used, `--omni` will output profiling information for all subroutines, including rooflines, cache usage, register usage, and more after the simulation is run.
+Adding this argument will moderately slow down the simulation and run the MFC executable several times.
+For this reason it should only be used with case files that have a few timesteps.
+
+
 ### Restarting Cases
 
 When running a simulation, MFC generates a `./restart_data` folder in the case directory that contains `lustre_*.dat` files that can be used to restart a simulation from saved timesteps.

toolchain/mfc/run/run.py

@@ -45,6 +45,18 @@ def __profiler_prepend() -> typing.List[str]:
 
         return ["nsys", "profile", "--stats=true", "--trace=mpi,nvtx,openacc"] + ARG("nsys")
 
+    if ARG("omni") is not None:
+        if not does_command_exist("omniperf"):
+            raise MFCException("Failed to locate [bold red]ROCM Omniperf[/bold red] (omniperf).")
+
+        return ["omniperf", "profile"] + ARG("omni") + ["--"]
+
+    if ARG("roc") is not None:
+        if not does_command_exist("rocprof"):
+            raise MFCException("Failed to locate [bold red]ROCM rocprof[/bold red] (rocprof).")
+
+        return ["rocprof"] + ARG("roc")
+
     return []