Skip to content

Commit

Permalink
Batch files per computer (MFlowCode#240 & MFlowCode#287)
Browse files Browse the repository at this point in the history
  • Loading branch information
@henryleberre
henryleberre committed Jan 14, 2024
1 parent f28faa7 commit 66663f2
Show file tree
Hide file tree
Showing 19 changed files with 457 additions and 863 deletions.
73 changes: 32 additions & 41 deletions docs/documentation/running.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,31 @@

MFC can be run using `mfc.sh`'s `run` command.
It supports both interactive and batch execution, the latter being designed for multi-socket systems, namely supercomputers, equipped with a scheduler such as PBS, SLURM, and LSF.
A full (and updated) list of available arguments can be acquired with `./mfc.sh run -h`.
A full (and updated) list of available arguments can be acquired with `./mfc.sh run -h`.

MFC supports running simulations locally (Linux, MacOS, and Windows) as well as
several supercomputer clusters, both interactively and through batch submission.

> [!IMPORTANT]
> Running simulations locally should work out of the box. On supported clusters,
> you can append `-c <computer name>` on the command line to instruct the MFC toolchain
> to make use of the template file `toolchain/templates/<computer name>.mako`. You can
> browse that directory and contribute your own files. Since systems and their schedulers
> do not have a standardized syntax to request certain resources, MFC can only provide
> support for a restricted subset of common or user-contributed configuration options.
>
> Adding a new template file or modifying an existing one will most likely be required if:
> - You are on a cluster that does not have a template yet.
> - Your cluster is configured with SLURM and but fails when interactive jobs are
> launched with `mpirun`.
> - Something in the existing default or computer template file is incompatible with
> your system or does not provide a feature you need.
>
> If `-c <computer name>` is left unspecified, it defaults to `-c default`.
Additional flags can be appended to the MPI executable call using the `-f` (i.e `--flags`) option.

Please refer to `./mfc.sh run -h` for a complete list of arguments and options, along with their defaults.

## Interactive Execution

Expand Down Expand Up @@ -32,24 +56,16 @@ using 4 cores:
$ ./mfc.sh run examples/2D_shockbubble/case.py -t simulation post_process -n 4
```

On some computer clusters, MFC might select the wrong MPI program to execute your application
because it uses a general heuristic for its selection. Notably, `srun` is known to fail on some SLURM
systems when using GPUs or MPI implementations from different vendors, whereas `mpirun` functions properly. To override and manually specify which
MPI program you wish to run your application with, please use the `-b <program name>` option (i.e `--binary`).

Additional flags can be appended to the MPI executable call using the `-f` (i.e `--flags`) option.

Please refer to `./mfc.sh run -h` for a complete list of arguments and options, along with their defaults.

## Batch Execution

The MFC detects which scheduler your system is using and handles the creation and execution of batch scripts.
The batch engine is requested with the `-e batch` option.
Whereas the interactive engine can execute all of MFC's codes in succession, the batch engine requires you to only specify one target with the `-t` option.
The number of nodes and GPUs can, respectively be specified with the `-N` (i.e `--nodes`) and `-g` (i.e `--gpus-per-node`) options.
The batch engine is requested via the `-e batch` option.
The number of nodes can be specified with the `-N` (i.e `--nodes`) option.

We provide a list of (baked-in) submission batch scripts in the `toolchain/templates` folder.

```console
$ ./mfc.sh run examples/2D_shockbubble/case.py -e batch -N 2 -n 4 -g 4 -t simulation
$ ./mfc.sh run examples/2D_shockbubble/case.py -e batch -N 2 -n 4 -t simulation -c <computer name>
```

Other useful arguments include:
Expand All @@ -60,26 +76,8 @@ Other useful arguments include:
- `-a <account name>` to identify the account to be charged for the job. (i.e `--account`)
- `-p <partition name>` to select the job's partition. (i.e `--partition`)

Since some schedulers don't have a standardized syntax to request certain resources, MFC can only provide support for a restricted subset of common configuration options.
If MFC fails to execute on your system, or if you wish to adjust how the program runs and resources are requested to be allocated, you are invited to modify the template batch script for your queue system.
Upon execution of `./mfc.sh run`, MFC fills in the template with runtime parameters, to generate the batch file it will submit.
These files are located in the [templates](https://github.com/MFlowCode/MFC/tree/master/toolchain/templates/) directory.
To request GPUs, modification of the template will be required on most systems.

- Lines that begin with `#>` are ignored and won't figure in the final batch script, not even as a comment.

- Statements of the form `${expression}` are string-replaced to provide runtime parameters, most notably execution options.
You can perform therein any Python operation recognized by the built-in `expr()` function.

As an example, one might request GPUs on a SLURM system using the following:

```
#SBATCH --gpus=v100-32:{gpus_per_node*nodes}
```

- Statements of the form `{MFC::expression}` tell MFC where to place the common code, across all batch files, that is required for proper execution.
They are not intended to be modified by users.

**Disclaimer**: IBM's JSRUN on LSF-managed computers does not use the traditional node-based approach to
allocate resources. Therefore, the MFC constructs equivalent resource-sets in task and GPU count.

Expand Down Expand Up @@ -173,13 +171,6 @@ $ ./mfc.sh run examples/1D_vacuum_restart/restart_case.py -t post_process
- Oak Ridge National Laboratory's [Summit](https://www.olcf.ornl.gov/summit/):

```console
$ ./mfc.sh run examples/2D_shockbubble/case.py -e batch \
-N 2 -n 4 -g 4 -t simulation -a <redacted>
```

- University of California, San Diego's [Expanse](https://www.sdsc.edu/services/hpc/expanse/):

```console
$ ./mfc.sh run examples/2D_shockbubble/case.py -e batch -p GPU -t simulation \
-N 2 -n 8 -g 8 -f="--gpus=v100-32:16" -b mpirun –w 00:30:00
$ ./mfc.sh run examples/2D_shockbubble/case.py -e batch \
-N 2 -n 4 -t simulation -a <redacted> -c summit
```
59 changes: 28 additions & 31 deletions toolchain/mfc/args.py
View file
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
import re, os.path, argparse, dataclasses

from .build import TARGETS, DEFAULT_TARGETS, DEPENDENCY_TARGETS
from .common import MFCException, format_list_to_string
from .test.cases import generate_cases
from .run.engines import ENGINES
from .run.mpi_bins import BINARIES
from .run.run import get_baked_templates
from .build import TARGETS, DEFAULT_TARGETS, DEPENDENCY_TARGETS
from .common import MFCException, format_list_to_string
from .test.cases import generate_cases

# pylint: disable=too-many-locals, too-many-statements
# pylint: disable=too-many-locals, too-many-branches, too-many-statements
def parse(config):
parser = argparse.ArgumentParser(
prog="./mfc.sh",
Expand Down Expand Up @@ -75,8 +74,6 @@ def add_common_arguments(p, mask = None):
# === CLEAN ===
add_common_arguments(clean, "jg")

binaries = [ b.bin for b in BINARIES ]

# === TEST ===
test_cases = generate_cases()

Expand All @@ -100,29 +97,28 @@ def add_common_arguments(p, mask = None):
test.add_argument(metavar="FORWARDED", default=[], dest="--", nargs="*", help="Arguments to forward to the ./mfc.sh run invocations.")

# === RUN ===
engines = [ e.slug for e in ENGINES ]

add_common_arguments(run)
run.add_argument("input", metavar="INPUT", type=str, help="Input file to run.")
run.add_argument("arguments", metavar="ARGUMENTS", nargs="*", type=str, default=[], help="Additional arguments to pass to the case file.")
run.add_argument("-e", "--engine", choices=engines, type=str, default=engines[0], help="Job execution/submission engine choice.")
run.add_argument("input", metavar="INPUT", type=str, help="Input file to run.")
run.add_argument("arguments", metavar="ARGUMENTS", nargs="*", type=str, default=[], help="Additional positional arguments to pass to the case file.")
run.add_argument("-e", "--engine", choices=["interactive", "batch"], type=str, default="interactive", help="Job execution/submission engine choice.")
run.add_argument("--output-summary", type=str, default=None, help="(Interactive) Output a YAML summary file.")
run.add_argument("-p", "--partition", metavar="PARTITION", type=str, default="", help="(Batch) Partition for job submission.")
run.add_argument("-N", "--nodes", metavar="NODES", type=int, default=1, help="(Batch) Number of nodes.")
run.add_argument("-n", "--tasks-per-node", metavar="TASKS", type=int, default=1, help="Number of tasks per node.")
run.add_argument("-w", "--walltime", metavar="WALLTIME", type=str, default="01:00:00", help="(Batch) Walltime.")
run.add_argument("-a", "--account", metavar="ACCOUNT", type=str, default="", help="(Batch) Account to charge.")
run.add_argument("-@", "--email", metavar="EMAIL", type=str, default="", help="(Batch) Email for job notification.")
run.add_argument("-#", "--name", metavar="NAME", type=str, default="MFC", help="(Batch) Job name.")
run.add_argument("-b", "--binary", choices=binaries, type=str, default=None, help="(Interactive) Override MPI execution binary")
run.add_argument("-s", "--scratch", action="store_true", default=False, help="Build from scratch.")
run.add_argument("--ncu", nargs=argparse.REMAINDER, type=str, help="Profile with NVIDIA Nsight Compute.")
run.add_argument("--nsys", nargs=argparse.REMAINDER, type=str, help="Profile with NVIDIA Nsight Systems.")
run.add_argument( "--dry-run", action="store_true", default=False, help="(Batch) Run without submitting batch file.")
run.add_argument("--case-optimization", action="store_true", default=False, help="(GPU Optimization) Compile MFC targets with some case parameters hard-coded.")
run.add_argument( "--no-build", action="store_true", default=False, help="(Testing) Do not rebuild MFC.")
run.add_argument("--wait", action="store_true", default=False, help="(Batch) Wait for the job to finish.")
run.add_argument("-f", "--flags", metavar="FLAGS", dest="--", nargs=argparse.REMAINDER, type=str, default=[], help="(Interactive) Arguments to forward to the MPI invocation.")
run.add_argument("-p", "--partition", metavar="PARTITION", type=str, default="", help="(Batch) Partition for job submission.")
run.add_argument("-q", "--quality_of_service", metavar="QOS", type=str, default="", help="(Batch) Quality of Service for job submission.")
run.add_argument("-N", "--nodes", metavar="NODES", type=int, default=1, help="(Batch) Number of nodes.")
run.add_argument("-n", "--tasks-per-node", metavar="TASKS", type=int, default=1, help="Number of tasks per node.")
run.add_argument("-w", "--walltime", metavar="WALLTIME", type=str, default="01:00:00", help="(Batch) Walltime.")
run.add_argument("-a", "--account", metavar="ACCOUNT", type=str, default="", help="(Batch) Account to charge.")
run.add_argument("-@", "--email", metavar="EMAIL", type=str, default="", help="(Batch) Email for job notification.")
run.add_argument("-#", "--name", metavar="NAME", type=str, default="MFC", help="(Batch) Job name.")
run.add_argument("-s", "--scratch", action="store_true", default=False, help="Build from scratch.")
run.add_argument("--ncu", nargs=argparse.REMAINDER, type=str, help="Profile with NVIDIA Nsight Compute.")
run.add_argument("--nsys", nargs=argparse.REMAINDER, type=str, help="Profile with NVIDIA Nsight Systems.")
run.add_argument( "--dry-run", action="store_true", default=False, help="(Batch) Run without submitting batch file.")
run.add_argument("--case-optimization", action="store_true", default=False, help="(GPU Optimization) Compile MFC targets with some case parameters hard-coded.")
run.add_argument( "--no-build", action="store_true", default=False, help="(Testing) Do not rebuild MFC.")
run.add_argument("--wait", action="store_true", default=False, help="(Batch) Wait for the job to finish.")
run.add_argument("-f", "--flags", metavar="FLAGS", dest="--", nargs=argparse.REMAINDER, type=str, default=[], help="(Interactive) Arguments to forward to the MPI invocation.")
run.add_argument("-c", "--computer", metavar="COMPUTER", type=str, default="default", help=f"(Batch) Path to a custom submission file template or one of {format_list_to_string(list(get_baked_templates().keys()))}.")

# === BENCH ===
add_common_arguments(bench, "t")
Expand Down Expand Up @@ -153,10 +149,11 @@ def add_common_arguments(p, mask = None):
# "Slugify" the name of the job
args["name"] = re.sub(r'[\W_]+', '-', args["name"])

# build's --case-optimization and --input depend on each other
# We need to check for some invalid combinations of arguments because of
# the limitations of argparse.
if args["command"] == "build":
if (args["input"] is not None) ^ args["case_optimization"] :
raise MFCException("./mfc.sh build's --case-optimization requires --input")
raise MFCException("./mfc.sh build's --case-optimization and --input must be used together.")

# Input files to absolute paths
for e in ["input", "input1", "input2"]:
Expand Down
2 changes: 2 additions & 0 deletions toolchain/mfc/build.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,8 @@ def configure(self):
delete_directory(build_dirpath)
create_directory(build_dirpath)

input.load({}).generate_fpp(self)

if system(command).returncode != 0:
raise MFCException(f"Failed to configure the [bold magenta]{self.name}[/bold magenta] target.")

Expand Down
11 changes: 2 additions & 9 deletions toolchain/mfc/common.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@
from .printer import cons


MFC_ROOTDIR = normpath(f"{dirname(realpath(__file__))}/../..")
MFC_ROOTDIR = abspath(normpath(f"{dirname(realpath(__file__))}/../.."))
MFC_TESTDIR = abspath(f"{MFC_ROOTDIR}/tests")
MFC_SUBDIR = abspath(f"{MFC_ROOTDIR}/build")
MFC_TEMPLATEDIR = abspath(f"{MFC_ROOTDIR}/toolchain/templates")
MFC_LOCK_FILEPATH = abspath(f"{MFC_SUBDIR}/lock.yaml")
MFC_BENCH_FILEPATH = abspath(f"{MFC_ROOTDIR}/toolchain/bench.yaml")

Expand Down Expand Up @@ -179,14 +180,6 @@ def does_system_use_modules() -> bool:
return does_command_exist("module")


def get_loaded_modules() -> typing.List[str]:
"""
Returns a list of loaded modules.
"""

return [ l for l in subprocess.getoutput("module -t list").splitlines() if ' ' not in l ]


def is_number(x: str) -> bool:
if x is None:
return False
Expand Down
Loading

0 comments on commit 66663f2

Please sign in to comment.