Running Apptainer images on HPC

[orliac]

Hello.
Is there any reason why the Docker container does not contain the Python environment and bindings?

[hajdik]

The Docker images do have all necessary Python packages installed. Make sure you're creating a container as specified here.

[orliac]

Thanks. Yes I started from that link but I had to convert to a SIF using Apptainer as we do not have Docker on our cluster. I'll try with another version to see if the problem remains.

[orliac]

We compared the Docker and Apptainer versions. Some paths are not exported hence the issues I was facing.

I have another question: is the Openmpi in the container built with Slurm support? If not, would that be possible to add Slurm support in the future builds?

[marcomangano]

Hi, can you clarify what you mean by Slurm support? Slurm is a scheduler, do you refer to general issues with MPI on a cluster?

We do not have experience with Apptainer so we unfortunately can't help addressing issues with that. We used Singularity in the past to run on certain HPC systems. That required building a dedicated Docker image starting from a base image provided by the HPC maintainers (needed to correctly link to openmpi and other system libraries) and then convert it to Singularity. That is likely the reason why converting the existing Docker images does not work on the cluster after the conversion to Apptainer.

[orliac]

Hi,
See https://slurm.schedmd.com/mpi_guide.html#open_mpi.
Actually I was able to submit a job add the --mpi=pmix option with something like:

srun --mpi=pmix apptainer exec $SIF \
     /home/mdolabuser/.pyenv/shims/python $PY_SCRIPT \
     -gridFile $CGNS_FILE \
     -output $OUTPUT \
     -evalOnly

Now there are other issues to solve, due to some missing path definitions I guess:

ERROR 7: VSPAERO Viewer Not Found. 
        Expected here: /home/mdolabuser/packages/OpenVSP/python/openvsp/openvsp/vspviewer
Traceback (most recent call last):
  File "/home/mdolabuser/.pyenv/versions/3.9.16/lib/python3.9/site-packages/adflow/pyADflow.py", line 91, in __init__
    self.adflow
AttributeError: 'ADFLOW' object has no attribute 'adflow'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/work/scitas-ge/orliac/mach-aero/surrogate_adflow_boxfish_standalone.py", line 66, in <module>
    CFDSolver = ADFLOW(options=aeroOptions, comm=comm)
  File "/home/mdolabuser/.pyenv/versions/3.9.16/lib/python3.9/site-packages/adflow/pyADflow.py", line 94, in __init__
    self.adflow = MExt.MExt("libadflow", curDir, debug=debug)._module
  File "/home/mdolabuser/.pyenv/versions/3.9.16/lib/python3.9/site-packages/adflow/MExt.py", line 49, in __init__
    self._pkg = __import__(self._pkgname, globals(), locals(), [self.name])
ImportError: libcgns.so.4.2: cannot open shared object file: No such file or directory
Exception ignored in: <function MExt.__del__ at 0x7f3912f36b80>
Traceback (most recent call last):
  File "/home/mdolabuser/.pyenv/versions/3.9.16/lib/python3.9/site-packages/adflow/MExt.py", line 60, in __del__
    del sys.modules[self._module.__name__]
AttributeError: 'MExt' object has no attribute '_module'
srun: error: jst043: task 0: Exited with exit code 1
srun: Terminating StepId=27301678.0

[orliac]

In the end I could run the software submitting the following script to Slurm via sbatch:

#!/bin/bash -l

#SBATCH --nodes 2
#SBATCH --ntasks 144
#SBATCH --cpus-per-task 1
#SBATCH --mem 30G
#SBATCH --output test.log
#SBATCH --qos parallel

module load gcc
module load openmpi

SIF=/work/scitas-ge/orliac/mach-aero/u22-gcc-ompi-stable.sif

DIR_PROJ=/work/scitas-ge/orliac/mach-aero
PY_SCRIPT=$DIR_PROJ/surrogate_adflow_boxfish_standalone.py
CGNS_FILE=$DIR_PROJ/mesh_initial.cgns
OUTPUT=$DIR_PROJ/test_output
[ ! -d $OUTPUT ] && mkdir -pv $OUTPUT
#export SLURM_MPI_TYPE=pmi2

srun --mpi=pmix apptainer exec \
     --env "LD_LIBRARY_PATH=/home/mdolabuser/packages/CGNS-4.2.0/opt-gcc/lib" \
     $SIF \
     /home/mdolabuser/.pyenv/shims/python $PY_SCRIPT \
     -gridFile $CGNS_FILE \
     -output $OUTPUT \
     -evalOnly

[marcomangano]

I see. I don't have experience with containers on clusters so I don't know why the CGNS path is not picked up. If that workaround does not create other library issues then this solution is a good reference for now.

It looks like you can now run the container, does it actually run on multiple procs? I don't see any salloc command in the script above. Anyway, I am updating the discussion title to reflect the main topic here.

[orliac]

Yes, the lines :

#SBATCH --nodes 2
#SBATCH --ntasks 144
#SBATCH --cpus-per-task 1

actually say that 144 MPI tasks (1 CPU core each) were launched on 2 compute nodes (72 physical CPU cores each) on our cluster.
I know nothing about the code but from initial testing it was running much faster than with a single task or a reduced number of tasks.
Maybe the user I helped will contribute with some scaling plots if you think this is usefull, I'll ping him with that respect.