Multi-GPU segmentation fault (gslib)

[abettini99]

Hi,

I am attempting to execute a small test case using two GPUs on a single node using nekRS 23.0. I am facing similar issues as #568 and I am unsure if the FAQ is addressing this problem. My test case segfaults right after meshParallelGatherScatterSetup, but works completely fine when running on a single GPU, single node. My test case executes up to:

done :: set initial conditions

calling nek_userchk ...

generating mesh ...
loading mesh from nek ... Nelements: 57600, NboundaryIDs: 1, NboundaryFaces: 6400 done (0.00321798s)
polynomial order N: 7, over-integration order cubN: 10
meshParallelGatherScatterSetup N=7
timing gs: 2.57e-04s 3.31e-04s 3.62e-04s 3.65e-04s 3.67e-04s

Which then directly segfaults:

[gpu019:698717:0:698717] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x18)
[gpu019:698718:0:698718] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x18)
==== backtrace (tid: 698717) ====
 0 0x0000000000012d20 __funlockfile()  :0
 1 0x0000000000030e3b uct_ib_mlx5_devx_reg_exported_key()  ???:0
 2 0x0000000000044ffd ucp_rkey_pack_memh()  ???:0
 3 0x0000000000094444 ucp_tag_rndv_proto_rts_pack()  ???:0
 4 0x000000000001c022 uct_mm_ep_am_bcopy()  ???:0
 5 0x00000000000947ad ucp_tag_rndv_rts_progress()  ???:0
 6 0x000000000009933d ucp_tag_send_nbx()  ???:0
 7 0x0000000000233378 mca_pml_ucx_isend()  ???:0
 8 0x00000000000b7795 MPI_Isend()  ???:0
 9 0x000000000032c3b0 pairwiseExchange()  /home/andreabettini/nekRS/nekRS/3rd_party/gslib/ogs/src/oogs.cpp:188
10 0x000000000032fbae oogs::finish()  /home/andreabettini/nekRS/nekRS/3rd_party/gslib/ogs/src/oogs.cpp:804
11 0x0000000000331853 oogs::setup()  /home/andreabettini/nekRS/nekRS/3rd_party/gslib/ogs/src/oogs.cpp:495
12 0x00000000002a89d9 createMesh()  /home/andreabettini/nekRS/nekRS/src/mesh/meshSetup.cpp:245
13 0x00000000000dcee0 nrsSetup()  /home/andreabettini/nekRS/nekRS/src/setup/setup.cpp:236
14 0x00000000000acce8 nekrs::setup()  /home/andreabettini/nekRS/nekRS/src/lib/nekrs.cpp:193
15 0x000000000040e118 main()  /home/andreabettini/nekRS/nekRS/src/main.cpp:469
16 0x000000000003a7e5 __libc_start_main()  ???:0
17 0x000000000040f8fe _start()  ???:0
=================================
==== backtrace (tid: 698718) ====
 0 0x0000000000012d20 __funlockfile()  :0
 1 0x0000000000030e3b uct_ib_mlx5_devx_reg_exported_key()  ???:0
 2 0x0000000000044ffd ucp_rkey_pack_memh()  ???:0
 3 0x0000000000094444 ucp_tag_rndv_proto_rts_pack()  ???:0
 4 0x000000000001c022 uct_mm_ep_am_bcopy()  ???:0
 5 0x00000000000947ad ucp_tag_rndv_rts_progress()  ???:0
 6 0x000000000009933d ucp_tag_send_nbx()  ???:0
 7 0x0000000000233378 mca_pml_ucx_isend()  ???:0
 8 0x00000000000b7795 MPI_Isend()  ???:0
 9 0x000000000032c3b0 pairwiseExchange()  /home/andreabettini/nekRS/nekRS/3rd_party/gslib/ogs/src/oogs.cpp:188
10 0x000000000032fbae oogs::finish()  /home/andreabettini/nekRS/nekRS/3rd_party/gslib/ogs/src/oogs.cpp:804
11 0x0000000000331853 oogs::setup()  /home/andreabettini/nekRS/nekRS/3rd_party/gslib/ogs/src/oogs.cpp:495
12 0x00000000002a89d9 createMesh()  /home/andreabettini/nekRS/nekRS/src/mesh/meshSetup.cpp:245
13 0x00000000000dcee0 nrsSetup()  /home/andreabettini/nekRS/nekRS/src/setup/setup.cpp:236
14 0x00000000000acce8 nekrs::setup()  /home/andreabettini/nekRS/nekRS/src/lib/nekrs.cpp:193
15 0x000000000040e118 main()  /home/andreabettini/nekRS/nekRS/src/main.cpp:469
16 0x000000000003a7e5 __libc_start_main()  ???:0
17 0x000000000040f8fe _start()  ???:0
=================================
srun: error: gpu019: tasks 0-1: Segmentation fault (core dumped)
srun: launch/slurm: _step_signal: Terminating StepId=4140199.1

I presume that this is also related to gslib testing GPU-aware MPI based on the backtrace and #568. I would appreciate some insight/confirmation. If it is about GPU-aware MPI, then I can quicken the hotfix with the HPC peeps :)

Thanks a lot for the support!

• Andrea

[stgeke]

Did you try to run without GPU MPI support (set env-var NEKRS_GPU_MPI=0)?

[abettini99]

Sorry for the late reply, my job was on the queue until today for a testcase using NEKRS_GPU_MPI=0.

With NEKRS_GPU_MPI=0, the testcase runs fine.