Error when running nekRS 21.1 over multiple nodes

[wlu00]

Hello all,

I've been trying to run nekRS 21.1 on the HPC and I've been running into issues when scaling up to more then a single node.

When I run the turbPipePeriodic case using 4 GPUs (on a single node) the case runs perfectly, but when I increase the number of GPUs to 8 (over two nodes) I get the following error.

terminate called after throwing an instance of 'occa::exception'
  what():  
---[ Error ]--------------------------------------------------------------------
    File     : /home/563/wl8035/nekRS-21.1/3rd_party/occa/src/occa/internal/modes/cuda/device.cpp
    Line     : 30
    Function : device
    Message  : Device: Creating Device
               CUDA Error [ 101 ]: CUDA_ERROR_INVALID_DEVICE
    Stack
      12 occa::error(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
      11 occa::cuda::error(cudaError_enum, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
      10 occa::cuda::device::device(occa::json const&)
       9 occa::cuda::cudaMode::newDevice(occa::json const&)
       8 occa::device::setup(occa::json const&)
       7 occa::device::setup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
       6 device_t::device_t(setupAide&, ompi_communicator_t*)
       5 platform_t::platform_t(setupAide&, ompi_communicator_t*)
       4 nekrs::setup(ompi_communicator_t*, int, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
       3 /home/563/wl8035/.local/nekrs/bin/nekrs()
       2 /lib64/libc.so.6(__libc_start_main+0xf3)
       1 /home/563/wl8035/.local/nekrs/bin/nekrs()
================================================================================

[gadi-gpu-v100-0032:282634] *** Process received signal ***
[gadi-gpu-v100-0032:282634] Signal: Aborted (6)
[gadi-gpu-v100-0032:282634] Signal code:  (-6)
terminate called after throwing an instance of 'occa::exception'
  what():  
---[ Error ]--------------------------------------------------------------------
    File     : /home/563/wl8035/nekRS-21.1/3rd_party/occa/src/occa/internal/modes/cuda/device.cpp
    Line     : 30
    Function : device
    Message  : Device: Creating Device
               CUDA Error [ 101 ]: CUDA_ERROR_INVALID_DEVICE
    Stack
      12 occa::error(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
      11 occa::cuda::error(cudaError_enum, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
      10 occa::cuda::device::device(occa::json const&)
       9 occa::cuda::cudaMode::newDevice(occa::json const&)
       8 occa::device::setup(occa::json const&)
       7 occa::device::setup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
       6 device_t::device_t(setupAide&, ompi_communicator_t*)
       5 platform_t::platform_t(setupAide&, ompi_communicator_t*)
       4 nekrs::setup(ompi_communicator_t*, int, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
       3 /home/563/wl8035/.local/nekrs/bin/nekrs()
       2 /lib64/libc.so.6(__libc_start_main+0xf3)
       1 /home/563/wl8035/.local/nekrs/bin/nekrs()
================================================================================

[gadi-gpu-v100-0032:282631] *** Process received signal ***
[gadi-gpu-v100-0032:282631] Signal: Aborted (6)
[gadi-gpu-v100-0032:282631] Signal code:  (-6)
[gadi-gpu-v100-0032:282631] [ 0] terminate called after throwing an instance of 'occa::exception'
  what():  
---[ Error ]--------------------------------------------------------------------
    File     : /home/563/wl8035/nekRS-21.1/3rd_party/occa/src/occa/internal/modes/cuda/device.cpp
    Line     : 30
    Function : device
    Message  : Device: Creating Device
               CUDA Error [ 101 ]: CUDA_ERROR_INVALID_DEVICE
    Stack
      12 occa::error(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
      11 occa::cuda::error(cudaError_enum, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
      10 occa::cuda::device::device(occa::json const&)
       9 occa::cuda::cudaMode::newDevice(occa::json const&)
       8 occa::device::setup(occa::json const&)
       7 occa::device::setup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
       6 device_t::device_t(setupAide&, ompi_communicator_t*)
       5 platform_t::platform_t(setupAide&, ompi_communicator_t*)
       4 nekrs::setup(ompi_communicator_t*, int, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
       3 /home/563/wl8035/.local/nekrs/bin/nekrs()
       2 /lib64/libc.so.6(__libc_start_main+0xf3)
       1 /home/563/wl8035/.local/nekrs/bin/nekrs()
================================================================================

[gadi-gpu-v100-0032:282632] *** Process received signal ***
[gadi-gpu-v100-0032:282632] Signal: Aborted (6)
[gadi-gpu-v100-0032:282632] Signal code:  (-6)
[gadi-gpu-v100-0032:282632] [ 0] /lib64/libpthread.so.0(+0x12b30)[0x14b0be8a6b30]
[gadi-gpu-v100-0032:282632] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x14b0be50638f]
terminate called after throwing an instance of 'occa::exception'
  what():  
---[ Error ]--------------------------------------------------------------------
    File     : /home/563/wl8035/nekRS-21.1/3rd_party/occa/src/occa/internal/modes/cuda/device.cpp
    Line     : 30
    Function : device
    Message  : Device: Creating Device
               CUDA Error [ 101 ]: CUDA_ERROR_INVALID_DEVICE
    Stack
      12 occa::error(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
      11 occa::cuda::error(cudaError_enum, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
      10 occa::cuda::device::device(occa::json const&)
       9 occa::cuda::cudaMode::newDevice(occa::json const&)
       8 occa::device::setup(occa::json const&)
       7 occa::device::setup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
       6 device_t::device_t(setupAide&, ompi_communicator_t*)
       5 platform_t::platform_t(setupAide&, ompi_communicator_t*)
       4 nekrs::setup(ompi_communicator_t*, int, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
       3 /home/563/wl8035/.local/nekrs/bin/nekrs()
       2 /lib64/libc.so.6(__libc_start_main+0xf3)
       1 /home/563/wl8035/.local/nekrs/bin/nekrs()
================================================================================

[gadi-gpu-v100-0032:282633] *** Process received signal ***
[gadi-gpu-v100-0032:282633] Signal: Aborted (6)
[gadi-gpu-v100-0032:282633] Signal code:  (-6)
[gadi-gpu-v100-0032:282633] [ 0] /lib64/libpthread.so.0(+0x12b30)[0x1521f8500b30]
[gadi-gpu-v100-0032:282633] [ 1] [gadi-gpu-v100-0032:282634] [ 0] /lib64/libpthread.so.0(+0x12b30)[0x15108c11bb30]
[gadi-gpu-v100-0032:282634] [ 1] /lib64/libpthread.so.0(+0x12b30)[0x153cfd788b30]
[gadi-gpu-v100-0032:282631] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x153cfd3e838f]
[gadi-gpu-v100-0032:282631] [ 2] [gadi-gpu-v100-0032:282632] [ 2] /lib64/libc.so.6(abort+0x127)[0x14b0be4f0dc5]
[gadi-gpu-v100-0032:282632] [ 3] /lib64/libc.so.6(gsignal+0x10f)[0x1521f816038f]
[gadi-gpu-v100-0032:282633] [ 2] /lib64/libc.so.6(abort+0x127)[0x1521f814adc5]
[gadi-gpu-v100-0032:282633] [ 3] /lib64/libc.so.6(gsignal+0x10f)[0x15108bd7b38f]
[gadi-gpu-v100-0032:282634] [ 2] /lib64/libc.so.6(abort+0x127)[0x15108bd65dc5]
[gadi-gpu-v100-0032:282634] [ 3] /lib64/libc.so.6(abort+0x127)[0x153cfd3d2dc5]
[gadi-gpu-v100-0032:282631] [ 3] /lib64/libstdc++.so.6(+0x9009b)[0x14b0bf31609b]
[gadi-gpu-v100-0032:282632] [ 4] /lib64/libstdc++.so.6(+0x9009b)[0x153cfe1f809b]
[gadi-gpu-v100-0032:282631] [ 4] /lib64/libstdc++.so.6(+0x9009b)[0x1521f8f7009b]
[gadi-gpu-v100-0032:282633] [ 4] /lib64/libstdc++.so.6(+0x9653c)[0x14b0bf31c53c]
[gadi-gpu-v100-0032:282632] [ 5] /lib64/libstdc++.so.6(+0x9009b)[0x15108cb8b09b]
[gadi-gpu-v100-0032:282634] [ 4] /lib64/libstdc++.so.6(+0x96597)[0x14b0bf31c597]
[gadi-gpu-v100-0032:282632] [ 6] /lib64/libstdc++.so.6(+0x9653c)[0x1521f8f7653c]
[gadi-gpu-v100-0032:282633] [ 5] /lib64/libstdc++.so.6(+0x9653c)[0x153cfe1fe53c]
[gadi-gpu-v100-0032:282631] [ 5] /lib64/libstdc++.so.6(+0x967f8)[0x14b0bf31c7f8]
[gadi-gpu-v100-0032:282632] [ 7] /lib64/libstdc++.so.6(+0x9653c)[0x15108cb9153c]
[gadi-gpu-v100-0032:282634] [ 5] /lib64/libstdc++.so.6(+0x96597)[0x1521f8f76597]
[gadi-gpu-v100-0032:282633] [ 6] /lib64/libstdc++.so.6(+0x96597)[0x153cfe1fe597]
[gadi-gpu-v100-0032:282631] [ 6] /lib64/libstdc++.so.6(+0x96597)[0x15108cb91597]
[gadi-gpu-v100-0032:282634] [ 6] /lib64/libstdc++.so.6(+0x967f8)[0x1521f8f767f8]
[gadi-gpu-v100-0032:282633] [ 7] /lib64/libstdc++.so.6(+0x967f8)[0x153cfe1fe7f8]
[gadi-gpu-v100-0032:282631] [ 7] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa8_messageERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbS7_S7_iS7_+0x138)[0x14b0c19c7b28]
[gadi-gpu-v100-0032:282632] [ 8] /lib64/libstdc++.so.6(+0x967f8)[0x15108cb917f8]
[gadi-gpu-v100-0032:282634] [ 7] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa5errorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_iS7_+0x4f)[0x14b0c19c7c2f]
[gadi-gpu-v100-0032:282632] [ 9] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa8_messageERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbS7_S7_iS7_+0x138)[0x1521fb621b28]
[gadi-gpu-v100-0032:282633] [ 8] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa8_messageERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbS7_S7_iS7_+0x138)[0x153d008a9b28]
[gadi-gpu-v100-0032:282631] [ 8] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa8_messageERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbS7_S7_iS7_+0x138)[0x15108f23cb28]
[gadi-gpu-v100-0032:282634] [ 8] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa4cuda5errorE14cudaError_enumRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_iS9_+0x301)[0x14b0c1954f31]
[gadi-gpu-v100-0032:282632] [10] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa5errorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_iS7_+0x4f)[0x1521fb621c2f]
[gadi-gpu-v100-0032:282633] [ 9] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa5errorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_iS7_+0x4f)[0x153d008a9c2f]
[gadi-gpu-v100-0032:282631] [ 9] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa5errorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_iS7_+0x4f)[0x15108f23cc2f]
[gadi-gpu-v100-0032:282634] [ 9] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa4cuda6deviceC2ERKNS_4jsonE+0x81f)[0x14b0c1946d6f]
[gadi-gpu-v100-0032:282632] [11] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa4cuda5errorE14cudaError_enumRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_iS9_+0x301)[0x1521fb5aef31]
[gadi-gpu-v100-0032:282633] [10] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa4cuda5errorE14cudaError_enumRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_iS9_+0x301)[0x153d00836f31]
[gadi-gpu-v100-0032:282631] [10] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa4cuda5errorE14cudaError_enumRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_iS9_+0x301)[0x15108f1c9f31]
[gadi-gpu-v100-0032:282634] [10] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa4cuda8cudaMode9newDeviceERKNS_4jsonE+0x34)[0x14b0c1952df4]
[gadi-gpu-v100-0032:282632] [12] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa4cuda6deviceC2ERKNS_4jsonE+0x81f)[0x1521fb5a0d6f]
[gadi-gpu-v100-0032:282633] [11] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa6device5setupERKNS_4jsonE+0x20b)[0x14b0c18516cb]
[gadi-gpu-v100-0032:282632] [13] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa4cuda6deviceC2ERKNS_4jsonE+0x81f)[0x153d00828d6f]
[gadi-gpu-v100-0032:282631] [11] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa4cuda6deviceC2ERKNS_4jsonE+0x81f)[0x15108f1bbd6f]
[gadi-gpu-v100-0032:282634] [11] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa6device5setupERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x22)[0x14b0c1851952]
[gadi-gpu-v100-0032:282632] [14] /home/563/wl8035/.local/nekrs/lib/libnekrs.so(_ZN8device_tC1ER9setupAideP19ompi_communicator_t+0x443)[0x14b0c1d76293]
[gadi-gpu-v100-0032:282632] [15] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa4cuda8cudaMode9newDeviceERKNS_4jsonE+0x34)[0x1521fb5acdf4]
[gadi-gpu-v100-0032:282633] [12] /home/563/wl8035/.local/nekrs/lib/libnekrs.so(_ZN10platform_tC1ER9setupAideP19ompi_communicator_t+0x35)[0x14b0c1d703c5]
[gadi-gpu-v100-0032:282632] [16] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa4cuda8cudaMode9newDeviceERKNS_4jsonE+0x34)[0x153d00834df4]
[gadi-gpu-v100-0032:282631] [12] /home/563/wl8035/.local/nekrs/lib/libnekrs.so(_ZN5nekrs5setupEP19ompi_communicator_tiiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_S7_S7_+0x2446)[0x14b0c1cfcbe6]
[gadi-gpu-v100-0032:282632] [17] /home/563/wl8035/.local/nekrs/bin/nekrs[0x40a0a9]
[gadi-gpu-v100-0032:282632] [18] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa4cuda8cudaMode9newDeviceERKNS_4jsonE+0x34)[0x15108f1c7df4]
[gadi-gpu-v100-0032:282634] [12] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa6device5setupERKNS_4jsonE+0x20b)[0x1521fb4ab6cb]
[gadi-gpu-v100-0032:282633] [13] /lib64/libc.so.6(__libc_start_main+0xf3)[0x14b0be4f24a3]
[gadi-gpu-v100-0032:282632] [19] /home/563/wl8035/.local/nekrs/bin/nekrs[0x40a6ce]
[gadi-gpu-v100-0032:282632] *** End of error message ***
/home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa6device5setupERKNS_4jsonE+0x20b)[0x153d007336cb]
[gadi-gpu-v100-0032:282631] [13] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa6device5setupERKNS_4jsonE+0x20b)[0x15108f0c66cb]
[gadi-gpu-v100-0032:282634] [13] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa6device5setupERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x22)[0x1521fb4ab952]
[gadi-gpu-v100-0032:282633] [14] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa6device5setupERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x22)[0x153d00733952]
[gadi-gpu-v100-0032:282631] [14] /home/563/wl8035/.local/nekrs/lib/libnekrs.so(_ZN8device_tC1ER9setupAideP19ompi_communicator_t+0x443)[0x1521fb9d0293]
[gadi-gpu-v100-0032:282633] [15] /home/563/wl8035/.local/nekrs/lib/libocca.so(_ZN4occa6device5setupERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x22)[0x15108f0c6952]
[gadi-gpu-v100-0032:282634] [14] /home/563/wl8035/.local/nekrs/lib/libnekrs.so(_ZN10platform_tC1ER9setupAideP19ompi_communicator_t+0x35)[0x1521fb9ca3c5]
[gadi-gpu-v100-0032:282633] [16] /home/563/wl8035/.local/nekrs/lib/libnekrs.so(_ZN8device_tC1ER9setupAideP19ompi_communicator_t+0x443)[0x153d00c58293]
[gadi-gpu-v100-0032:282631] [15] /home/563/wl8035/.local/nekrs/lib/libnekrs.so(_ZN5nekrs5setupEP19ompi_communicator_tiiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_S7_S7_+0x2446)[0x1521fb956be6]
[gadi-gpu-v100-0032:282633] [17] /home/563/wl8035/.local/nekrs/bin/nekrs[0x40a0a9]
[gadi-gpu-v100-0032:282633] [18] /home/563/wl8035/.local/nekrs/lib/libnekrs.so(_ZN8device_tC1ER9setupAideP19ompi_communicator_t+0x443)[0x15108f5eb293]
[gadi-gpu-v100-0032:282634] [15] /home/563/wl8035/.local/nekrs/lib/libnekrs.so(_ZN10platform_tC1ER9setupAideP19ompi_communicator_t+0x35)[0x153d00c523c5]
[gadi-gpu-v100-0032:282631] [16] /lib64/libc.so.6(__libc_start_main+0xf3)[0x1521f814c4a3]
[gadi-gpu-v100-0032:282633] [19] /home/563/wl8035/.local/nekrs/bin/nekrs[0x40a6ce]
[gadi-gpu-v100-0032:282633] *** End of error message ***
/home/563/wl8035/.local/nekrs/lib/libnekrs.so(_ZN10platform_tC1ER9setupAideP19ompi_communicator_t+0x35)[0x15108f5e53c5]
[gadi-gpu-v100-0032:282634] [16] /home/563/wl8035/.local/nekrs/lib/libnekrs.so(_ZN5nekrs5setupEP19ompi_communicator_tiiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_S7_S7_+0x2446)[0x153d00bdebe6]
[gadi-gpu-v100-0032:282631] [17] /home/563/wl8035/.local/nekrs/bin/nekrs[0x40a0a9]
[gadi-gpu-v100-0032:282631] [18] /home/563/wl8035/.local/nekrs/lib/libnekrs.so(_ZN5nekrs5setupEP19ompi_communicator_tiiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_S7_S7_+0x2446)[0x15108f571be6]
[gadi-gpu-v100-0032:282634] [17] /home/563/wl8035/.local/nekrs/bin/nekrs[0x40a0a9]
[gadi-gpu-v100-0032:282634] [18] /lib64/libc.so.6(__libc_start_main+0xf3)[0x153cfd3d44a3]
[gadi-gpu-v100-0032:282631] [19] /home/563/wl8035/.local/nekrs/bin/nekrs[0x40a6ce]
[gadi-gpu-v100-0032:282631] *** End of error message ***
/lib64/libc.so.6(__libc_start_main+0xf3)[0x15108bd674a3]
[gadi-gpu-v100-0032:282634] [19] /home/563/wl8035/.local/nekrs/bin/nekrs[0x40a6ce]
[gadi-gpu-v100-0032:282634] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 7 with PID 0 on node gadi-gpu-v100-0032 exited on signal 6 (Aborted).
--------------------------------------------------------------------------

Any help would be much appreciated.

Thanks,
Wilson

[stgeke]

By default nekRS maps each MPI tasks to a GPU (id is derived based on the local MPI rank). In your case it looks like the requested device IDs (0,1,2,3) are not available.

[wlu00]

Hi stgeke,

Thanks for the reply. Indeed it was an issue with mapping MPI ranks to nonexistent GPUs.

For reference, we we're able to solve the issue using the commands

mpirun -np 1 $NEKRS_HOME/bin/nekrs --setup ext_cyl_Re3900 --backend CUDA --build-only $PBS_NGPUS > compile.txt

mpirun -np $PBS_NGPUS --map-by ppr:1:numa $NEKRS_HOME/bin/nekrs --setup ext_cyl_Re3900 > logfile.txt