GPU memory call problem

[DiscoverXV]

I am running the simulation in the example and transitioning from CPU simulation to GPU simulation. I can get the fluid simulation results normally in CPU simulation, but after using GPU, an error "Invalid permissions (2)" will appear. The specific error details are as follows: (The MPI RANK = 2)

...
generating mesh ...
loading mesh from nek ... Nelements: 320, NboundaryIDs: 3, NboundaryFaces: 496 done (5.4389e-05s)
polynomial order N: 7, over-integration order cubN: 10
meshParallelGatherScatterSetup N=7
timing gs: 5.00e-05s 3.11e-04s 3.72e-04s 4.28e-04s 4.04e-04s [D74563CAFABD9:33314] *** Process received signal ***
[D74563CAFABD9:33314] Signal: Segmentation fault (11)
[D74563CAFABD9:33314] Signal code: Invalid permissions (2)
[D74563CAFABD9:33314] Failing at address: 0x70264f000
[D74563CAFABD9:33313] *** Process received signal ***
[D74563CAFABD9:33313] Signal: Segmentation fault (11)
[D74563CAFABD9:33313] Signal code: Invalid permissions (2)
[D74563CAFABD9:33313] Failing at address: 0x70264f000
[D74563CAFABD9:33314] [ 0] [D74563CAFABD9:33313] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f874166a520]
[D74563CAFABD9:33314] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7ffaf0478520]
[D74563CAFABD9:33313] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x1a07cd)[0x7f87417c87cd]
[D74563CAFABD9:33314] [ 2] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_btl_vader.so(+0x3244)[0x7f873df8a244]
[D74563CAFABD9:33314] [ 3] /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_pml_ob1.so(mca_pml_ob1_send_request_start_prepare+0x44)[0x7f873d9db784]
...

If you have any suggestions or questions, we can discuss further. Thank you for your help.
Best regards.

[stgeke]

Try to use latest next branch and run one of the examples e.g. turbPipe first.

[stgeke]

The error message suggest that your using an MPI implementation which doesn't support GPU buffers. To disable GPU aware MPI just set the following env-var NEKRS_GPU_MPI=0.

[DiscoverXV]

Thanks for your help, I have successfully run it on the "Next" branch of NEKRS.

I would like to ask, when I use the original example turbPipePeriodic, when N = 7, I will get the error "CUDA Error [2]: CUDA_ERROR_OUT_OF_MEMORY". Therefore, I reduced it to N = 5, and the example can run.

My GPU configuration is as follows:

If I use "mpirun --mca osc ucx -np 1 nekrs --setup turbPipe", does it mean that as mentioned in Problem#467, CPU core : GPU = 1:1. Similarly, is it possible to achieve CPU core : GPU = n:1?

Thanks again for your help.

[stgeke]

Why do you want to run nekRS on this GPU? Running it on the CPU will likely be faster. Each MPI rank typically uses a separate GPU, so using the same GPU across multiple ranks doesn't make sense and would result in slower performance.

[DiscoverXV]

Thanks for your reply, which helped me better understand the architecture of NEKRS.

As for using this GPU for simulation, I am doing a preliminary attempt of NEKRS on my own computer. After the initial test, I will use HPC for large model simulation.

Thanks again for your reply.
Best regards,