Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In ncarenv/23.09 ncarcompilers does not wrap nvcc #12

Open
roryck opened this issue Nov 15, 2023 · 3 comments
Open

In ncarenv/23.09 ncarcompilers does not wrap nvcc #12

roryck opened this issue Nov 15, 2023 · 3 comments
Assignees

Comments

@roryck
Copy link

roryck commented Nov 15, 2023

The ncarcompilers isn't wrapping nvcc

module --force purge
module load ncarenv/23.09
module load nvhpc cray-mpich ncarcompilers
which nvcc
/glade/u/apps/common/23.08/spack/opt/spack/nvhpc/23.7/Linux_x86_64/23.7/compilers/bin/nvcc

Which leads to things like:

nvcc -c test.c
test.c:1:10: fatal error: mpi.h: No such file or directory
 #include <mpi.h>
          ^~~~~~~
compilation terminated.

unless you manually add

-I${CRAY_MPICH_PREFIX}/include
@vanderwb
Copy link
Collaborator

I think the issue here isn't so much that nvcc isn't being wrapped, but rather that nvcc isn't calling the MPI compiler wrapper by default:

[16:59] nvcc-test$ nvcc -c mpi.c
mpi.c:16:10: fatal error: mpi.h: No such file or directory
 #include "mpi.h"
          ^~~~~~~
compilation terminated.
[16:59] nvcc-test$ nvcc -ccbin mpicxx -c mpi.c
[16:59] nvcc-test$

I don't see a nice environment variable to set a default for that, and other options seem sub-optimal. The only thing I've found online that I could do would be to set NVCC_PREPEND_FLAGS:

[17:12] nvcc-test$ export NVCC_PREPEND_FLAGS=-ccbin=mpicxx
[17:12] nvcc-test$ nvcc -c mpi.c
[17:12] nvcc-test$

But that would hijack this variable which may be useful to folks/build systems? I'm thinking this may just be a user education situation, in which we need to inform people to use -ccbin. What do you think?

@vanderwb vanderwb self-assigned this Nov 18, 2023
@roryck
Copy link
Author

roryck commented Nov 18, 2023

If ncarcompilers does wrap nvcc, can't it include the header path when MPI is loaded?

When using nvcc there are probably other C/C++ compilers in the environment, e.g. icx or nvc, and it makes sense for mpicc to prefer wrapping those over wrapping nvcc. Maybe it makes more sense to declare the CUDA compiler to be mpicc with nvcc as the underlying, inside makefiles an so on. I know how to do that with some MPIs, but not cray-mpich. I'll look into that as a better solution.

@vanderwb
Copy link
Collaborator

Well, ncarcompilers doesn't wrap nvcc, it wraps the compiler that nvcc calls. That distinction probably doesn't matter.

I can certainly add include and library paths for cray-mpich that the NCAR wrapper would pick up on. It just didn't seem necessary since the Cray wrapper CC will do this too. Unless of course you don't call CC, as is happening here with nvcc.

I'd like to confirm that having ncarcompilers do this doesn't upset the Cray wrapper. If not, I'll add those -I and -L flags soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants