Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CUDA-aware MPI failure on some Nvidia GPUs #62

Closed
luraess opened this issue Jan 10, 2025 · 1 comment
Closed

CUDA-aware MPI failure on some Nvidia GPUs #62

luraess opened this issue Jan 10, 2025 · 1 comment

Comments

@luraess
Copy link
Member

luraess commented Jan 10, 2025

Currently, CUDA-aware MPI may fail on some GPUs because of an issue with the CUDA.resize! function used in the buffer allocation:

Chmy.jl/src/Distributed/stack_allocator.jl

Line 45 in c011eef

resize!(sa.buffer, ceil(Int, 1.5 * sz)) # add extra capacity

A fix got merged in CUDA.jl JuliaGPU/CUDA.jl#2613 and a patch release should be tagged soon.
@luraess luraess changed the title CUDA-aware MPI failure in GH200 CUDA-aware MPI failure on Nvidia GH200 Jan 10, 2025
@luraess luraess changed the title CUDA-aware MPI failure on Nvidia GH200 CUDA-aware MPI failure on some Nvidia GPUs Jan 10, 2025
@luraess
Copy link
Member Author

luraess commented Jan 15, 2025

Closing this as the fix got released https://github.com/JuliaGPU/CUDA.jl/releases/tag/v5.6.1

@luraess luraess closed this as completed Jan 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@luraess