RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR

[Aayush2003Gupta]

GaNDLF Version

gandlf 0.0.16
Desktop (please complete the following information):

• Tried both on Ubuntu 18.04 and on Google Colab

How did you install GaNDLF
pip install gandlf

Dataset description
Working on the BraTS competition 2023

Describe your question/problem

Even after preparing the .csv as per the gandlf_constructCSV.py and using the .config_segmentation_brats.yaml , I get the error
On changing as image.data = image.data.float() I get the error:-
RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR

Also on doing torch.backends.cudnn.enabled = False I end up getting:-
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DFALT_TENSOR_OP)

Please tell me the error I have tried every possible thing. Thanks a lot for your help!!!!

[AashrayGupta2003]

I have the same error popping up. Could you please resolve, any help is deeply awaited, thanks!

[sarthakpati]

Thanks for your interest in GaNDLF!

You mentioned this change:

On changing as image.data = image.data.float()

Where and why would this be needed? We have performed extensive tests on multi-GPU systems on the current code base and everything works as expected.

How are you ensuring that the current CUDA device is visible to GaNDLF? Before calling gandlf_run, the environment variable CUDA_VISIBLE_DEVICES needs to be set to the appropriate GPU device.

[Aayush2003Gupta]

I changed the function def calculate_std_ranges(self, image: torch.Tensor) -> tuple: in /GANDLF/data/augmentation/blur_enhanced.py

It was required because the dtype of 1 image.data came out to be int16 which produced the error :-
RuntimeError : std and var only support floating point and complex dtypes

I had set the CUDA_VISIBLE_DEVICES environment variable already as export CUDA_VISIBLE_DEVICES=0 , I am using a Tesla V100 (also tried the same on Google Colab T4 GPU)

[sarthakpati]

Hmm, I am unsure why that line would be needed. Regardless, it would be very helpful if you can give me an example using the unit test data so that I can reproduce the error.

You can't add a .float() call on its own, unless you are computing on the default device (which is CPU). You need to add it with a .to call, like shown in https://github.com/mlcommons/GaNDLF/blob/master/GANDLF/compute/training_loop.py#L98

[Aayush2003Gupta]

Thanks a lot the error has been resolved .😀
I was not giving the full path in the train_data.csv file as mentioned ....

Thanks a lot for your time and support
Aayush Gupta

[sarthakpati]

Thanks a lot the error has been resolved .😀

Thanks for the update!

So you didn't need to make any changes in the code?

[Aayush2003Gupta]

No the code worked perfectly fine .

Thank you 🙏

[sarthakpati]

Awesome! Glad to have heard that. 😄

All the best with your work!