CUDA Error

[Meddebma]

Hi,

I tried to run classification on my own dataset on colab, unfortunately I have a problem with the GPU, I get this error:
"RuntimeError: CUDA error: device-side assert triggered"
I also deleted pin_memory=torch.cuda.is_available() from the loaders

Here is the code:
https://github.com/Meddebma/pyradiomics/blob/master/Projekt2.ipynb

Could you help me with this?
Thank you

[rijobro]

Hi @Meddebma, I moved your tutorials issue to a discussion on the main repository. I think CUDA side-assert errors can occur when, for example, indexing out of range. In this case, I think debugging can be done by working on the CPU. I know this will be slow, but hopefully you'll have to run it for the first epoch. You can make your images considerably smaller (even 2d) if that would help.

[Meddebma]

Ah now I get it!
I think in some patients there were CT-Scans in different Kontrast-media phases, that is why there is an extra channel. How can I find out whether a channel is empty and discard one of the channels?

[Meddebma]

Dear @rijobro ,

I found out that some of the CT-Scans have this shape: torch.Size([1, 512, 512, 56, 1, 2],
I tried to remove the two last channels with squeezedim() but it did not work, here is what I tried:

class Squeeze_ChannelFirst(Transform):
    def __init__(self):
        self.channel_first = AsChannelFirst()
        self.squeezedim = SqueezeDim()
    def __call__(self, im):
        ndims = im.ndim
        if ndims > 5:
            return self.squeezedim(dim=6)   
          elif ndims == 5:
            return self.squeezedim(dim=5)
        else:
            return self.channel_first(im)

is there another way to do it?

[rijobro]

Don't think you're using SqueezeDim correctly. The constructor should take dim as argument, and when you call it (self.squeezedim), you should pass im as argument.

You could squeeze your image at the beginning to remove all excess dimensions eg: [1, 320, 320, 320, 1, 2] -> [320, 320, 320, 2]. And from there you use the same logic as before:

class Squeeze_ChannelFirst(Transform):
    def __init__(self):
        self.channel_first = AsChannelFirst()
        self.add_channel = AddChannel()
        # squeeze all singleton dimensions eg: (1, 320, 320, 320, 1, 2) -> (320, 320, 320, 2)
        self.squeezedim = SqueezeDim(dim=None)
    def __call__(self, im):
        im = self.squeezedim(im)
        ndims = im.ndim
        if ndims == 4:
            return self.channel_first(im)
        elif ndims == 3:
            return self.add_channel(im)
        else:
            raise RuntimeError

[rijobro]

How can I find out whether a channel is empty and discard one of the channels?

You'll have to create a transform that checks the two channels and return the one that you consider to be "non-empty". I can't tell you how to do that as I don't know what you mean by "empty". All zeros? NaNs?

[Meddebma]

@rijobro
Thank you for your help, the problem is solved and all the data are now homogeneous :)