RuntimeError: applying transform <monai.transforms.croppad.dictionary.RandCropByPosNegLabeld

[talhaanwarch]

After some successful iteration, I got this error.
monai versions: 0.6.0+34.g7f42efe0

RuntimeError                              Traceback (most recent call last)
<ipython-input-24-7114b32222fa> in <module>()
----> 1 get_ipython().run_cell_magic('time', '', 'trainer.fit(model)')

23 frames
<decorator-gen-53> in time(self, line, cell, local_ns)

<timed eval> in <module>()

/usr/local/lib/python3.7/dist-packages/torch/_utils.py in reraise(self)
    423             # have message field
    424             raise self.exc_type(message=msg)
--> 425         raise self.exc_type(msg)
    426 
    427 

RuntimeError: Caught RuntimeError in DataLoader worker process 1.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/monai/transforms/transform.py", line 84, in apply_transform
    return _apply_transform(transform, data, unpack_items)
  File "/usr/local/lib/python3.7/dist-packages/monai/transforms/transform.py", line 52, in _apply_transform
    return transform(parameters)
  File "/usr/local/lib/python3.7/dist-packages/monai/transforms/croppad/dictionary.py", line 1097, in __call__
    self.randomize(label, fg_indices, bg_indices, image)
  File "/usr/local/lib/python3.7/dist-packages/monai/transforms/croppad/dictionary.py", line 1082, in randomize
    fg_indices_, bg_indices_ = map_binary_to_indices(label, image, self.image_threshold)
  File "/usr/local/lib/python3.7/dist-packages/monai/transforms/utils.py", line 268, in map_binary_to_indices
    label_flat = np.any(label, axis=0).ravel()  # in case label has multiple dimensions
  File "<__array_function__ internals>", line 6, in any
  File "/usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py", line 2330, in any
    return _wrapreduction(a, np.logical_or, 'any', axis, None, out, keepdims=keepdims)
  File "/usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py", line 85, in _wrapreduction
    return reduction(axis=axis, out=out, **passkwargs)
TypeError: any() received an invalid combination of arguments - got (out=NoneType, axis=int, ), but expected one of:
 * ()
 * (int dim, bool keepdim)
      didn't match because some of the keywords were incorrect: out, axis
 * (name dim, bool keepdim)
      didn't match because some of the keywords were incorrect: out, axis


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.7/dist-packages/monai/data/dataset.py", line 95, in __getitem__
    return self._transform(index)
  File "/usr/local/lib/python3.7/dist-packages/monai/data/dataset.py", line 284, in _transform
    return self._post_transform(pre_random_item)
  File "/usr/local/lib/python3.7/dist-packages/monai/data/dataset.py", line 230, in _post_transform
    item_transformed = apply_transform(_transform, item_transformed)
  File "/usr/local/lib/python3.7/dist-packages/monai/transforms/transform.py", line 108, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.croppad.dictionary.RandCropByPosNegLabeld object at 0x7fa3f126d750>

This is my transformation

train_transforms = T.Compose(
            [
                T.LoadImaged(keys=["image", "label"]),
                T.AddChanneld(keys=["image", "label"]),
                T.Spacingd(keys=["image", "label"],pixdim=(1.0, 1.0, 1.0),mode=("bilinear", "nearest"),),
                T.Orientationd(keys=["image", "label"], axcodes="RAS"),
                #T.ScaleIntensityd(keys=["image"]),
                T.ScaleIntensityRanged(keys=["image"],a_min=-11,a_max=264,b_min=0.0,b_max=1.0,clip=True,),
                T.CropForegroundd(keys=["image", "label"], source_key="image"),
                T.RandCropByPosNegLabeld(keys=["image", "label"],label_key="label",spatial_size=(128, 128, 64),pos=1,neg=1,
                    num_samples=4,image_key="image", image_threshold=0,),
                T.RandAxisFlipd(keys=["image", "label"],prob=0.20,),
                T.RandShiftIntensityd(keys=["image"],offsets=0.1),
                T.RandAdjustContrastd(keys=["image"]),
                T.RandHistogramShiftd(keys=["image"]),
                T.RandGaussianNoised(keys=["image"]),
                T.RandRotated(keys=["image", "label"],range_x=0.1, range_y=0.1, range_z=0.1,),
                T.RandRotate90d(keys=["image", "label"],prob=0.2),
                T.RandZoomd(keys=["image", "label"]),
                T.EnsureTyped(keys=["image", "label"]),

            ]
        )

I am using PersistentDataset

ds = PersistentDataset(data=train_files, transform=train_transforms, cache_dir=persistent_cache)
loader=DataLoader(ds, batch_size=self.batch_size, shuffle=True,num_workers=self.numworker, collate_fn=list_data_collate)

This error does not arise in Dataset

[Nic-Ma]

Hi @talhaanwarch ,

I think maybe your numpy version is not compatible? Could you please help try again with numpy>=1.17?

Thanks.

[talhaanwarch]

thanks. is it normal that cache folder take 100+ gb space for 13 gb data?

[Nic-Ma]

Hi @talhaanwarch ,

I think that could be possible because the raw image files should be smaller than saved raw numpy arrays, especially after spacing transform it may expand the size and the data type became float number.

Thanks.