Following the Prostate Example for a Liver & Liver Tumor Segmentation Task

[atnassar]

Hello,
I am following the steps in the prostate example for liver & liver tumor segmentation in a similar way the prostate example was implemented.
I used the LITS dataset (liver and liver tumor segmentation challenge) which contains 131 nifti images (512x512) for the volumes and 131 nii images for the segmentations. I split the 131 nii images and masks on 4 sites and ran the experiments similar to the prostate example.

I faced the below errors that need your help to resolve:

1. Intel MKL ERROR: Parameter 4 was incorrect on entry to cblas_dgemm.

2. pixdim[0] (qfac) should be 1 (default) or -1; setting qfac to 1

3. LearnerExecutor - ERROR - [run=2, peer=example_project, peer_run=2, task_name=train, task_id=ac3edbb7-7c99-4988-95ff-22e3b42ef483]: learner execute exception: Caught RuntimeError in DataLoader worker process 0.
   Original Traceback (most recent call last):
   File "/home/anaconda3/envs/nvflare/lib/python3.8/site-packages/monai/data/utils.py", line 274, in list_data_collate
   return default_collate(data)
   File "/home/anaconda3/envs/nvflare/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 74, in default_collate
   return {key: default_collate([d[key] for d in batch]) for key in elem}
   File "/home/anaconda3/envs/nvflare/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 74, in
   return {key: default_collate([d[key] for d in batch]) for key in elem}
   File "/home/anaconda3/envs/nvflare/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 56, in default_collate
   return torch.stack(batch, 0, out=out)
   RuntimeError: stack expects each tensor to be equal size, but got [1, 224, 224, 32] at entry 0 and [1, 224, 129, 32] at entry 2

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/anaconda3/envs/nvflare/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/home/anaconda3/envs/nvflare/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
return self.collate_fn(data)
File "/home/anaconda3/envs/nvflare/lib/python3.8/site-packages/monai/data/utils.py", line 285, in list_data_collate
raise RuntimeError(re_str) from re
RuntimeError: stack expects each tensor to be equal size, but got [1, 224, 224, 32] at entry 0 and [1, 224, 129, 32] at entry 2

MONAI hint: if your transforms intentionally create images of different shapes, creating your DataLoader with collate_fn=pad_list_data_collate might solve this problem (check its documentation).

** Note: I think [224, 224, 32] is the ROI size used in the prostate example but not sure what it should be for liver and liver tumor segmentation.

[holgerroth]

Hi @atnassar,

it looks like there's some nii volume that has a smaller dimension than what you specified in your ROI crop size.

After applying Spacingd, some volumes might be smaller than [224, 224, 32] in one of the dimensions. Try a smaller ROI size, like 96^3. Have a look here for good hyperparameters https://catalog.ngc.nvidia.com/orgs/nvidia/teams/med/models/clara_pt_liver_and_tumor_ct_segmentation for liver & tumor segmentation.

[atnassar]

Thank you @holgerroth for your advice! I will implement smaller ROI size and will check the liver and tumor segmentation model from Clara.

I appreciate your usual support

[atnassar]

Hi @holgerroth, I could not find the model py file to go over the structure and the used hyperparameters for training and validation.
Could you help please?

[ZiyueXu77]

For different tasks, most possibly you can not directly reuse the transforms / configurations purposed for prostate to liver, given that they are different modalities and different organs. You will need to write your own customized learner similar to this one: https://github.com/NVIDIA/NVFlare/blob/main/examples/prostate/pt/learners/supervised_prostate_learner.py
This question is better suited for MONAI rather than Flare, I will suggest you first get a standalone, non-FL, MONAI trainer for LiTS running, then replace the proper sections in the learner script.

[atnassar]

Thank you @ZiyueXu77 for your prompt reply!

I will try building a learner script using the available Clara models for liver and liver tumor segmentation. I would like to build my own FL application using NVFlare but the structure is somehow complicated. Is there any videos from NVIDIA explaining the NVFlare framework and structure in the provided examples? That would be really helpful to understand the structure and the interaction between various .sh, json, and .py files to build our own FL applications.

Your usual support is really appreciated!

[holgerroth]

We just had the NVFlare Dev Day at GTC https://www.nvidia.com/gtc/session-catalog/?search=SE1991&search=SE1991&tab.scheduledorondemand=1583520458947001NJiE#/session/1638904712976001sY6W

[holgerroth]

The training transforms are defined here in the prostate example

NVFlare/examples/prostate/pt/learners/supervised_prostate_learner.py

Line 124 in bb45e7d

self.transform_train = Compose(

[atnassar]

Yes @holgerroth , but I didn't find a similar code for the liver and liver tumor segmentation model in the link you shared [https://catalog.ngc.nvidia.com/orgs/nvidia/teams/med/models/clara_pt_liver_and_tumor_ct_segmentation]

[holgerroth]

I see. There's a training configuration file: configs/config_train.json. This is Clara Train based which is basically a wrapper around MONAI. You can see the same transform names and argument names as you would use when writing python code with MONAI.

[holgerroth]

Hi @atnassar, was your issue resolved?