Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Localizer based labeling #46

Merged
merged 23 commits into from
Sep 27, 2024
Merged

Localizer based labeling #46

merged 23 commits into from
Sep 27, 2024

Conversation

yw7
Copy link
Collaborator

@yw7 yw7 commented Sep 3, 2024

Currently, the iterative_label.py script does not support images without any of the landmarks (C2, T1, T12, Sacrum).
To solve this issue, we can update the iterative_label.py script to get localizer segmentation as an input to initialize labels for discs and vertebrae. The localizer have to be in same space, but no need for exact match, only that most of the voxels in the first vertebrae and disc will be overlapping with the provided image. The script acts as follows:

  1. Transform localizer into the image space.
  2. For upper vertebrae and upper disc in the localizer:
    1. Get the superior disc/vertebra label from the matched disc/vertebra in localizer.
    2. Loop over the rest of the discs/vertebrae in the input image and map them sequentially as usual.

Instructions for new localizer based labeling:

totalspineseg/README.md

Lines 51 to 86 in 07069b9

## Installation
1. Open a `bash` terminal in the directory where you want to work.
1. Create the installation directory:
```bash
mkdir TotalSpineSeg
cd TotalSpineSeg
```
1. Create and activate a virtual environment (highly recommended):
```bash
python3 -m venv venv
source venv/bin/activate
```
1. Clone and install this repository:
```bash
git clone https://github.com/neuropoly/totalspineseg.git
python3 -m pip install -e totalspineseg
```
1. For CUDA GPU support, install **PyTorch** following the instructions on their [website](https://pytorch.org/). Be sure to add the `--upgrade` flag to your installation command to replace any existing PyTorch installation.
Example:
```bash
python3 -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 --upgrade
```
1. Set the path to TotalSpineSeg and data folders in the virtual environment:
```bash
mkdir data
export TOTALSPINESEG="$(realpath totalspineseg)"
export TOTALSPINESEG_DATA="$(realpath data)"
echo "export TOTALSPINESEG=\"$TOTALSPINESEG\"" >> venv/bin/activate
echo "export TOTALSPINESEG_DATA=\"$TOTALSPINESEG_DATA\"" >> venv/bin/activate
```

checkout this branch:

cd totalspineseg && git checkout localizer-based-labeling && cd ..

totalspineseg/README.md

Lines 175 to 209 in 07069b9

## Localizer based labeling
TotalSpineSeg supports using localizer images to improve the labeling process, particularly useful for images with different fields of view (FOV) where landmarks like C1 and sacrum may not be visible. It uses localizer information to accurately label vertebrae and discs in the main image.
![Localizer](https://github.com/user-attachments/assets/5acf0208-a322-46f9-bbde-b3c961a87ec4)
Example of directory structure:
```
.
├── images/
│ ├── sub-01_T2w.nii.gz
│ └── sub-02_T2w.nii.gz
└── localizers/
├── sub-01_T1w.nii.gz
└── sub-02_T1w.nii.gz
```
In this example, main images are placed in the `images` folder and corresponding localizer images in the `localizers` folder.
To use localizer-based labeling:
```bash
# Process localizer images
totalspineseg localizers localizers_output
# Run model on main images using localizer output
totalspineseg images output --loc localizers_output/step2_output --suffix _T2w --loc-suffix _T1w
```
- `--loc`: Specifies the path to the localizer output
- `--suffix`: Suffix for the main images (e.g., "_T2w")
- `--loc-suffix`: Suffix for the localizer images (e.g., "_T1w")
Note: If the localizer and main image files have the same names, you can omit the `--suffix` and `--loc-suffix` arguments.

Localizer

@yw7 yw7 linked an issue Sep 3, 2024 that may be closed by this pull request
@yw7 yw7 force-pushed the localizer-based-labeling branch from b41af61 to 96c0ca8 Compare September 3, 2024 14:08
@yw7 yw7 mentioned this pull request Sep 5, 2024
@yw7 yw7 added documentation Improvements or additions to documentation enhancement New feature or request labels Sep 5, 2024
Copy link
Member

@jcohenadad jcohenadad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a lot of changes in this PR, so it is impossible for me to review all of them exhaustively. I could try the new segmentation method on data, but for that I would need instructions on how to test this new segmentation method (what command to run, etc.)

@yw7
Copy link
Collaborator Author

yw7 commented Sep 6, 2024

There is a lot of changes in this PR, so it is impossible for me to review all of them exhaustively. I could try the new segmentation method on data, but for that I would need instructions on how to test this new segmentation method (what command to run, etc.)

Thank you for reviewing the PR!
Regarding the changes, I've fix the issue with rebase to main.
I've aded instructions in the readme for new localizer based labeling: is it ok?

totalspineseg/README.md

Lines 51 to 86 in 07069b9

## Installation
1. Open a `bash` terminal in the directory where you want to work.
1. Create the installation directory:
```bash
mkdir TotalSpineSeg
cd TotalSpineSeg
```
1. Create and activate a virtual environment (highly recommended):
```bash
python3 -m venv venv
source venv/bin/activate
```
1. Clone and install this repository:
```bash
git clone https://github.com/neuropoly/totalspineseg.git
python3 -m pip install -e totalspineseg
```
1. For CUDA GPU support, install **PyTorch** following the instructions on their [website](https://pytorch.org/). Be sure to add the `--upgrade` flag to your installation command to replace any existing PyTorch installation.
Example:
```bash
python3 -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 --upgrade
```
1. Set the path to TotalSpineSeg and data folders in the virtual environment:
```bash
mkdir data
export TOTALSPINESEG="$(realpath totalspineseg)"
export TOTALSPINESEG_DATA="$(realpath data)"
echo "export TOTALSPINESEG=\"$TOTALSPINESEG\"" >> venv/bin/activate
echo "export TOTALSPINESEG_DATA=\"$TOTALSPINESEG_DATA\"" >> venv/bin/activate
```

checkout this branch:

cd totalspineseg && git checkout localizer-based-labeling && cd ..

totalspineseg/README.md

Lines 175 to 209 in 07069b9

## Localizer based labeling
TotalSpineSeg supports using localizer images to improve the labeling process, particularly useful for images with different fields of view (FOV) where landmarks like C1 and sacrum may not be visible. It uses localizer information to accurately label vertebrae and discs in the main image.
![Localizer](https://github.com/user-attachments/assets/5acf0208-a322-46f9-bbde-b3c961a87ec4)
Example of directory structure:
```
.
├── images/
│ ├── sub-01_T2w.nii.gz
│ └── sub-02_T2w.nii.gz
└── localizers/
├── sub-01_T1w.nii.gz
└── sub-02_T1w.nii.gz
```
In this example, main images are placed in the `images` folder and corresponding localizer images in the `localizers` folder.
To use localizer-based labeling:
```bash
# Process localizer images
totalspineseg localizers localizers_output
# Run model on main images using localizer output
totalspineseg images output --loc localizers_output/step2_output --suffix _T2w --loc-suffix _T1w
```
- `--loc`: Specifies the path to the localizer output
- `--suffix`: Suffix for the main images (e.g., "_T2w")
- `--loc-suffix`: Suffix for the localizer images (e.g., "_T1w")
Note: If the localizer and main image files have the same names, you can omit the `--suffix` and `--loc-suffix` arguments.

yw7 added 12 commits September 7, 2024 12:16
Enhanced the segmentation script to include localizer images, enabling detection of the first vertebrae and disc when initial labels are absent. Introduced loc-suffix, loc-disc-labels, loc-vertebrea-labels, default-superior-disc, and default-superior-vertebrea parameters to improve labeling accuracy and reduce errors. Updated command-line arguments and processing steps accordingly.

Implements #43
Updated help texts for '--loc-disc-labels' and '--loc-vertebrea-labels' to specify their use in the localizer for detecting first disc and vertebrae respectively. This improves the clarity of the argument purposes for better usability and understanding by end-users.
Expanded the help description for the --locs-dir argument to provide clearer instructions. Now it specifies that the localizer segmentations are transformed and used for detecting the first vertebrae and disc if the initial label is missing. This enhances user understanding, particularly in scenarios where the initial labeling is absent.
Extended inference script to support localizers segmentations, enhancing the detection of vertebrae and discs when certain reference points are missing in the images. Added new arguments for localizer directory, input and output suffixes, and adjusted the iterative labeling process to handle the additional configuration.

This improves the robustness and flexibility of the inference process, especially in cases where standard detection is insufficient or fails due to missing landmarks.
Improved the clarity and precision of masking logic by introducing an intermediate variable, enhancing readability and reducing potential errors. Revised the calculation of the target label to align with the new masking approach for better accuracy. This change addresses potential edge cases where the masks intersect, ensuring more robust label handling.
Adjusted the conditional logic to ensure the localizer image is loaded only when the localizer path exists, addressing potential `NoneType` errors and improving robustness in image loading routines.
Enhanced README with instructions for using localizer images to improve vertebral labeling, addressing issues with varying fields of view. Included data structure description and example commands for processing. This ensures more accurate and consistent labeling across different image sets.
Refined document structure for better readability, including updating titles formatting and clarifying sections on localizer-based labeling. Ensured a single, accurate description of using localizer images to streamline and enhance label accuracy. Removed duplication and enhanced visual support with relevant image placement.
Condensed the section linking to the PDF preview by combining multiple lines into one for improved readability. This change helps streamline the document and makes it more concise.
Updated section titles in README to streamline and simplify 'Output Examples' to 'Examples'. This enhances readability and consistency in the document. No functional changes to the content.
@yw7 yw7 force-pushed the localizer-based-labeling branch from d8810e6 to 30f3243 Compare September 7, 2024 09:20
yw7 added 7 commits September 7, 2024 12:47
Renamed "Examples" section to "Results" to better reflect the content. This improves clarity by providing a more accurate description of the model's output display section.
Included an important note in the README about the limitations of the spinal cord segmentation provided by TotalSpineSeg. The added disclaimer highlights that this tool is not validated for CSA analysis and should not be used for assessing spinal cord compressions or MS lesions. Users are directed to use Spinal Cord Toolbox for validated CSA analysis tools.
Updated README and inference script to support processing of a single .nii.gz file in addition to folder inputs. Renamed arguments for consistency and improved help documentation.

- `input_dir` to `input`
- `output_dir` to `output`
- `localizers-dir` to `loc`
- `localizers-suffix` to `loc-suffix`

These changes enhance usability by allowing more flexible model input options and clearer documentation.
Enhanced the disclaimer regarding the valid uses of TotalSpineSeg, emphasizing the lack of validation for cross-sectional area (CSA) measurements and various spinal conditions. Simplified explanation for omitting suffix arguments in cases where localizer and main images have identical names.
Enhanced the localizer search path to include nested directories, improving the comprehensiveness of localizer discovery. Added functionality to generate preview images for the localizers, assisting in quick visual validation. These changes streamline the workflow and enhance usability.
Corrected the spelling of "vertebrea" to "vertebrae" in various files
to maintain consistency and accuracy in anatomical terminology. This
change affects function parameters, documentation, and help texts,
providing clearer and more professional codebase and usage instructions.

Fix #44
@yw7 yw7 linked an issue Sep 9, 2024 that may be closed by this pull request
Copy link
Member

@jcohenadad jcohenadad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue:

julien-macbook:~/code/totalspineseg $ python3 -m venv venv
julien-macbook:~/code/totalspineseg $ source venv/bin/activate 
(venv) julien-macbook:~/code/totalspineseg $ python3 -m pip install -e totalspineseg 
Obtaining file:///Users/julien/code/totalspineseg/totalspineseg
ERROR: file:///Users/julien/code/totalspineseg/totalspineseg does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

[notice] A new release of pip is available: 23.0.1 -> 24.2
[notice] To update, run: pip install --upgrade pip
(venv) julien-macbook:~/code/totalspineseg $ pip install --upgrade pip
Requirement already satisfied: pip in ./venv/lib/python3.10/site-packages (23.0.1)
Collecting pip
  Downloading pip-24.2-py3-none-any.whl (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 33.7 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.0.1
    Uninstalling pip-23.0.1:
      Successfully uninstalled pip-23.0.1
Successfully installed pip-24.2
(venv) julien-macbook:~/code/totalspineseg $ python3 -m pip install -e totalspineseg 
Obtaining file:///Users/julien/code/totalspineseg/totalspineseg
ERROR: file:///Users/julien/code/totalspineseg/totalspineseg does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

(venv) julien-macbook:~/code/totalspineseg $ ll
total 40
drwxr-xr-x   9 julien  staff    288 10 Sep 16:52 .
drwxr-xr-x  43 julien  staff   1376 10 Sep 16:52 ..
drwxr-xr-x  12 julien  staff    384 10 Sep 16:52 .git
-rw-r--r--   1 julien  staff   3316 10 Sep 16:52 .gitignore
-rw-r--r--   1 julien  staff  10262 10 Sep 16:52 README.md
-rw-r--r--   1 julien  staff   3308 10 Sep 16:52 pyproject.toml
drwxr-xr-x   6 julien  staff    192 10 Sep 16:52 scripts
drwxr-xr-x   6 julien  staff    192 10 Sep 16:52 totalspineseg
drwxr-xr-x   6 julien  staff    192 10 Sep 16:52 venv

correct instruction:

(venv) julien-macbook:~/code/totalspineseg $ python3 -m pip install -e .

nevermind-- i see that we were not supposed to go into the git cloned repos (which is usually common practice, so it's a bit confusing)-- i suggest to instruct to go into the repos and then install from there

@jcohenadad
Copy link
Member

image

is this really necessary? can't we have a CLI that just points to the image to run the inference on?

@yw7
Copy link
Collaborator Author

yw7 commented Sep 10, 2024

Issue:

julien-macbook:~/code/totalspineseg $ python3 -m venv venv
julien-macbook:~/code/totalspineseg $ source venv/bin/activate 
(venv) julien-macbook:~/code/totalspineseg $ python3 -m pip install -e totalspineseg 
Obtaining file:///Users/julien/code/totalspineseg/totalspineseg
ERROR: file:///Users/julien/code/totalspineseg/totalspineseg does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

[notice] A new release of pip is available: 23.0.1 -> 24.2
[notice] To update, run: pip install --upgrade pip
(venv) julien-macbook:~/code/totalspineseg $ pip install --upgrade pip
Requirement already satisfied: pip in ./venv/lib/python3.10/site-packages (23.0.1)
Collecting pip
  Downloading pip-24.2-py3-none-any.whl (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 33.7 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.0.1
    Uninstalling pip-23.0.1:
      Successfully uninstalled pip-23.0.1
Successfully installed pip-24.2
(venv) julien-macbook:~/code/totalspineseg $ python3 -m pip install -e totalspineseg 
Obtaining file:///Users/julien/code/totalspineseg/totalspineseg
ERROR: file:///Users/julien/code/totalspineseg/totalspineseg does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

(venv) julien-macbook:~/code/totalspineseg $ ll
total 40
drwxr-xr-x   9 julien  staff    288 10 Sep 16:52 .
drwxr-xr-x  43 julien  staff   1376 10 Sep 16:52 ..
drwxr-xr-x  12 julien  staff    384 10 Sep 16:52 .git
-rw-r--r--   1 julien  staff   3316 10 Sep 16:52 .gitignore
-rw-r--r--   1 julien  staff  10262 10 Sep 16:52 README.md
-rw-r--r--   1 julien  staff   3308 10 Sep 16:52 pyproject.toml
drwxr-xr-x   6 julien  staff    192 10 Sep 16:52 scripts
drwxr-xr-x   6 julien  staff    192 10 Sep 16:52 totalspineseg
drwxr-xr-x   6 julien  staff    192 10 Sep 16:52 venv

correct instruction:

(venv) julien-macbook:~/code/totalspineseg $ python3 -m pip install -e .

nevermind-- i see that we were not supposed to go into the git cloned repos (which is usually common practice, so it's a bit confusing)-- i suggest to instruct to go into the repos and then install from there

Thank you!! this is actually confusing. We should make it more intuitive.
Will it be more intuitive if we change the instructions to:

cd totalspineseg
python3 -m pip install -e .
cd ..

@jcohenadad should we change it like this? cd .. for the later command mkdir data
We have to solve this issue before merge

@yw7
Copy link
Collaborator Author

yw7 commented Sep 10, 2024

image

is this really necessary? can't we have a CLI that just points to the image to run the inference on?

This is actually good point to discuss. The current implementation is not perfect.
We need it mainly for training, so tha training script will know where the data is and where the resources are. For infernce this used for constand dir to store the model pretrained weights.
For inference only the --data-dir or -d argument available instead to tell the code where the model weights are stored. But wouldn't it be more convinced to have a constant variable in the venv to save the path to the model weights?

@jcohenadad
Copy link
Member

This is actually good point to discuss. The current implementation is not perfect.
We need it mainly for training, so tha training script will know where the data is and where the resources are. For infernce this used for constand dir to store the model pretrained weights.
For inference only the --data-dir or -d argument available instead to tell the code where the model weights are stored. But wouldn't it be more convinced to have a constant variable in the venv to save the path to the model weights?

I wouldn't worry too much about these fine details, given that ultimately this method will be callable via SCT. But we should already think of an appropriate API to plug it inside SCT. Tagging @joshuacwnewton @mguaypaq so we can start thinking about it

@joshuacwnewton
Copy link
Member

joshuacwnewton commented Sep 11, 2024

Thank you for the ping. I'll take a look at this first thing tomorrow and begin thinking about ways that this project can integrate with SCT. I appreciate you keeping us in the loop! :)

EDIT: I've opened an issue (see below) for the topic of integrating TotalSpineSeg into SCT. So, feel free to proceed with this PR, and I'll keep an eye out from the SCT side of things to prepare for a potential future integration. :)

@NathanMolinier
Copy link
Collaborator

NathanMolinier commented Sep 13, 2024

Regarding this PR I am having troubles. It seems like localizer are not being used
Screenshot 2024-09-13 at 10 29 48

I ran this command:

totalspineseg raw_crop_reg/MTS out_reg_loc/ --loc out_loc/step2_output/MTS --suffix _MTS --loc-suffix _T2w

Edit: the problem was caused by the misalignment between the localizer and the image
--> We should improve the error message

@NathanMolinier
Copy link
Collaborator

NathanMolinier commented Sep 13, 2024

Sorry for the delay but here are some results that I created to validate the relevance of the PR:
Screenshot 2024-09-13 at 17 08 45

Each square corresponds to a different subject, from left to right inside the square:

  • The first image represents a T2w localizer on which I ran the totalspineseg
  • The second image corresponds to a T2star weighted image of the same patient on which I ran the totalspineseg
  • The third image corresponds to the same T2star weighted image of the same patient on which I ran the totalspineseg with the localizer information for the labeling
  • The fourth image is the cropped T2star weighted image on which I ran the model
  • The fifth image is the cropped T2star weighted image on which I ran the model with the help of the localizer
Screenshot 2024-09-13 at 17 13 28

I believe these images show that the method is working and is relevant to help the labeling.

@yw7 yw7 linked an issue Sep 24, 2024 that may be closed by this pull request
@NathanMolinier
Copy link
Collaborator

Last question regarding this PR, as raised by @jcohenadad, what would be the best practice regarding the installation of the package ?

Currently we ask the user to do this

git clone https://github.com/neuropoly/totalspineseg.git 
pip install -e totalspineseg 

But we were wondering if instead it would be more intuitive for the user to do this:

git clone https://github.com/neuropoly/totalspineseg.git
cd totalspineseg
pip install -e .
cd ..

Note: After this installation we are creating a data folder outside of the repository and initializing variables, that's why we need to do move back with cd ...

Do you have any recommandations on this @joshuacwnewton @mguaypaq ?

@yw7
Copy link
Collaborator Author

yw7 commented Sep 27, 2024

Last question regarding this PR, as raised by @jcohenadad, what would be the best practice regarding the installation of the package ?

Currently we ask the user to do this

git clone https://github.com/neuropoly/totalspineseg.git 
pip install -e totalspineseg 

But we were wondering if instead it would be more intuitive for the user to do this:

git clone https://github.com/neuropoly/totalspineseg.git
cd totalspineseg
pip install -e .
cd ..

Note: After this installation we are creating a data folder outside of the repository and initializing variables, that's why we need to do move back with cd ...

Do you have any recommandations on this @joshuacwnewton @mguaypaq ?

Can we consider merge this PR and open an issue for that as this is not related to something that changed in this PR?

@joshuacwnewton
Copy link
Member

Agree with @yw7 -- Please feel free to tag me on any new issues discussing installation/packaging. :)

@yw7
Copy link
Collaborator Author

yw7 commented Sep 27, 2024

Agree with @yw7 -- Please feel free to tag me on any new issues discussing installation/packaging. :)

Thanks!!
Issue opened #64

@NathanMolinier NathanMolinier merged commit 5059d64 into main Sep 27, 2024
@NathanMolinier NathanMolinier deleted the localizer-based-labeling branch September 27, 2024 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Input single file instead of directory typo: vertebrea --> vertebrae Implement localizer based labeling
5 participants