templates/4a_compensate_image_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ca8b60ef",
   "metadata": {},
   "source": [
    "# Image Compensation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "237267d0",
   "metadata": {
    "tags": []
   },
   "source": [
    "This notebook will guide you through the Rosetta algorithm, which is used to remove background and noise from image data prior to analysis. It is based on the compensation matrix approach that has been used for correcting flow-cytometry data. The Rosetta matrix contains rows for each of the sources of noise, and columns for each of the output channels. Each entry in the matrix represents the proportional contamination from a given noise channel to a given output channel.\n",
    "\n",
    "The majority of the entries in the matrix will never need to be modified. For example, the isotopic impurities present in the elements that are used for conjugation, the proportion of hydride and oxide contamination, and other intrinsic features of the instrument. However, some channels, in particular the \"Noodle\" channel, which we use to remove organic contamination, can be influenced by sample preparation and instrument configuration. Therefore, this notebook gives the user the opportunity to modify that coefficient.\n",
    "\n",
    "For example, we illustrate Pre and Post Rosetta processing on the CD11c channel.\n",
    "\n",
    "<table><tr>\n",
    "    <td> <img src=\"./img/CD11c_pre_rosetta_cropped.png\" style=\"width:100%\"/> </td>\n",
    "    <td> <img src=\"./img/CD11c_post_rosetta_cropped.png\" style=\"width:100%\"/> </td>\n",
    "</tr></table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83dbaa7f-a2bc-47f5-80bf-3f39509ea316",
   "metadata": {},
   "source": [
    "## This notebook consists of 3 steps:\n",
    "**1. Define directories and copy necessary files, which includes the random selection of FOVs from the provided run folders.**\n",
    "\n",
    "**2. Test Rosetta on this subset of FOVs to find good coefficients for the compensation matrix.**\n",
    "\n",
    "**3. Use the finalized matrix to process all of the data.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0b3f474f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('../')\n",
    "\n",
    "import os\n",
    "import shutil\n",
    "\n",
    "import skimage.io as io\n",
    "from toffy import rosetta\n",
    "from toffy.panel_utils import load_panel\n",
    "from toffy.image_stitching import get_max_img_size\n",
    "from tmi.io_utils import list_folders, list_files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95eb4672",
   "metadata": {},
   "source": [
    "## 1. Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1db8364",
   "metadata": {},
   "source": [
    "Below, you will set up the necessary structure for testing rosetta on all of your runs.\n",
    "- `cohort_name` is a descriptive name for the folder that will store the rosetta testing files\n",
    "- `run_names` is a list of all the runs you would like to retrieve FOV images from for testing\n",
    "- `panel_path` should point to a panel csv specifying the targets on your panel (see [panel format](https://github.com/angelolab/toffy#panel-format) for more information)\n",
    "\n",
    "A new directory based on the provided `cohort_name` above will be created within `C:\\\\Users\\\\Customer.ION\\\\Documents\\\\rosetta_testing`. This folder will contain all the files need for and produced in **Section 2** of the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e91ef0fd-77ae-4c93-a3ca-e27181368a57",
   "metadata": {},
   "outputs": [],
   "source": [
    "# run specifications\n",
    "cohort_name = '20220101_new_cohort'\n",
    "run_names = ['20220101_TMA1', '20220102_TMA2']\n",
    "panel_path = 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\panel_files\\\\my_cool_panel.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eea948ca-5983-4644-bfc1-418ad79a4ec6",
   "metadata": {},
   "source": [
    "By default, the `commercial_rosetta_matrix_v1.csv` from the `files` directory of toffy will be used for rosetta. If you would like to use a different matrix, specify the path below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21593608-e2d5-498a-8c76-1f8a135547c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# default rosetta matrix provided in toffy\n",
    "default_matrix_path = os.path.join('..', 'files', 'commercial_rosetta_matrix_v1.csv')\n",
    "\n",
    "rosetta_testing_dir = 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\rosetta_testing'\n",
    "extracted_imgs_dir = 'D:\\\\Extracted_Images'\n",
    "\n",
    "# read in toffy panel file\n",
    "panel = load_panel(panel_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4522356-44eb-4348-839b-08956bedf0a6",
   "metadata": {},
   "source": [
    "With the provided run names, we will randomly choose a few FOVs per run to rescale and then test rosetta on. The testing subset should be approximately **10-20 FOVs in total**; by default the number of FOVs per run is 5 (i.e. 2 runs with 5 FOVs each will produce a 10 FOV testing set). You can adjust the `fovs_per_run` variable below to create an appropriate testing subset. The rosetta matrix provided by the path above will also be copied into the new testing directory to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c49d5357",
   "metadata": {},
   "outputs": [],
   "source": [
    "# copy random fovs from each run\n",
    "rosetta.copy_image_files(cohort_name, run_names, rosetta_testing_dir, extracted_imgs_dir, fovs_per_run=5)\n",
    "\n",
    "# copy rosetta matrix\n",
    "shutil.copyfile(default_matrix_path, \n",
    "                os.path.join(rosetta_testing_dir, cohort_name, 'commercial_rosetta_matrix.csv'))\n",
    "\n",
    "# rescale images to allow direct comparison with rosetta\n",
    "img_out_dir = os.path.join(rosetta_testing_dir, cohort_name, 'extracted_images')\n",
    "rosetta.rescale_raw_imgs(img_out_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7239e333",
   "metadata": {},
   "source": [
    "## 2. Rosetta - Remove Signal Contamination\n",
    "We'll now process the images with rosetta to remove signal contamination at varying levels. **By default we'll be testing out coefficient multipliers in proportion to their value in the default matrix for the Noodle channel, since it is the main source of noise in most images.** For example, specifying multipliers of 0.5, 1, and 2 would test coefficients that are half the size, the same size, and twice the size of the Noodle coefficients in the default matrix, respectively. **This will give us a new set of compensated images, using different values in each compensation matrix.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "38596718-1c90-4fc1-9ecc-3390b20eefe2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pick the channel that you will be optimizing the coefficient for\n",
    "current_channel_name = 'Noodle'\n",
    "\n",
    "# set multipliers\n",
    "multipliers = [0.5, 1, 2]\n",
    "\n",
    "# pick an informative name\n",
    "folder_name = 'rosetta_test1'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cee0977a-2ab6-4253-962b-4887d24ebebc",
   "metadata": {},
   "source": [
    "Compensating the example images for 3 multipliers can take upwards of about 30 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b23917f7-92c7-4b12-9468-833bab4b142f",
   "metadata": {},
   "outputs": [],
   "source": [
    "rosetta_mat_path = os.path.join(rosetta_testing_dir, cohort_name, 'commercial_rosetta_matrix.csv')\n",
    "\n",
    "# create sub-folder to hold images and files from this set of multipliers\n",
    "folder_path = os.path.join(rosetta_testing_dir, cohort_name, folder_name)\n",
    "if os.path.exists(folder_path):\n",
    "    raise ValueError('This folder {} already exists, please' \n",
    "                     'pick a new name for each set of parameters'.format(folder_name))\n",
    "else:\n",
    "    os.makedirs(folder_path)\n",
    "\n",
    "# compensate the example fov images \n",
    "rosetta.generate_rosetta_test_imgs(rosetta_mat_path, img_out_dir, multipliers, folder_path, \n",
    "                                   panel, current_channel_name, output_channel_names=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e0c72cb",
   "metadata": {},
   "source": [
    "Now that we've generated the compensated data for the given multipliers, we'll generate stitched images to make comparing the different coefficients easier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "459da391-3e7a-4299-b44f-10bc5291af8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# stitch images together to enable easy visualization of outputs\n",
    "stitched_dir = os.path.join(folder_path, 'stitched_images')\n",
    "os.makedirs(stitched_dir)\n",
    "\n",
    "rosetta_dirs=[img_out_dir]\n",
    "for mult in multipliers:\n",
    "    rosetta_dirs.append(os.path.join(folder_path, f'compensated_data_{mult}'))\n",
    "\n",
    "img_size = get_max_img_size(img_out_dir)\n",
    "rosetta.create_tiled_comparison(input_dir_list=rosetta_dirs, output_dir=stitched_dir, max_img_size=img_size, \n",
    "                                channels=None)\n",
    "\n",
    "# add the source channel as first row to make evaluation easier\n",
    "output_dir = os.path.join(rosetta_testing_dir, cohort_name, folder_name + '-stitched_with_' + current_channel_name)\n",
    "os.makedirs(output_dir)\n",
    "rosetta.add_source_channel_to_tiled_image(raw_img_dir=img_out_dir, tiled_img_dir=stitched_dir,\n",
    "                                          output_dir=output_dir, source_channel=current_channel_name, max_img_size=img_size)\n",
    "\n",
    "# remove the intermediate compensated_data_{mult} and stitched_image dirs\n",
    "rosetta.clean_rosetta_test_dir(folder_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21d6a3fe-19a1-4df1-a952-6c4b2bb7dc07",
   "metadata": {},
   "source": [
    "### Evaluating the Images"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "963ee77e-d096-4e87-8fa5-d4615bd23f3d",
   "metadata": {},
   "source": [
    "There will now exist a folder named `folder_name-stitched_with_Noodle` (based on the folder name you provided above for this test) in your cohort testing directory. You can look through these stitched images to visualize what signal is being removed from the Noodle channel.\n",
    "\n",
    "These files will contain 5 rows of images: \n",
    "- row 1: the Noodle signal\n",
    "- row 2: the raw extracted image\n",
    "- row 3-5: images after applying the rosetta matrix with coefficients adjusted for the multipliers (i.e. [0.5, 1, 2])\n",
    "\n",
    "<center>\n",
    "    <img src=\"./img/CD4_stitched_with_Noodle.jpg\" style=\"width:50%\"> \n",
    "<center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af0802bd-a6e2-4393-8dff-18eeb1c8a62b",
   "metadata": {},
   "source": [
    "**If the images in either row 3, 4, or 5 are satisfactory, then you can save your compensation matrix to complete step 2.**\n",
    "\n",
    "Within the `folder_name` directory, you will find matrices files updated with the provided multipliers. If you're happy with one multiplier (e.g. 0.5), **find the corresponding matrix** `commercial_rosetta_matrix_mult_0.5.csv` **and rename it**. You can then run the cell below with your updated `final_matrix_name` and move on to the final section. \n",
    "    \n",
    "**A copy of your final rosetta matrix will be saved to `C:\\\\Users\\\\Customer.ION\\\\Documents\\\\rosetta_matrices`.**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "223f9ea4-0ba7-47a6-88c5-ab0fa55762ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# rename your rosetta matrix and put the path to final file here\n",
    "final_matrix_name = 'cohort_name_rosetta_matrix.csv'\n",
    "\n",
    "rosetta_path = os.path.join(rosetta_testing_dir, cohort_name, folder_name, final_matrix_name)\n",
    "\n",
    "# copy final rosetta matrix to matrix folder\n",
    "rosetta_matrix_dir = 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\rosetta_matrices'\n",
    "shutil.copyfile(rosetta_path, os.path.join(rosetta_matrix_dir, final_matrix_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fa61822-3984-4d83-adf6-4587028d6688",
   "metadata": {},
   "source": [
    "**Once you've finalized your matrix, please let us know [here](https://github.com/angelolab/toffy/issues/55).** This will help us better fine tune the matrix and improve rosetta for future users."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b2274b4",
   "metadata": {
    "tags": []
   },
   "source": [
    "### (Optional) Optimize Your Compensation Matrix\n",
    "However, if you would like to further adjust the amount of noise being removed, you can **re-run the code cells in Section 2** to optimize the compensation matrix for your data; you can try new multiplier values until you find one that is able to give you your desired images. When re-running the code, you will need to update the `multipliers` variable and a provide a new `folder_name`. The previously generated stitched images can help you determine whether the new multipliers need to be higher or lower.\n",
    "\n",
    "**After re-running rosetta, examine your new set of stitched images and check if you are happy with the images produced. Be sure to rename and save your chosen compensation matrix before proceeding to step 3.** "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0307342",
   "metadata": {},
   "source": [
    "## 3. Rosetta - Compensate Your Runs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd4f51e0-8bcc-4b5b-b2d3-6bc00140e0ca",
   "metadata": {
    "tags": []
   },
   "source": [
    "**Once you're satisfied that the Rosetta is working appropriately, you can use it to process your runs.** First select the runs you want to process, and define the relevant top-level folders.\n",
    "- `runs` is a list of all the runs you would like to process, by default uses the run list provided in Step 1 for testing\n",
    "- `final_matrix_name` is the name of the matrix you wish to use for compensation, stored in **`C:\\\\Users\\\\Customer.ION\\\\Documents\\\\rosetta_matrices`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc7b8a9a-3fa5-47ca-b845-e0e08b5ad190",
   "metadata": {},
   "outputs": [],
   "source": [
    "# by default uses the run list provided in Step 1 for testing, or provide your own list\n",
    "runs = run_names    # runs = []\n",
    "\n",
    "# provide the matrix file name\n",
    "final_matrix_name = 'cohort_name_rosetta_matrix.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70164970-6e95-4df8-95fb-54bf87ca7211",
   "metadata": {},
   "source": [
    "Everything necessary for and subsequently outputted from this section of the notebook is stored in the automatic directories established in `1_set_up_toffy.ipynb`. More information on the uses and locations of the directories in toffy can be found in the [README](https://github.com/angelolab/toffy#directory-structure)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c094e5a-cef8-4a6b-9e8b-2926ff178778",
   "metadata": {},
   "outputs": [],
   "source": [
    "# this folder will hold the post-rosetta images\n",
    "rosetta_image_dir = 'D:\\\\Rosetta_Compensated_Images'\n",
    "\n",
    "rosetta_matrix_dir = 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\rosetta_matrices'\n",
    "final_rosetta_path = os.path.join(rosetta_matrix_dir, final_matrix_name)\n",
    "\n",
    "extracted_imgs_dir = 'D:\\\\Extracted_Images'\n",
    "\n",
    "# if you would like to process all of the run folders in the image dir instead of just the runs tested, you can use the below line\n",
    "# runs = list_folders(extracted_imgs_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4df6224-9839-43f7-9122-62bca0668fbc",
   "metadata": {},
   "source": [
    "Now, you can compensate the data using rosetta. Depending on how many runs and FOVs you will be processing, this can take a while. Feel free to leave the notebook running overnight (do not to close jupyter lab or the terminal window), and also make sure you have enough storage space for the new images produced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13d28546-47db-4827-9283-1a7fc7eb8ec2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# perform rosetta on the provided runs\n",
    "for run in runs:\n",
    "    print(\"processing run {}\".format(run))\n",
    "    if not os.path.exists(os.path.join(rosetta_image_dir, run)):\n",
    "        os.makedirs(os.path.join(rosetta_image_dir, run))\n",
    "    rosetta.compensate_image_data(raw_data_dir=os.path.join(extracted_imgs_dir, run), \n",
    "                                  comp_data_dir=os.path.join(rosetta_image_dir, run), \n",
    "                                  comp_mat_path=final_rosetta_path, panel_info=panel, batch_size=1)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "5a55edcf046d85fe15e2be55257d5f0a71929bdfb61b433d935bb794f8010f3b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}