Time-Ahead-Basic-LSTM-Avg-Voxel-Entropy-Young-3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rN8VtTibW_kL"
   },
   "source": [
    "Set Colab to GPU Mode if you are training the models!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ESNkUUOEoS8j"
   },
   "source": [
    "### Mounting Google Drive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GWpiHXuGn6Oy",
    "outputId": "2ccc1d4e-1516-4e2e-e666-e28ef97cde69"
   },
   "outputs": [],
   "source": [
    "# from google.colab import drive\n",
    "# drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bpxEP-yeoWPX",
    "outputId": "a581e75e-fd8e-47d4-a5fa-db65a86e8412"
   },
   "outputs": [],
   "source": [
    "# cd 'drive/MyDrive/IIT J Summer Internship 2022/Code/Neural-Uncertainty-Representation/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Hq9mNRMMomT7"
   },
   "source": [
    "### Importing Necessary Libraries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "id": "SRXBkwcIogPe"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "from tensorflow.keras.layers import Dense, Dropout, LSTM, TimeDistributed\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_R2-Cetmxned"
   },
   "source": [
    "### Importing Data\n",
    "We import the data of a young subject's 3D-Tensor Data of dimensions (TR, Voxel, Number of Subjects) for the 5 different ROIs as follows:\n",
    "- **Prefrontal areas:** dlPFC, vlPFC, lOFC, dmPFC, mPFC, mOFC\n",
    "- **Default-mode areas:** mPFC, ACC, PCC, Precuneus, mOFC\n",
    "- **Sensory areas:** VC\n",
    "- **Multisensory area:** pSTS\n",
    "- **Some other areas:** TP, IPL, mCC, Put, PCG, Nac, INS\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "id": "XUJPFBg1898z"
   },
   "outputs": [],
   "source": [
    "# file_names = [name for name in os.listdir('/content/drive/MyDrive/IIT J Summer Internship 2022/Hitchcock/Data/YOUNG/Voxel_BOLD/Numpy') if name.endswith(\"npy\") and (\"(1)\" not in name)]\n",
    "file_names = [name for name in os.listdir('./Data/YOUNG/Voxel_BOLD/Numpy') if name.endswith(\"npy\") and (\"(1)\" not in name)]\n",
    "file_names = [name for name in file_names if (\"537\" not in name)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "id": "5lVnzf5N_Ufy"
   },
   "outputs": [],
   "source": [
    "ROI_names = [file_name.replace(\"data_\",\"\").replace(\".npy\",\"\") for file_name in file_names]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "id": "Kg22rkIGpfP2"
   },
   "outputs": [],
   "source": [
    "data_ori = list()\n",
    "for file_name in file_names:\n",
    "    # data_ori.append(np.load('/content/drive/MyDrive/IIT J Summer Internship 2022/Hitchcock/Data/YOUNG/Voxel_BOLD/Numpy/'+file_name))\n",
    "    data_ori.append(np.load('./Data/YOUNG/Voxel_BOLD/Numpy/'+file_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Kcaxzm8WDjgO"
   },
   "source": [
    "### Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K9rJ8CAb97K2"
   },
   "source": [
    "Now, here we prepare the data to be fed to the Neural Network Architechtures. Following on the previous developement of the LSTM model to predict BOLD, as a proxy to calculate **Uncertainty**, I use entropy at each time step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "im3Oqy2ueeEU"
   },
   "source": [
    "Right from the starting we decide some of the data preparation parameters, so that we can use them as we require."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "id": "X1DoZ2Fcec20"
   },
   "outputs": [],
   "source": [
    "TRAIN_BATCH_SIZE = 20\n",
    "TEST_BATCH_SIZE = 10\n",
    "LOOK_AHEADS = list(range(1, 2)) \n",
    "NUM_TEST_SUBS = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TYuu8ZM2-u9z"
   },
   "source": [
    "We will start a basic data preparation where we will average across all the voxels for a ROI and fit the model which predicts this average BOLD value. Later we will avoid doing an average since, all voxels in a ROI is not equally important and hence we will want to predict the BOLD values of several voxels simultaneously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "id": "pFvaFL69geft"
   },
   "outputs": [],
   "source": [
    "data = [np.mean(dat, axis=1, keepdims=True) for dat in data_ori]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mY3usMssn3T5"
   },
   "source": [
    "Let's Start the Data Preparation for lOFC later we will iterate the same procedure for the other ROIs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "71dv3Zkcgu3N",
    "outputId": "9dfa0cb6-dc7b-4404-cc7b-32268ad7201f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TP Average BOLD Shape: (189, 1, 111)\n"
     ]
    }
   ],
   "source": [
    "print(ROI_names[-1], \"Average BOLD Shape:\", data[-1].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jG7ofnvRczcN"
   },
   "source": [
    "Below, we see the average BOLD time series plot for all the ROIs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-wIOccLpE0d-"
   },
   "source": [
    "We have changed the data in the form (Number of Subjects, TR, Voxels) below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "id": "0pHK54KjDnah"
   },
   "outputs": [],
   "source": [
    "data = [np.transpose(dat, (2,0,1)) for dat in data]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HRXHJ1cwgrzp",
    "outputId": "bb9ef066-2eae-47cc-deeb-4743cda8a59e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TP Average BOLD Shape: (111, 189, 1)\n"
     ]
    }
   ],
   "source": [
    "print(ROI_names[-1], \"Average BOLD Shape:\", data[-1].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "id": "AseI4ZNeg26E"
   },
   "outputs": [],
   "source": [
    "# for i in range(len(data)):  \n",
    "#   plt.figure(figsize=(10, 8))\n",
    "#   plt.plot(data[i][0,:,0])\n",
    "#   plt.title(\"Average BOLD of 1st Subject for \"+ROI_names[i])\n",
    "#   plt.xlabel(\"TRs\")\n",
    "#   plt.ylabel(\"Average BOLD(over voxels)\")\n",
    "#   plt.savefig(\"./Plots/YOUNG/AVG-BOLD/AVG-BOLD-\"+ROI_names[i]+\".png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gg50q17pp5xQ"
   },
   "source": [
    "We split the dataset of 111 young subjects into 101 subjects in the training data and 10 subject in the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "id": "hrldyz8XnDlv"
   },
   "outputs": [],
   "source": [
    "train_data_unscaled = [dat[:-NUM_TEST_SUBS] for dat in data]\n",
    "test_data_unscaled = [dat[-NUM_TEST_SUBS:] for dat in data]  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MDV7H0sYt6jd"
   },
   "source": [
    "We scaled the BOLD values of each voxel in the train data between 0 and 1 for each voxel. We use the same MinMixScaler fit on the train data on the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "id": "-xkJQLj3tORj"
   },
   "outputs": [],
   "source": [
    "scalers = list()\n",
    "for i in range(len(ROI_names)):\n",
    "  scalers.append(MinMaxScaler(feature_range = (0, 1)))\n",
    "train_data = [sc.fit_transform(dat[:,:,0]) for sc, dat in zip(scalers, train_data_unscaled)] \n",
    "test_data = [sc.transform(dat[:,:,0]) for sc, dat in zip(scalers, test_data_unscaled)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "id": "Ail126iVuR6s"
   },
   "outputs": [],
   "source": [
    "train_data = [np.expand_dims(dat, axis = -1) for dat in train_data]\n",
    "test_data = [np.expand_dims(dat, axis = -1) for dat in test_data]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_cOAv_VrnbLU",
    "outputId": "ca52cae0-fbc2-4a11-ae29-af58d9fe53e0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TP Train Data Shape:  (101, 189, 1)\n",
      "TP Test Data Shape:  (10, 189, 1)\n"
     ]
    }
   ],
   "source": [
    "print(ROI_names[-1], \"Train Data Shape: \", train_data[-1].shape)\n",
    "print(ROI_names[-1], \"Test Data Shape: \", test_data[-1].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, for the purpose of binning which we will use for calculating entropy, each time point in BOLD signal, after standard scaling needs to be split up into regions based on the value being some standard deviation away from $0$. So, how to split the data is what we will decide here. I go with the split such that each bin contains equal number of values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "NUM_BINS = 10 # Per ROI how many bins should be there"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "def equalObs(x, nbin):\n",
    "  nlen = len(x)\n",
    "  return np.interp(np.linspace(0, nlen, nbin + 1), np.arange(nlen), np.sort(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "roi_bins = list()\n",
    "train_target = list()\n",
    "for roi_data in train_data:\n",
    "  _, bins, _ = plt.hist(np.squeeze(np.concatenate(roi_data)), equalObs(np.squeeze(np.concatenate(roi_data)), NUM_BINS))\n",
    "  plt.close()\n",
    "  roi_bins.append(bins)\n",
    "  train_target.append(np.sum(roi_data > np.tile(bins, roi_data.shape), axis=-1, keepdims=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(roi_bins[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we generate the binned plots to see how the results look like for the first subject."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(data)):  \n",
    "  plt.figure(figsize=(10, 8))\n",
    "  plt.plot(train_data[i][0,:,0], label='Actual')\n",
    "  plt.plot(roi_bins[i][train_target[i][0,:,0]], label='Binned')\n",
    "  plt.legend()\n",
    "  plt.title(\"Average BOLD of 1st Subject for \"+ROI_names[i]+\" with \"+str(NUM_BINS)+\" binned targets (Train Data Based Bins)\")\n",
    "  plt.xlabel(\"TRs\")\n",
    "  plt.ylabel(\"Average BOLD(over voxels)[Scaled between 0 and 1]\")\n",
    "  plt.savefig(\"./Plots/YOUNG/AVG-BOLD/AVG-BOLD-SCALED-TRAIN-\"+str(NUM_BINS)+\"-BINS-\"+ROI_names[i]+\".png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, from the bins generated from the training data, I will use them to bin the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_target = list()\n",
    "for roi_data, bins in zip(test_data, roi_bins):\n",
    "  test_target.append(np.sum(roi_data > np.tile(bins, roi_data.shape), axis=-1, keepdims=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0yjlAwWeiEru"
   },
   "source": [
    "We now build a data generator which would produce mini-batches during the training of the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "id": "Blko7LXpgtwA"
   },
   "outputs": [],
   "source": [
    "class KerasBatchGenerator(object):\n",
    "    def __init__(self, data, bins_data, num_TRs, batch_size, look_ahead=1):\n",
    "        self.data = data\n",
    "        self.bins_data = bins_data\n",
    "        self.num_TRs = num_TRs\n",
    "        self.batch_size = batch_size\n",
    "        self.current_idx = 0\n",
    "        self.look_ahead = look_ahead\n",
    "\n",
    "    def generate(self):\n",
    "        while True:\n",
    "            if self.current_idx + self.batch_size >= self.data.shape[0]:\n",
    "              self.current_idx = 0\n",
    "            x = self.data[self.current_idx:(self.current_idx+self.batch_size),:(-self.look_ahead),:]\n",
    "            y = self.bins_data[self.current_idx:(self.current_idx+self.batch_size),self.look_ahead:,:]\n",
    "            self.current_idx = self.current_idx + self.batch_size\n",
    "            yield x, y "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "id": "lZN56Eg3mrsc"
   },
   "outputs": [],
   "source": [
    "train_data_generator, test_data_generator = list(), list()\n",
    "for LOOK_AHEAD in LOOK_AHEADS:\n",
    "    train_data_generator.append([KerasBatchGenerator(dat, bin_dat, dat.shape[1], TRAIN_BATCH_SIZE, LOOK_AHEAD) for dat, bin_dat in zip(train_data, train_target)])\n",
    "    test_data_generator.append([KerasBatchGenerator(dat, bin_dat, dat.shape[1], TEST_BATCH_SIZE, LOOK_AHEAD) for dat, bin_dat in zip(test_data, test_target)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5UPT6iRxpJ7c",
    "outputId": "3512d1f5-6a04-4819-94b9-0229bc0cdef4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TP Train Batch X Shape for 1 Ahead:  (20, 188, 1)\n",
      "TP Train Batch y Shape for 1 Ahead:  (20, 188, 1)\n"
     ]
    }
   ],
   "source": [
    "batch_xs, batch_ys = list(), list()\n",
    "for LOOK_AHEAD in LOOK_AHEADS:\n",
    "    batch = next(train_data_generator[LOOK_AHEAD-1][-1].generate())\n",
    "    batch_xs.append(batch[0])\n",
    "    batch_ys.append(batch[1])\n",
    "    print(f\"{ROI_names[-1]} Train Batch X Shape for {LOOK_AHEAD} Ahead: \", batch_xs[LOOK_AHEAD-1].shape)\n",
    "    print(f\"{ROI_names[-1]} Train Batch y Shape for {LOOK_AHEAD} Ahead: \", batch_ys[LOOK_AHEAD-1].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Clavfro-pSEV",
    "outputId": "2f9380f5-1bf0-47f7-c4c5-4fe25999be8a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TP Test Batch X Shape for 1 Ahead:  (10, 188, 1)\n",
      "TP Test Batch y Shape for 1 Ahead:  (10, 188, 1)\n"
     ]
    }
   ],
   "source": [
    "batch_xs, batch_ys = list(), list()\n",
    "for LOOK_AHEAD in LOOK_AHEADS:\n",
    "    batch = next(test_data_generator[LOOK_AHEAD-1][-1].generate())\n",
    "    batch_xs.append(batch[0])\n",
    "    batch_ys.append(batch[1])\n",
    "    print(f\"{ROI_names[-1]} Test Batch X Shape for {LOOK_AHEAD} Ahead: \", batch_xs[LOOK_AHEAD-1].shape)\n",
    "    print(f\"{ROI_names[-1]} Test Batch y Shape for {LOOK_AHEAD} Ahead: \", batch_ys[LOOK_AHEAD-1].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "31NRewZjqoi1"
   },
   "source": [
    "### Building Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8Ry-jGmHqqx0"
   },
   "source": [
    "We will start off by building a LSTM Model:\n",
    "- `Number of LSTM Layers = 1`\n",
    "- `Number of Dense Layers = 2`\n",
    "- `hidden units = 32`\n",
    "- `dropout = 0.3`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "id": "AKDAJkiVgK2n"
   },
   "outputs": [],
   "source": [
    "HIDDEN_UNITS = 64\n",
    "DROPOUT_PROB = 0.3\n",
    "LSTM_LAYERS = 1\n",
    "DENSE_LAYERS = 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "id": "fqoXwHTTqqCR"
   },
   "outputs": [],
   "source": [
    "def avg_bold_model():\n",
    "  model = Sequential()\n",
    "  for _ in range(LSTM_LAYERS):\n",
    "    model.add(LSTM(HIDDEN_UNITS, return_sequences=True))\n",
    "  model.add(Dropout(0.3))\n",
    "  for _ in range(DENSE_LAYERS-1):\n",
    "    model.add(TimeDistributed(Dense(HIDDEN_UNITS, activation='relu')))\n",
    "  model.add(TimeDistributed(Dense(NUM_BINS + 1, activation='softmax')))\n",
    "  return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nYHgnyz-v5nA",
    "outputId": "9b71e39f-53a1-4ca7-a151-34fc6f874624"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***1 Look Ahead Model***\n",
      "Model: \"sequential_64\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " lstm_85 (LSTM)              (10, 188, 64)             16896     \n",
      "                                                                 \n",
      " dropout_64 (Dropout)        (10, 188, 64)             0         \n",
      "                                                                 \n",
      " time_distributed_171 (TimeD  (10, 188, 64)            4160      \n",
      " istributed)                                                     \n",
      "                                                                 \n",
      " time_distributed_172 (TimeD  (10, 188, 64)            4160      \n",
      " istributed)                                                     \n",
      "                                                                 \n",
      " time_distributed_173 (TimeD  (10, 188, 11)            715       \n",
      " istributed)                                                     \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 25,931\n",
      "Trainable params: 25,931\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "for LOOK_AHEAD in LOOK_AHEADS:\n",
    "    print(f\"***{LOOK_AHEAD} Look Ahead Model***\")\n",
    "    sample_model = avg_bold_model()\n",
    "    sample_model.build(input_shape=batch_xs[LOOK_AHEAD-1].shape)\n",
    "    sample_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "id": "fUZSBnNft2Dl"
   },
   "outputs": [],
   "source": [
    "models = [[avg_bold_model() for _ in ROI_names] for _ in LOOK_AHEADS]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3PRKFwUahAKG"
   },
   "source": [
    "#### Training Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LsHnfCSrhC0g"
   },
   "source": [
    "If you haven't already trained the models then uncomment and train your models which will be saved and will be loaded for inference later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "id": "kf-dFQmfzqTA"
   },
   "outputs": [],
   "source": [
    "for i in range(len(LOOK_AHEADS)):\n",
    "    for j in range(len(ROI_names)):\n",
    "        models[i][j].compile(optimizer = 'adam', loss = tf.keras.losses.SparseCategoricalCrossentropy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "id": "vcZsfhkz3lCH"
   },
   "outputs": [],
   "source": [
    "EPOCHS = 300\n",
    "checkpoint_filepaths = [[f\"./Models/YOUNG/LSTM/AVG-BINNED-BOLD/{name}/model_weights-best-val-loss-{LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop-{LOOK_AHEAD}Ahead-{name}\" for name in ROI_names] for LOOK_AHEAD in LOOK_AHEADS]\n",
    "model_checkpoint_callbacks = [[tf.keras.callbacks.ModelCheckpoint(\n",
    "    filepath=checkpoint_filepat,\n",
    "    save_weights_only=True,\n",
    "    monitor = \"val_loss\",\n",
    "    mode='min',\n",
    "    save_best_only=True) for checkpoint_filepat in checkpoint_filepath] for checkpoint_filepath in checkpoint_filepaths]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "id": "wECZbf2i_AH7"
   },
   "outputs": [],
   "source": [
    "def plot_train_test_loss(history, model_name, i):  \n",
    "  plt.figure(figsize=(10,10))\n",
    "  plt.plot(history.history['loss'])\n",
    "  plt.plot(history.history['val_loss'])\n",
    "  plt.title(model_name+' loss')\n",
    "  plt.ylabel('loss')\n",
    "  plt.xlabel('epochs')\n",
    "  plt.legend(['train', 'val'], loc='upper left')\n",
    "  plt.savefig(\"./Plots/YOUNG/LSTM/AVG-BINNED-BOLD/\"+ROI_names[i]+\"/\"+model_name+\"-loss.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AFb-kTLR21hl"
   },
   "outputs": [],
   "source": [
    "histories = list()\n",
    "with tf.device('/device:GPU:0'):\n",
    "    for LOOK_AHEAD in LOOK_AHEADS:\n",
    "        history = list()\n",
    "        for j in range(len(ROI_names)):\n",
    "            print(f'***{ROI_names[j]}***')\n",
    "            hist = models[LOOK_AHEAD-1][j].fit(\n",
    "                        train_data_generator[LOOK_AHEAD-1][j].generate(),\n",
    "                        steps_per_epoch=train_data[i].shape[0]//TRAIN_BATCH_SIZE,\n",
    "                        epochs=EPOCHS,\n",
    "                        validation_data=test_data_generator[LOOK_AHEAD-1][j].generate(),\n",
    "                        validation_steps=test_data[i].shape[0]//TEST_BATCH_SIZE,\n",
    "                        callbacks=[model_checkpoint_callbacks[LOOK_AHEAD-1][j]]\n",
    "                        )\n",
    "            models[LOOK_AHEAD-1][j].save_weights(f\"./Models/YOUNG/LSTM/AVG-BINNED-BOLD/{ROI_names[j]}/model_weights-{LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop-{LOOK_AHEAD}Ahead-{ROI_names[j]}\")\n",
    "            plot_train_test_loss(hist, f\"{LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop-{LOOK_AHEAD}Ahead-{ROI_names[j]}\", j)\n",
    "            history.append(hist)\n",
    "        histories.append(history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "id": "dKvn0145hViW"
   },
   "outputs": [],
   "source": [
    "for LOOK_AHEAD in LOOK_AHEADS:\n",
    "    for i in range(len(ROI_names)):\n",
    "        models[LOOK_AHEAD-1][i].load_weights(f\"./Models/YOUNG/LSTM/AVG-BINNED-BOLD/{ROI_names[i]}/model_weights-{LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop-{LOOK_AHEAD}Ahead-{ROI_names[i]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7mpQ5Rap6kQ0"
   },
   "outputs": [],
   "source": [
    "preds_raw = [[models[LOOK_AHEAD-1][i].predict(next(test_data_generator[LOOK_AHEAD-1][i].generate())[0]) for i in range(len(ROI_names))] for LOOK_AHEAD in LOOK_AHEADS]\n",
    "preds = [[np.argmax(preds_raw[LOOK_AHEAD-1][i], axis=-1) for i in range(len(ROI_names))] for LOOK_AHEAD in LOOK_AHEADS]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "id": "9a4QShbkj8lk"
   },
   "outputs": [],
   "source": [
    "a_TRs = np.array([56, 67, 73, 81, 134, 142, 155, 167, 174, 180])\n",
    "d_TRs = np.array([67, 72, 85, 89, 145, 154, 168, 174, 180, 188])\n",
    "and_TRs = np.union1d(a_TRs, d_TRs)\n",
    "\n",
    "def highlight_and(act, pred):\n",
    "  min_y_list = list()\n",
    "  max_y_list = list()\n",
    "  for i in range(5):\n",
    "    min_y_list.append(np.min(np.concatenate([act[a_TRs[2*i]:d_TRs[2*i+1]], pred[a_TRs[2*i]:d_TRs[2*i+1]]])))\n",
    "    max_y_list.append(np.max(np.concatenate([act[a_TRs[2*i]:d_TRs[2*i+1]], pred[a_TRs[2*i]:d_TRs[2*i+1]]])))\n",
    "    plt.fill_betweenx(y=[min_y_list[i], max_y_list[i]], x1=a_TRs[2*i], x2=d_TRs[2*i+1], color='gray', alpha=0.1)\n",
    "    plt.fill_betweenx(y=[min_y_list[i], max_y_list[i]], x1=a_TRs[2*i], x2=a_TRs[2*i+1], color='violet', alpha=0.4)\n",
    "    plt.fill_betweenx(y=[min_y_list[i], max_y_list[i]], x1=d_TRs[2*i], x2=d_TRs[2*i+1], color='greenyellow', alpha=0.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 188)"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds[0][-4].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "WBLOHLixAWnk"
   },
   "outputs": [],
   "source": [
    "for LOOK_AHEAD in LOOK_AHEADS:\n",
    "    for i in range(len(ROI_names)):\n",
    "      \n",
    "        actual_test_data_output = np.squeeze(test_target[i])[:,LOOK_AHEAD:] \n",
    "        pred_test_data_output = preds[LOOK_AHEAD-1][i]\n",
    "        plt.figure(figsize=(24, 18))\n",
    "\n",
    "        for j in range(NUM_TEST_SUBS-1):\n",
    "  \n",
    "            ax = plt.subplot(3, 3, j + 1)\n",
    "            plt.plot(actual_test_data_output[j,:])\n",
    "            plt.plot(pred_test_data_output[j,:])\n",
    "            highlight_and(actual_test_data_output[j,:], pred_test_data_output[j,:])\n",
    "            plt.xticks(np.concatenate([np.array([0]), and_TRs - LOOK_AHEAD]), np.concatenate([np.array([LOOK_AHEAD]), and_TRs]))\n",
    "            plt.ylabel(\"Average Binned BOLD\")\n",
    "            plt.xlabel(\"TRs\")\n",
    "            plt.title(f\"Avg Binned BOLD Preds of {LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop-{NUM_BINS}Bin-{LOOK_AHEAD}Ahead for {j+1}th Test Sub's {ROI_names[i]}\")\n",
    "            plt.legend(['actual', 'pred'])\n",
    "        plt.savefig(f\"./Plots/YOUNG/LSTM/AVG-BINNED-BOLD/{ROI_names[i]}/{LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop-{NUM_BINS}Bin-{LOOK_AHEAD}Ahead-Preds.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "J8hiJzxcKFuz",
    "outputId": "deaa6d44-de0d-463a-c437-9324739e5f64"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***Each Test Subject's Correlation Coefficient between 1 ahead predicted and actual average binned BOLD values for ROIs***\n",
      "\n",
      "dlPFC for 1th sub: 0.02\n",
      "dlPFC for 2th sub: 0.17\n",
      "dlPFC for 3th sub: -0.01\n",
      "dlPFC for 4th sub: 0.25\n",
      "dlPFC for 5th sub: 0.04\n",
      "dlPFC for 6th sub: 0.15\n",
      "dlPFC for 7th sub: -0.04\n",
      "dlPFC for 8th sub: -0.01\n",
      "dlPFC for 9th sub: 0.18\n",
      "dlPFC for 10th sub: -0.14\n",
      "INS for 1th sub: 0.37\n",
      "INS for 2th sub: 0.57\n",
      "INS for 3th sub: 0.21\n",
      "INS for 4th sub: 0.13\n",
      "INS for 5th sub: -0.08\n",
      "INS for 6th sub: 0.05\n",
      "INS for 7th sub: nan\n",
      "INS for 8th sub: 0.02\n",
      "INS for 9th sub: nan\n",
      "INS for 10th sub: 0.38\n",
      "VC for 1th sub: -0.01\n",
      "VC for 2th sub: 0.01\n",
      "VC for 3th sub: 0.57\n",
      "VC for 4th sub: -0.01\n",
      "VC for 5th sub: 0.52\n",
      "VC for 6th sub: -0.03\n",
      "VC for 7th sub: nan\n",
      "VC for 8th sub: 0.07\n",
      "VC for 9th sub: 0.29\n",
      "VC for 10th sub: 0.29\n",
      "Cau for 1th sub: 0.14\n",
      "Cau for 2th sub: -0.02\n",
      "Cau for 3th sub: 0.17\n",
      "Cau for 4th sub: nan\n",
      "Cau for 5th sub: nan\n",
      "Cau for 6th sub: -0.01\n",
      "Cau for 7th sub: -0.01\n",
      "Cau for 8th sub: 0.01\n",
      "Cau for 9th sub: 0.5\n",
      "Cau for 10th sub: 0.45\n",
      "PCC for 1th sub: 0.22\n",
      "PCC for 2th sub: -0.03\n",
      "PCC for 3th sub: 0.03\n",
      "PCC for 4th sub: 0.02\n",
      "PCC for 5th sub: nan\n",
      "PCC for 6th sub: 0.26\n",
      "PCC for 7th sub: -0.02\n",
      "PCC for 8th sub: 0.01\n",
      "PCC for 9th sub: 0.14\n",
      "PCC for 10th sub: nan\n",
      "PCG for 1th sub: nan\n",
      "PCG for 2th sub: 0.04\n",
      "PCG for 3th sub: 0.02\n",
      "PCG for 4th sub: nan\n",
      "PCG for 5th sub: nan\n",
      "PCG for 6th sub: -0.01\n",
      "PCG for 7th sub: -0.02\n",
      "PCG for 8th sub: nan\n",
      "PCG for 9th sub: 0.16\n",
      "PCG for 10th sub: nan\n",
      "mCC for 1th sub: -0.08\n",
      "mCC for 2th sub: 0.03\n",
      "mCC for 3th sub: 0.04\n",
      "mCC for 4th sub: nan\n",
      "mCC for 5th sub: nan\n",
      "mCC for 6th sub: 0.7\n",
      "mCC for 7th sub: -0.14\n",
      "mCC for 8th sub: 0.06\n",
      "mCC for 9th sub: 0.01\n",
      "mCC for 10th sub: nan\n",
      "mOFC for 1th sub: -0.06\n",
      "mOFC for 2th sub: nan\n",
      "mOFC for 3th sub: 0.2\n",
      "mOFC for 4th sub: 0.14\n",
      "mOFC for 5th sub: nan\n",
      "mOFC for 6th sub: 0.63\n",
      "mOFC for 7th sub: 0.02\n",
      "mOFC for 8th sub: 0.81\n",
      "mOFC for 9th sub: 0.75\n",
      "mOFC for 10th sub: 0.17\n",
      "Prec for 1th sub: -0.02\n",
      "Prec for 2th sub: 0.14\n",
      "Prec for 3th sub: 0.11\n",
      "Prec for 4th sub: 0.37\n",
      "Prec for 5th sub: -0.05\n",
      "Prec for 6th sub: nan\n",
      "Prec for 7th sub: 0.26\n",
      "Prec for 8th sub: 0.36\n",
      "Prec for 9th sub: -0.0\n",
      "Prec for 10th sub: 0.1\n",
      "mPFC for 1th sub: 0.02\n",
      "mPFC for 2th sub: nan\n",
      "mPFC for 3th sub: -0.0\n",
      "mPFC for 4th sub: 0.68\n",
      "mPFC for 5th sub: 0.2\n",
      "mPFC for 6th sub: 0.37\n",
      "mPFC for 7th sub: -0.02\n",
      "mPFC for 8th sub: 0.41\n",
      "mPFC for 9th sub: 0.29\n",
      "mPFC for 10th sub: -0.0\n",
      "pSTS for 1th sub: 0.03\n",
      "pSTS for 2th sub: 0.2\n",
      "pSTS for 3th sub: 0.18\n",
      "pSTS for 4th sub: 0.12\n",
      "pSTS for 5th sub: 0.24\n",
      "pSTS for 6th sub: 0.03\n",
      "pSTS for 7th sub: nan\n",
      "pSTS for 8th sub: 0.18\n",
      "pSTS for 9th sub: 0.37\n",
      "pSTS for 10th sub: 0.01\n",
      "vlPFC for 1th sub: 0.32\n",
      "vlPFC for 2th sub: 0.03\n",
      "vlPFC for 3th sub: 0.08\n",
      "vlPFC for 4th sub: nan\n",
      "vlPFC for 5th sub: 0.66\n",
      "vlPFC for 6th sub: 0.42\n",
      "vlPFC for 7th sub: nan\n",
      "vlPFC for 8th sub: 0.06\n",
      "vlPFC for 9th sub: 0.36\n",
      "vlPFC for 10th sub: -0.02\n",
      "dmPFC for 1th sub: 0.03\n",
      "dmPFC for 2th sub: 0.32\n",
      "dmPFC for 3th sub: nan\n",
      "dmPFC for 4th sub: 0.17\n",
      "dmPFC for 5th sub: -0.03\n",
      "dmPFC for 6th sub: 0.07\n",
      "dmPFC for 7th sub: nan\n",
      "dmPFC for 8th sub: 0.16\n",
      "dmPFC for 9th sub: 0.04\n",
      "dmPFC for 10th sub: -0.02\n",
      "Nac for 1th sub: 0.09\n",
      "Nac for 2th sub: 0.03\n",
      "Nac for 3th sub: 0.08\n",
      "Nac for 4th sub: 0.19\n",
      "Nac for 5th sub: nan\n",
      "Nac for 6th sub: 0.01\n",
      "Nac for 7th sub: nan\n",
      "Nac for 8th sub: nan\n",
      "Nac for 9th sub: 0.68\n",
      "Nac for 10th sub: 0.22\n",
      "IPL for 1th sub: nan\n",
      "IPL for 2th sub: -0.05\n",
      "IPL for 3th sub: 0.03\n",
      "IPL for 4th sub: -0.03\n",
      "IPL for 5th sub: 0.39\n",
      "IPL for 6th sub: 0.05\n",
      "IPL for 7th sub: nan\n",
      "IPL for 8th sub: 0.0\n",
      "IPL for 9th sub: 0.47\n",
      "IPL for 10th sub: 0.01\n",
      "ACC for 1th sub: 0.36\n",
      "ACC for 2th sub: 0.32\n",
      "ACC for 3th sub: 0.14\n",
      "ACC for 4th sub: 0.16\n",
      "ACC for 5th sub: 0.11\n",
      "ACC for 6th sub: 0.27\n",
      "ACC for 7th sub: 0.39\n",
      "ACC for 8th sub: 0.2\n",
      "ACC for 9th sub: 0.4\n",
      "ACC for 10th sub: 0.46\n",
      "lOFC for 1th sub: 0.54\n",
      "lOFC for 2th sub: 0.16\n",
      "lOFC for 3th sub: 0.74\n",
      "lOFC for 4th sub: nan\n",
      "lOFC for 5th sub: 0.66\n",
      "lOFC for 6th sub: nan\n",
      "lOFC for 7th sub: 0.52\n",
      "lOFC for 8th sub: 0.61\n",
      "lOFC for 9th sub: 0.82\n",
      "lOFC for 10th sub: nan\n",
      "Put for 1th sub: 0.07\n",
      "Put for 2th sub: 0.38\n",
      "Put for 3th sub: 0.36\n",
      "Put for 4th sub: 0.04\n",
      "Put for 5th sub: nan\n",
      "Put for 6th sub: 0.42\n",
      "Put for 7th sub: 0.01\n",
      "Put for 8th sub: 0.08\n",
      "Put for 9th sub: 0.08\n",
      "Put for 10th sub: 0.04\n",
      "AMY for 1th sub: 0.27\n",
      "AMY for 2th sub: -0.01\n",
      "AMY for 3th sub: -0.04\n",
      "AMY for 4th sub: 0.1\n",
      "AMY for 5th sub: nan\n",
      "AMY for 6th sub: 0.18\n",
      "AMY for 7th sub: 0.33\n",
      "AMY for 8th sub: 0.19\n",
      "AMY for 9th sub: 0.41\n",
      "AMY for 10th sub: 0.25\n",
      "TP for 1th sub: 0.01\n",
      "TP for 2th sub: 0.81\n",
      "TP for 3th sub: 0.89\n",
      "TP for 4th sub: 0.25\n",
      "TP for 5th sub: 0.66\n",
      "TP for 6th sub: 0.56\n",
      "TP for 7th sub: nan\n",
      "TP for 8th sub: nan\n",
      "TP for 9th sub: 0.15\n",
      "TP for 10th sub: -0.02\n",
      "\n",
      "***Average across test subjects Correlation Coefficient between 1 ahead predicted and actual average binned BOLD values for ROIs***\n",
      "\n",
      "dlPFC: 0.06\n",
      "INS: 0.16\n",
      "VC: 0.17\n",
      "Cau: 0.12\n",
      "PCC: 0.06\n",
      "PCG: 0.02\n",
      "mCC: 0.06\n",
      "mOFC: 0.26\n",
      "Prec: 0.13\n",
      "mPFC: 0.19\n",
      "pSTS: 0.14\n",
      "vlPFC: 0.19\n",
      "dmPFC: 0.07\n",
      "Nac: 0.13\n",
      "IPL: 0.09\n",
      "ACC: 0.28\n",
      "lOFC: 0.4\n",
      "Put: 0.15\n",
      "AMY: 0.17\n",
      "TP: 0.33\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/rishideychowdhury/Desktop/Neural-Uncertainty-Representation/env/lib/python3.8/site-packages/numpy/lib/function_base.py:2829: RuntimeWarning: invalid value encountered in true_divide\n",
      "  c /= stddev[:, None]\n",
      "/Users/rishideychowdhury/Desktop/Neural-Uncertainty-Representation/env/lib/python3.8/site-packages/numpy/lib/function_base.py:2830: RuntimeWarning: invalid value encountered in true_divide\n",
      "  c /= stddev[None, :]\n"
     ]
    }
   ],
   "source": [
    "TR_ROI_corr_subs, AVG_TR_ROI_corr_subs = list(), list()\n",
    "for LOOK_AHEAD in LOOK_AHEADS:\n",
    "    print(f\"***Each Test Subject's Correlation Coefficient between {LOOK_AHEAD} ahead predicted and actual average binned BOLD values for ROIs***\")\n",
    "    print()\n",
    "    avg_r = list()\n",
    "    ROI_corr_subs = list()\n",
    "    for i in range(len(ROI_names)):\n",
    "        actual_test_data_output = np.squeeze(test_target[i])[:,LOOK_AHEAD:] \n",
    "        pred_test_data_output = preds[LOOK_AHEAD-1][i]\n",
    "        corr_subs = list()\n",
    "        for j in range(NUM_TEST_SUBS):\n",
    "            corr_subs.append(np.round(np.corrcoef(actual_test_data_output[j,:], pred_test_data_output[j,:])[0,1], 2))\n",
    "            print(f\"{ROI_names[i]} for {j+1}th sub: {corr_subs[j]}\")\n",
    "        ROI_corr_subs.append(corr_subs)\n",
    "    TR_ROI_corr_subs.append(ROI_corr_subs)\n",
    "    print()\n",
    "\n",
    "    print(f\"***Average across test subjects Correlation Coefficient between {LOOK_AHEAD} ahead predicted and actual average binned BOLD values for ROIs***\")\n",
    "    print()\n",
    "    ROI_corr_subs = list()\n",
    "    for i in range(len(ROI_names)):\n",
    "        actual_test_data_output = np.squeeze(test_target[i])[:,LOOK_AHEAD:] \n",
    "        pred_test_data_output = preds[LOOK_AHEAD-1][i]\n",
    "        sum_r = 0\n",
    "        for j in range(NUM_TEST_SUBS):\n",
    "            correl = np.corrcoef(actual_test_data_output[j,:], pred_test_data_output[j,:])[0,1]\n",
    "            if not np.isnan(correl):\n",
    "              sum_r += correl\n",
    "        ROI_corr_subs.append(np.round(sum_r / NUM_TEST_SUBS, 2))\n",
    "        print(f\"{ROI_names[i]}: {ROI_corr_subs[i]}\")\n",
    "    AVG_TR_ROI_corr_subs.append(ROI_corr_subs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(20, 1, 10) (20, 1)\n"
     ]
    }
   ],
   "source": [
    "TR_ROI_corr_subs = np.transpose(np.array(TR_ROI_corr_subs), (1,0,2))\n",
    "AVG_TR_ROI_corr_subs = np.array(AVG_TR_ROI_corr_subs).T\n",
    "print(TR_ROI_corr_subs.shape, AVG_TR_ROI_corr_subs.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for i in range(len(ROI_names)):\n",
    "#     plt.figure(figsize=(24, 18))\n",
    "\n",
    "#     for j in range(NUM_TEST_SUBS-1):\n",
    "\n",
    "#         ax = plt.subplot(3, 3, j + 1)\n",
    "#         plt.plot(TR_ROI_corr_subs[i, :, j])\n",
    "#         plt.xticks(np.arange(1, 16))\n",
    "#         plt.ylabel(\"Correlation\")\n",
    "#         plt.xlabel(\"Ahead TRs\")\n",
    "#         plt.title(f\"Avg BOLD Corr of {LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop for {j+1}th Test Sub's {ROI_names[i]}\")\n",
    "#     plt.savefig(f\"./Plots/YOUNG/LSTM/AVG-BOLD-TEST-ENLARGED/{ROI_names[i]}/{LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop-Upto-15Ahead-Corrs.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for i in range(len(ROI_names)):\n",
    "#     plt.figure(figsize=(10, 10))\n",
    "#     plt.plot(AVG_TR_ROI_corr_subs[i])\n",
    "#     plt.xticks(np.arange(1, 16))\n",
    "#     plt.ylabel(\"Correlation\")\n",
    "#     plt.xlabel(\"Ahead TRs\")\n",
    "#     plt.title(f\"Avg BOLD Corr of {LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop Across Test Subs\")\n",
    "#     plt.savefig(f\"./Plots/YOUNG/LSTM/AVG-BOLD-TEST-ENLARGED/{ROI_names[i]}/{LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop-Upto-15Ahead-Corrs-Avg-Across-Subs.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "f8gPew9YMI2B",
    "outputId": "35e82a11-86f5-47ea-918c-cebfe07c90d6"
   },
   "outputs": [],
   "source": [
    "# for LOOK_AHEAD in LOOK_AHEADS:\n",
    "#     print(f\"***Model's Final MSE(Train) Loss between {LOOK_AHEAD} ahead predicted and actual average BOLD values for ROIs for the Train Subjects***\")\n",
    "#     print()\n",
    "#     for i in range(len(ROI_names)):\n",
    "#         print(ROI_names[i]+\":\", np.round(histories[LOOK_AHEAD-1][i].history['loss'][-1], 5))\n",
    "#     print()\n",
    "#     print(f\"***Model's Final MSE(Validation) Loss between {LOOK_AHEAD} ahead predicted and actual average BOLD values for ROIs for the Test Subjects***\")\n",
    "#     print()\n",
    "#     for i in range(len(ROI_names)):\n",
    "#         print(ROI_names[i]+\":\", np.round(histories[LOOK_AHEAD-1][i].history['val_loss'][-1], 5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***Model's Final CXT(Validation) Loss between 1 ahead predicted and actual avg voxel binned BOLD values for ROIs for the Test Subjects***\n",
      "\n",
      "dlPFC for Each Validation Subject: [0.7626592  0.71646166 0.07031546 0.6541307  0.68146205 0.78286266\n",
      " 0.6010697  0.07001999 0.9580424  0.78703624]\n",
      "dlPFC : 0.608406\n",
      "INS for Each Validation Subject: [0.6831421  0.6182727  0.8830615  0.57596606 0.69537693 0.738047\n",
      " 0.09018223 0.20908889 0.7801981  0.267569  ]\n",
      "INS : 0.55409044\n",
      "VC for Each Validation Subject: [0.3277012  0.33676615 0.25404933 0.77223873 0.6409105  0.26991415\n",
      " 0.67629606 0.7642324  1.1591434  0.69591844]\n",
      "VC : 0.58971703\n",
      "Cau for Each Validation Subject: [0.7240464  0.3151796  0.5765524  0.04965448 0.09686583 0.07977322\n",
      " 0.18741074 0.01826539 1.0487945  0.3747378 ]\n",
      "Cau : 0.34712803\n",
      "PCC for Each Validation Subject: [0.4800789  0.4483758  0.9400967  0.69725734 0.05272963 0.76334834\n",
      " 0.7616476  0.772508   0.92437565 0.59135926]\n",
      "PCC : 0.6431777\n",
      "PCG for Each Validation Subject: [0.29443076 0.6146033  0.6025532  0.1716902  0.20156859 0.6313512\n",
      " 0.78209525 0.3654371  0.8175759  0.12775055]\n",
      "PCG : 0.46090555\n",
      "mCC for Each Validation Subject: [0.60588545 0.6382201  0.64856803 0.7130885  0.0343796  0.47407857\n",
      " 0.6846664  0.8428113  0.2695297  0.08938271]\n",
      "mCC : 0.50006104\n",
      "mOFC for Each Validation Subject: [0.4568541  0.35390997 2.56433    0.772529   0.02691469 0.5547708\n",
      " 3.772768   0.3676641  0.85700613 0.4446981 ]\n",
      "mOFC : 1.0171446\n",
      "Prec for Each Validation Subject: [0.79234654 0.7523058  0.99566454 0.68642634 0.4265515  0.07666204\n",
      " 0.8611489  0.5499486  0.5833159  0.7792401 ]\n",
      "Prec : 0.650361\n",
      "mPFC for Each Validation Subject: [0.4142163  0.43311498 2.5343003  0.6762036  0.7062279  0.8728429\n",
      " 0.8621093  0.22537306 0.8934131  0.15954429]\n",
      "mPFC : 0.77773464\n",
      "pSTS for Each Validation Subject: [0.77974474 0.5442419  0.7768732  0.67342174 0.7699775  0.5330413\n",
      " 0.9856254  0.6144163  0.8415743  0.18881455]\n",
      "pSTS : 0.6707731\n",
      "vlPFC for Each Validation Subject: [0.6667291  0.43319705 0.67941207 0.5208288  0.4222125  0.6452696\n",
      " 0.5515179  0.7961354  0.8558092  0.3307473 ]\n",
      "vlPFC : 0.5901859\n",
      "dmPFC for Each Validation Subject: [0.76034606 0.52329195 0.05298853 0.80057496 0.5047637  0.525844\n",
      " 0.4819908  0.7529061  0.5520437  0.17397475]\n",
      "dmPFC : 0.5128725\n",
      "Nac for Each Validation Subject: [0.43624067 0.83032435 0.77630544 0.546116   0.03888154 0.69854057\n",
      " 0.07375241 0.03274242 0.813166   0.51646847]\n",
      "Nac : 0.4762538\n",
      "IPL for Each Validation Subject: [0.6345543  0.3651255  0.62585855 0.9751629  1.0247667  0.68180066\n",
      " 0.81405526 0.17271034 0.89254093 0.02195853]\n",
      "IPL : 0.6208533\n",
      "ACC for Each Validation Subject: [0.6782301  0.6261217  0.68848413 0.41402647 0.66156596 0.5061223\n",
      " 0.4140802  0.3737548  0.7693885  0.71445626]\n",
      "ACC : 0.58462304\n",
      "lOFC for Each Validation Subject: [0.786999   0.72617316 0.14254552 0.03122003 0.73921627 0.05218704\n",
      " 0.34836692 0.83543015 0.94543946 0.02860362]\n",
      "lOFC : 0.4636181\n",
      "Put for Each Validation Subject: [0.7567875  0.63020474 0.72138065 0.58241    0.17087474 0.5668493\n",
      " 0.7707731  1.0776582  0.72916114 0.63190126]\n",
      "Put : 0.6638001\n",
      "AMY for Each Validation Subject: [0.6847967  0.8694079  0.1996358  0.58123446 0.53766906 0.9336304\n",
      " 0.60496235 0.7061889  0.62690884 0.7327678 ]\n",
      "AMY : 0.6477202\n",
      "TP for Each Validation Subject: [0.56358016 0.6308717  0.6172358  0.2919455  0.49062112 0.7442878\n",
      " 0.03919428 0.07155009 0.6962365  0.26302105]\n",
      "TP : 0.44085437\n"
     ]
    }
   ],
   "source": [
    "CXT_subs, AVG_CXT_subs = list(), list()\n",
    "for LOOK_AHEAD in LOOK_AHEADS:\n",
    "    print(f\"***Model's Final CXT(Validation) Loss between {LOOK_AHEAD} ahead predicted and actual avg voxel binned BOLD values for ROIs for the Test Subjects***\")\n",
    "    print()\n",
    "    ROI_CXT_subs, ROI_AVG_CXT_subs = list(), list()\n",
    "    for i in range(len(ROI_names)):\n",
    "        actual_test_data_output = np.squeeze(test_target[i])[:,LOOK_AHEAD:] \n",
    "        pred_test_data_output = preds_raw[LOOK_AHEAD-1][i]\n",
    "        ROI_CXT_subs.append(np.mean(tf.keras.losses.sparse_categorical_crossentropy(actual_test_data_output, pred_test_data_output).numpy(), axis=-1))\n",
    "        print(ROI_names[i]+\" for Each Validation Subject:\", ROI_CXT_subs[i])\n",
    "        ROI_AVG_CXT_subs.append(np.mean(ROI_CXT_subs[i]))\n",
    "        print(ROI_names[i]+\" :\", ROI_AVG_CXT_subs[i])\n",
    "    CXT_subs.append(ROI_CXT_subs)\n",
    "    AVG_CXT_subs.append(ROI_AVG_CXT_subs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(20, 1, 10) (20, 1)\n"
     ]
    }
   ],
   "source": [
    "CXT_subs = np.transpose(np.array(CXT_subs), (1, 0, 2))\n",
    "AVG_CXT_subs = np.array(AVG_CXT_subs).T\n",
    "print(CXT_subs.shape, AVG_CXT_subs.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for i in range(len(ROI_names)):\n",
    "#     plt.figure(figsize=(24, 18))\n",
    "\n",
    "#     for j in range(NUM_TEST_SUBS-1):\n",
    "\n",
    "#         ax = plt.subplot(3, 3, j + 1)\n",
    "#         plt.plot(MSE_subs[i, :, j])\n",
    "#         plt.xticks(np.arange(1, 16))\n",
    "#         plt.ylabel(\"MSE Loss\")\n",
    "#         plt.xlabel(\"Ahead TRs\")\n",
    "#         plt.title(f\"Avg BOLD MSE Losses of {LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop for {j+1}th Test Sub's {ROI_names[i]}\")\n",
    "#     plt.savefig(f\"./Plots/YOUNG/LSTM/AVG-BOLD-TEST-ENLARGED/{ROI_names[i]}/{LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop-Upto-15Ahead-Losses.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for i in range(len(ROI_names)):\n",
    "#     plt.figure(figsize=(10, 10))\n",
    "#     plt.plot(AVG_MSE_subs[i])\n",
    "#     plt.xticks(np.arange(1, 16))\n",
    "#     plt.ylabel(\"MSE Loss\")\n",
    "#     plt.xlabel(\"Ahead TRs\")\n",
    "#     plt.title(f\"Avg BOLD MSE Losses of {LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop Across Test Subs\")\n",
    "#     plt.savefig(f\"./Plots/YOUNG/LSTM/AVG-BOLD-TEST-ENLARGED/{ROI_names[i]}/{LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop-Upto-15Ahead-Losses-Avg-Across-Subs.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for LOOK_AHEAD in LOOK_AHEADS:\n",
    "  for i in range(len(ROI_names)):\n",
    "    entropy = -1 * np.sum(preds_raw[LOOK_AHEAD-1][i]*np.log(preds_raw[LOOK_AHEAD-1][i]), axis=-1)\n",
    "    plt.figure(figsize=(24, 18))\n",
    "    for j in range(NUM_TEST_SUBS-1):\n",
    "      \n",
    "      ax = plt.subplot(3, 3, j + 1)\n",
    "      plt.plot(entropy[j,:])\n",
    "      highlight_and(entropy[j,:], entropy[j,:])\n",
    "      plt.xticks(np.concatenate([np.array([0]), and_TRs - LOOK_AHEAD]), np.concatenate([np.array([LOOK_AHEAD]), and_TRs]))\n",
    "      plt.ylabel(\"Entropy of preds avg binned BOLD\")\n",
    "      plt.xlabel(\"TRs\")\n",
    "      plt.title(f\"Entropy Avg BOLD of {LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop-{NUM_BINS}Bin-{LOOK_AHEAD}Ahead for {j+1}th Test Sub's {ROI_names[i]}\")\n",
    "    plt.savefig(f\"./Plots/YOUNG/LSTM/AVG-BINNED-BOLD/{ROI_names[i]}/{LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop-{NUM_BINS}Bin-{LOOK_AHEAD}Ahead-Preds-Entropy.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for LOOK_AHEAD in LOOK_AHEADS:\n",
    "  entropy = [-1 * np.sum(preds_raw[LOOK_AHEAD-1][i]*np.log(preds_raw[LOOK_AHEAD-1][i]), axis=-1) for i in range(len(ROI_names))]\n",
    "  entropy_avg = [np.mean(entropy_ind, axis=0) for entropy_ind in entropy]\n",
    "  for i in range(len(ROI_names)):\n",
    "    plt.figure(figsize=(10, 10))\n",
    "    plt.plot(entropy_avg[i])\n",
    "    highlight_and(entropy_avg[i], entropy_avg[i])\n",
    "    plt.xticks(np.concatenate([np.array([0]), and_TRs - LOOK_AHEAD]), np.concatenate([np.array([LOOK_AHEAD]), and_TRs]))\n",
    "    plt.ylabel(\"Entropy of preds avg binned BOLD\")\n",
    "    plt.xlabel(\"TRs\")\n",
    "    plt.title(f\"Entropy Avg BOLD of {LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop-{NUM_BINS}Bin-{LOOK_AHEAD}Ahead for All Test Subs Averaged {ROI_names[i]}\")\n",
    "    plt.savefig(f\"./Plots/YOUNG/LSTM/AVG-BINNED-BOLD/{ROI_names[i]}/{LSTM_LAYERS}L-{HIDDEN_UNITS}H-{DENSE_LAYERS}D-{int(DROPOUT_PROB*10)}Drop-{NUM_BINS}Bin-{LOOK_AHEAD}Ahead-Preds-All-Averaged-Entropy.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "authorship_tag": "ABX9TyO379dB7oIxYoBIQaqTB5jX",
   "collapsed_sections": [],
   "name": "Neural-Prediction-Uncertainty-Young-All-TestSet-Enlarged-LSTM-Layers-Increased-Hidden-Units-Increased-Dense-Layers-Increased.ipynb",
   "provenance": []
  },
  "gpuClass": "standard",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}