DLL52_Basics_NLP_Radiology_Danny (1).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Aa9xRyBkjaPF"
      },
      "source": [
        "# DLL 52 Basics of NLP in Radiology - RSNA 2021\n",
        "\n",
        "This session will feature a brief lecture introducing the basics of natural language processing (NLP) and highlight potential uses for it in radiology. This accompanying notebook will provide hands-on opportunity to explore some of the basic tools of NLP available in Python. \n",
        "\n",
        "**Learning Objectives:**\n",
        "1.   Understand common terminology and preprocessing techniques in NLP\n",
        "2.   Identify several classic and modern approaches used to capture semantic meaning in text\n",
        "3. Learn how to implement a basic NLP pipeline from preprocessing to performance evaluation\n",
        "\n",
        "**Session Authors**\n",
        "\n",
        "Timothy L. Chen MD - University of California San Francisco, San Francisco, USA\n",
        "\n",
        "Jae Ho Sohn MD, MS - University of California San Francisco, San Francisco, USA\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rFZdK4accSlG"
      },
      "source": [
        "# Install Packages\n",
        "\n",
        "We will use a number of prepackaged python NLP tools for this demonstration. Google colab instances come preloaded with a number of packages but not everything. We will install the missing packages here.\n",
        "\n",
        "Natural Language Toolkit ([NLTK](https://www.nltk.org/))\n",
        "\n",
        "[spacy](https://spacy.io/) and [scispacy](https://allenai.github.io/scispacy/)\n",
        "\n",
        "[gensim](https://radimrehurek.com/gensim/)\n",
        "\n",
        "[sklearn](https://scikit-learn.org/stable/)\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W4QbilZBcYyx",
        "outputId": "b70a57d6-c82c-482a-d636-5985fb027970",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "#@title Installing missing required packages\n",
        "!pip install xmltodict >/dev/null\n",
        "!pip install spacy > /dev/null \n",
        "!pip install scispacy > /dev/null\n",
        "!pip install negspacy > /dev/null\n",
        "!pip install https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.4.0/en_core_sci_md-0.4.0.tar.gz> /dev/null\n",
        "\n",
        "#! pip freeze"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "en-core-sci-md 0.4.0 requires spacy<3.1.0,>=3.0.1, but you have spacy 3.4.3 which is incompatible.\u001b[0m\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "scispacy 0.5.1 requires spacy<3.5.0,>=3.4.0, but you have spacy 3.0.8 which is incompatible.\n",
            "en-core-web-sm 3.4.1 requires spacy<3.5.0,>=3.4.0, but you have spacy 3.0.8 which is incompatible.\u001b[0m\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NxJj3ct4Q58_"
      },
      "source": [
        "# Acquiring Data\n",
        "\n",
        "This demo will utilize data from the [Indiana University CXR dataset](https://openi.nlm.nih.gov/faq). \n",
        "\n",
        "More information regarding the creation of this dataset is detailed in [1] *Demner-Fushman et. al. \"Preparing a collection of radiology examinations for distirbution and retrieval\". J. Am. Med. Inform. Assoc. 2016. doi: [10.1093/jamia/ocv080](https://academic.oup.com/jamia/article/23/2/304/2572395)*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LdlPCIiCLYHe",
        "outputId": "2530ee88-200c-4b54-f609-38127411c4c0"
      },
      "source": [
        "# Download the data\n",
        "# tar options - x -extract, v - show progress of archive file, z - filter archive through gzip\n",
        "!curl https://openi.nlm.nih.gov/imgs/collections/NLMCXR_reports.tgz > NLMCXR_reports.tgz\n",
        "!tar xvf /content/NLMCXR_reports.tgz >/dev/null\n"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r100 1086k  100 1086k    0     0  10.7M      0 --:--:-- --:--:-- --:--:-- 10.7M\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mAsCZXsE-HQe",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f8e9082d-daf7-4483-b7fe-a5189292e547"
      },
      "source": [
        "# taking a closer look at our file hierarchy\n",
        "!dir"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ecgen-radiology  NLMCXR_reports.tgz  sample_data\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Converting .csv files to .xml File\n",
        "#Credit, Heavily Inspired: https://stackoverflow.com/questions/41059264/simple-csv-to-xml-conversion-python\n",
        "import csv  \n",
        "import pandas as pd\n",
        "\n",
        "csv_file = ''\n",
        "output_xml = ''\n",
        "def csv_to_xml(csv_file, output_xml):\n",
        "  # Parameters: \n",
        "  #   csv_file: The location of the csv file \n",
        "  #   output_xml: The location of the output xml file\n",
        "  # Output: None, writes the content of the csv file to output_xml\n",
        "  f = open(csv_file)\n",
        "  csv_f = csv.reader(f)   \n",
        "  data = []\n",
        "\n",
        "  for row in csv_f: \n",
        "    data.append(row)\n",
        "  f.close()\n",
        "\n",
        "  df = pd.read_csv(csv_file)\n",
        "  header = list(df.columns)\n",
        "\n",
        "  def convert_row(row):\n",
        "      str_row = \"\"\"<%s>%s</%s> \\n\"\"\"*(len(header)-1)\n",
        "      str_row = \"\"\"<%s>%s\"\"\" +\"\\n\"+ str_row + \"\"\"</%s>\"\"\"\n",
        "      var_values = [list_of_elments[k] for k in range(1,len(header)) for list_of_elments in [header,row,header]]\n",
        "      var_values = [header[0],row[0]]+var_values+[header[0]]\n",
        "      var_values =tuple(var_values)\n",
        "      return str_row % var_values\n",
        "\n",
        "  text = \"\"\"<collection shelf=\"New Arrivals\">\"\"\"+\"\\n\"+'\\n'.join([convert_row(row) for row in data[1:]])+\"\\n\" +\"</collection >\"\n",
        "  print(text)\n",
        "  with open('output.xml', 'w') as myfile: \n",
        "    myfile.write(text)\n",
        "\n",
        "  df= pd.read_csv(csv_file)\n",
        "  with open(output_xml, 'w') as myfile: \n",
        "    myfile.write(df.to_xml())"
      ],
      "metadata": {
        "id": "AULu6133mRRC"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rPkhaPx3gXmI"
      },
      "source": [
        "With permission from [2] Wiggins et. al., we borrow code to parse the xml. More information about this code can be found here: [Natural Language Processing of Radiology Text Reports: Interactive Text Classification](https://pubs.rsna.org/doi/full/10.1148/ryai.2021210035)\n",
        "\n",
        "The XML file format stores data under various headers which we store as key-value pairs in a python dictionary object.\n",
        "\n",
        "Here we first examine the data stucture of one such xml using an explicit file path. We want to extract the relevant text information from each file. We are interested in the text from the \"FINDINGS\" AND \"IMPRESSION\" keys. In addition, the reports are labeled as normal under the MeSH key - we will use this for labeling the reports as normal (if normal is present under MeSH) or abnormal.\n",
        "\n",
        "If we closely examine the data, there are some oddities. Some entries are multiple xrays together. For example, 875.xml contains text for both CXR and a knee X-ray. Some CXR have no text under FINDINGS and IMPRESSION. For some chest x-ray, there is no label given (\"No Indexing\"). These \"No Indexing\" labels will be considered as normal for this exercise. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MIA9a7rckKil"
      },
      "source": [
        "import xmltodict\n",
        "\n",
        "# function from Wiggins et. al.\n",
        "def xml_parse(f):\n",
        "  with open(f) as xml:\n",
        "    report_dict = xmltodict.parse(xml.read())\n",
        "    xml.close()\n",
        "  return report_dict"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GeqrbiKb2R_z"
      },
      "source": [
        "Let's take a look at a random xml file and see the contents after parsing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vptTTkdh2RbE",
        "outputId": "e618a3de-dc14-4c92-b9e0-b19241d38eca"
      },
      "source": [
        "reportdict_1 = xml_parse(\"./ecgen-radiology/1404.xml\")\n",
        "reportdict_1"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'eCitation': {'meta': {'@type': 'rr'},\n",
              "  'uId': {'@id': 'CXR1404'},\n",
              "  'pmcId': {'@id': '1404'},\n",
              "  'docSource': 'CXR',\n",
              "  'IUXRId': {'@id': '1404'},\n",
              "  'licenseType': 'open-access',\n",
              "  'licenseURL': 'http://creativecommons.org/licenses/by-nc-nd/4.0/',\n",
              "  'ccLicense': 'byncnd',\n",
              "  'articleURL': None,\n",
              "  'articleDate': '2013-08-01',\n",
              "  'articleType': 'XR',\n",
              "  'publisher': 'Indiana University',\n",
              "  'title': 'Indiana University Chest X-ray Collection',\n",
              "  'note': 'The data are drawn from multiple hospital systems.',\n",
              "  'specialty': 'pulmonary diseases',\n",
              "  'subset': 'CXR',\n",
              "  'MedlineCitation': {'@Owner': 'Indiana University',\n",
              "   '@Status': 'supplied by publisher',\n",
              "   'Article': {'@PubModel': 'Electronic',\n",
              "    'Journal': {'JournalIssue': {'PubDate': {'Year': '2013',\n",
              "       'Month': '08',\n",
              "       'Day': '01'}}},\n",
              "    'ArticleTitle': 'Indiana University Chest X-ray Collection',\n",
              "    'Abstract': {'AbstractText': [{'@Label': 'COMPARISON', '#text': 'XXXX'},\n",
              "      {'@Label': 'INDICATION',\n",
              "       '#text': 'XXXX year old with chest pain, shortness of breath, hypertension.'},\n",
              "      {'@Label': 'FINDINGS',\n",
              "       '#text': 'No focal lung consolidation. Heart size and pulmonary vascularity are within normal limits. No pneumothorax or pleural effusion. No acute bony abnormalities. There are stable anterior wedge XXXX deformities of 2 midthoracic vertebral bodies.'},\n",
              "      {'@Label': 'IMPRESSION',\n",
              "       '#text': '1. No acute findings. 2. Stable midthoracic vertebral body XXXX fractures.'}]},\n",
              "    'Affiliation': 'Indiana University',\n",
              "    'AuthorList': {'@CompleteYN': 'Y',\n",
              "     'Author': [{'@ValidYN': 'Y',\n",
              "       'LastName': 'Kohli',\n",
              "       'ForeName': 'Marc',\n",
              "       'Initials': 'MD'},\n",
              "      {'@ValidYN': 'Y',\n",
              "       'LastName': 'Rosenman',\n",
              "       'ForeName': 'Marc',\n",
              "       'Initials': 'M'}]},\n",
              "    'Language': 'eng',\n",
              "    'PublicationTypeList': {'PublicationType': 'Radiology Report'},\n",
              "    'ArticleDate': {'Year': '2013', 'Month': '08', 'Day': '01'}},\n",
              "   'EssieArticleTitle': 'Indiana University Chest X-ray Collection',\n",
              "   'IMedAuthor': ['Marc David Kohli MD', 'Marc Rosenman M']},\n",
              "  'MeSH': {'major': ['Deformity/thoracic vertebrae/anterior/multiple',\n",
              "    'Fractures, Bone/Thoracic Vertebrae'],\n",
              "   'automatic': ['deformity', 'fracture', 'Spinal Fractures', 'Spine']},\n",
              "  'parentImage': [{'@id': 'CXR1404_IM-0258-1001',\n",
              "    'figureId': 'F1',\n",
              "    'caption': 'CHEST PA and Lateral on XXXX, XXXX',\n",
              "    'panel': {'@type': 'single',\n",
              "     'url': '/hadoop/storage/radiology/extract/CXR1404_IM-0258-1001.jpg',\n",
              "     'imgModality': '7',\n",
              "     'region': {'@type': 'panel',\n",
              "      'globalImageFeatures': {'CEDD': 'f2p0k680',\n",
              "       'ColorLayout': 'f1p0k26',\n",
              "       'EdgeHistogram': 'f0p0k338',\n",
              "       'FCTH': 'f4p0k369',\n",
              "       'SemanticContext60': 'f3p0k74'}}}},\n",
              "   {'@id': 'CXR1404_IM-0258-2001',\n",
              "    'figureId': 'F2',\n",
              "    'caption': 'CHEST PA and Lateral on XXXX, XXXX',\n",
              "    'panel': {'@type': 'single',\n",
              "     'url': '/hadoop/storage/radiology/extract/CXR1404_IM-0258-2001.jpg',\n",
              "     'imgModality': '7',\n",
              "     'region': {'@type': 'panel',\n",
              "      'globalImageFeatures': {'CEDD': 'f2p0k1637',\n",
              "       'ColorLayout': 'f1p0k25',\n",
              "       'EdgeHistogram': 'f0p0k518',\n",
              "       'FCTH': 'f4p0k1142',\n",
              "       'SemanticContext60': 'f3p0k151'}}}}]}}"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kBZb655Tp2qp"
      },
      "source": [
        "\n",
        "The entirety of the of the xml data files were uncompressed into the \"ecgen-radiology\" folder, so we will need to go through each file and extract the relevant information. The [glob](https://docs.python.org/3/library/glob.html) function in the glob module makes this easy by matching all files whose pathname matches a given pattern using [fnmatch](https://docs.python.org/3/library/fnmatch.html#module-fnmatch)\n",
        "\n",
        "Additionally, we will extract relevant information from each xml file and store it into a pandas dataframe (which is basically a table)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k-N2UbgWnU8P"
      },
      "source": [
        "# functions from Wiggins et. al. , slightly modified to remove dependence on fastai L class\n",
        "\n",
        "def get_label(report):\n",
        "    label = report['eCitation']['MeSH']['major']\n",
        "    return 'normal' if label == 'normal' or label == \"No Indexing\" else 'abnormal'\n",
        "\n",
        "def get_text(report):\n",
        "    text_dict = {}\n",
        "    text_dict['id'] = report['eCitation']['IUXRId']['@id']\n",
        "    text = report['eCitation']['MedlineCitation']['Article']['Abstract']['AbstractText']\n",
        "    findings = text[2]['#text'] if '#text' in text[2] else ''\n",
        "    text_dict['findings'] = findings\n",
        "    impression = text[3]['#text'] if '#text' in text[3] else ''\n",
        "    text_dict['impression'] = impression\n",
        "    text_dict['full-text'] = ' '.join([findings, impression])\n",
        "    return text_dict\n",
        "\n",
        "def process_report(report):\n",
        "    label = get_label(report)\n",
        "    report_dict = get_text(report)\n",
        "    report_dict['label'] = label\n",
        "    return report_dict\n"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tLhLsKYJoces"
      },
      "source": [
        "import glob\n",
        "import pandas as pd\n",
        "\n",
        "files = glob.glob(\"./ecgen-radiology/*\")\n",
        "reports = list()\n",
        "reports_unlabeled = list()\n",
        "\n",
        "for file in files:\n",
        "  report_extract = xml_parse(file)\n",
        "  if report_extract['eCitation']['MeSH']['major'] == \"No Indexing\":\n",
        "    reports_unlabeled.append(file)\n",
        "    \n",
        "  reports.append(process_report(report_extract))\n",
        "  \n",
        "reports_df = pd.DataFrame(reports)\n"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p48E2P5r2wMh"
      },
      "source": [
        "Let's examine some of the data in our table\n",
        "\n",
        "The .head() function will return the first n rows (default 5) of the table. The .sample() method will randomly choose n rows of the table to display (default is 1)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 459
        },
        "id": "kpOxPVK1uzHv",
        "outputId": "897da82c-3070-4cf6-e0b5-f6abe4e10103"
      },
      "source": [
        "print(\"# of reports:\",reports_df.shape[0])\n",
        "pd.options.display.max_colwidth = 0\n",
        "reports_df.head()"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "# of reports: 3955\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     id  \\\n",
              "0  3020   \n",
              "1  622    \n",
              "2  3731   \n",
              "3  825    \n",
              "4  3321   \n",
              "\n",
              "                                                                                                                                                                                                                findings  \\\n",
              "0  Heart size and mediastinal contours are normal in appearance. No consolidative airspace opacities. No radiographic evidence of pleural effusion or pneumothorax. Visualized osseous structures appear intact.           \n",
              "1  Both lungs are clear and expanded. Heart and mediastinum normal.                                                                                                                                                        \n",
              "2  Lungs are clear. Heart and mediastinum appear normal. No pleural effusion or pneumothorax.                                                                                                                              \n",
              "3                                                                                                                                                                                                                          \n",
              "4  XXXX XXXX and lateral chest examination was obtained. The heart silhouette is normal in size and contour. Aortic XXXX appear unremarkable. Lungs demonstrate no acute findings. There is no effusion or pneumothorax.   \n",
              "\n",
              "                                                                                                                                                                                   impression  \\\n",
              "0  No acute cardiopulmonary abnormality.                                                                                                                                                        \n",
              "1  No active disease.                                                                                                                                                                           \n",
              "2  Negative chest                                                                                                                                                                               \n",
              "3  Patchy bilateral airspace disease with pulmonary XXXX, XXXX a combination of atelectasis and consolidation. Borderline cardiac enlargement. Tracheostomy tube tip 6.4 cm above the carina.   \n",
              "4  1. No acute pulmonary disease.                                                                                                                                                               \n",
              "\n",
              "                                                                                                                                                                                                                                              full-text  \\\n",
              "0  Heart size and mediastinal contours are normal in appearance. No consolidative airspace opacities. No radiographic evidence of pleural effusion or pneumothorax. Visualized osseous structures appear intact. No acute cardiopulmonary abnormality.    \n",
              "1  Both lungs are clear and expanded. Heart and mediastinum normal. No active disease.                                                                                                                                                                    \n",
              "2  Lungs are clear. Heart and mediastinum appear normal. No pleural effusion or pneumothorax. Negative chest                                                                                                                                              \n",
              "3   Patchy bilateral airspace disease with pulmonary XXXX, XXXX a combination of atelectasis and consolidation. Borderline cardiac enlargement. Tracheostomy tube tip 6.4 cm above the carina.                                                            \n",
              "4  XXXX XXXX and lateral chest examination was obtained. The heart silhouette is normal in size and contour. Aortic XXXX appear unremarkable. Lungs demonstrate no acute findings. There is no effusion or pneumothorax. 1. No acute pulmonary disease.   \n",
              "\n",
              "      label  \n",
              "0  normal    \n",
              "1  normal    \n",
              "2  normal    \n",
              "3  abnormal  \n",
              "4  normal    "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-39619d8c-5f9c-4890-b172-f72e18f187bf\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>findings</th>\n",
              "      <th>impression</th>\n",
              "      <th>full-text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>3020</td>\n",
              "      <td>Heart size and mediastinal contours are normal in appearance. No consolidative airspace opacities. No radiographic evidence of pleural effusion or pneumothorax. Visualized osseous structures appear intact.</td>\n",
              "      <td>No acute cardiopulmonary abnormality.</td>\n",
              "      <td>Heart size and mediastinal contours are normal in appearance. No consolidative airspace opacities. No radiographic evidence of pleural effusion or pneumothorax. Visualized osseous structures appear intact. No acute cardiopulmonary abnormality.</td>\n",
              "      <td>normal</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>622</td>\n",
              "      <td>Both lungs are clear and expanded. Heart and mediastinum normal.</td>\n",
              "      <td>No active disease.</td>\n",
              "      <td>Both lungs are clear and expanded. Heart and mediastinum normal. No active disease.</td>\n",
              "      <td>normal</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3731</td>\n",
              "      <td>Lungs are clear. Heart and mediastinum appear normal. No pleural effusion or pneumothorax.</td>\n",
              "      <td>Negative chest</td>\n",
              "      <td>Lungs are clear. Heart and mediastinum appear normal. No pleural effusion or pneumothorax. Negative chest</td>\n",
              "      <td>normal</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>825</td>\n",
              "      <td></td>\n",
              "      <td>Patchy bilateral airspace disease with pulmonary XXXX, XXXX a combination of atelectasis and consolidation. Borderline cardiac enlargement. Tracheostomy tube tip 6.4 cm above the carina.</td>\n",
              "      <td>Patchy bilateral airspace disease with pulmonary XXXX, XXXX a combination of atelectasis and consolidation. Borderline cardiac enlargement. Tracheostomy tube tip 6.4 cm above the carina.</td>\n",
              "      <td>abnormal</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>3321</td>\n",
              "      <td>XXXX XXXX and lateral chest examination was obtained. The heart silhouette is normal in size and contour. Aortic XXXX appear unremarkable. Lungs demonstrate no acute findings. There is no effusion or pneumothorax.</td>\n",
              "      <td>1. No acute pulmonary disease.</td>\n",
              "      <td>XXXX XXXX and lateral chest examination was obtained. The heart silhouette is normal in size and contour. Aortic XXXX appear unremarkable. Lungs demonstrate no acute findings. There is no effusion or pneumothorax. 1. No acute pulmonary disease.</td>\n",
              "      <td>normal</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-39619d8c-5f9c-4890-b172-f72e18f187bf')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-39619d8c-5f9c-4890-b172-f72e18f187bf button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-39619d8c-5f9c-4890-b172-f72e18f187bf');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 684
        },
        "id": "gPf0e-SE9Uwc",
        "outputId": "59498fdf-f360-4f34-b7c3-5314422f30cb"
      },
      "source": [
        "reports_df.sample(5)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        id  \\\n",
              "1598  1865   \n",
              "2220  8      \n",
              "2362  3112   \n",
              "3633  1906   \n",
              "2377  3667   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  findings  \\\n",
              "1598  Heart size is normal. The lungs are clear. There are no focal air space consolidations. No pleural effusions or pneumothoraces. Calcified right upper lobe pulmonary granuloma and calcified right hilar lymph XXXX. The hilar and mediastinal contours are normal. Normal pulmonary vascularity.                                                                                                                                                                                                                                      \n",
              "2220  The heart, pulmonary XXXX and mediastinum are within normal limits. There is no pleural effusion or pneumothorax. There is no focal air space opacity to suggest a pneumonia. There is an interim XXXX cervical spinal fusion partly evaluated.                                                                                                                                                                                                                                                                                        \n",
              "2362  The heart is normal in size. The mediastinum is unremarkable. The lungs are clear.                                                                                                                                                                                                                                                                                                                                                                                                                                                     \n",
              "3633                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         \n",
              "2377  Density in the left upper lung on PA XXXX XXXX represents superimposed bony and vascular structures. There is calcification of the first rib costicartilage junction which XXXX contributes to this appearance. The lungs otherwise appear clear. The heart and pulmonary XXXX appear normal. In the pleural spaces are clear. The mediastinal contour is normal. There are degenerative changes of thoracic spine. There is an electronic cardiac device overlying the left chest wall with intact distal leads in the right heart.   \n",
              "\n",
              "                                                                                                                                                                                                                              impression  \\\n",
              "1598  No acute abnormality. .                                                                                                                                                                                                              \n",
              "2220  No acute cardiopulmonary disease.                                                                                                                                                                                                    \n",
              "2362  No acute disease.                                                                                                                                                                                                                    \n",
              "3633  Streaky bibasilar opacities most suggestive of atelectasis with hypoventilation. Heart size mildly enlarged for technique, stable mediastinal contours. No definite pleural effusion seen, no typical findings of pulmonary edema.   \n",
              "2377  1. Irregular density in the left upper lung on PA XXXX, XXXX artifact related to superimposed vascular bony structures. Chest fluoroscopy or XXXX would confirm this 2. Otherwise, no acute cardiopulmonary disease.                 \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      full-text  \\\n",
              "1598  Heart size is normal. The lungs are clear. There are no focal air space consolidations. No pleural effusions or pneumothoraces. Calcified right upper lobe pulmonary granuloma and calcified right hilar lymph XXXX. The hilar and mediastinal contours are normal. Normal pulmonary vascularity. No acute abnormality. .                                                                                                                                                                                                                                                                                                                                                                                                                                   \n",
              "2220  The heart, pulmonary XXXX and mediastinum are within normal limits. There is no pleural effusion or pneumothorax. There is no focal air space opacity to suggest a pneumonia. There is an interim XXXX cervical spinal fusion partly evaluated. No acute cardiopulmonary disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
              "2362  The heart is normal in size. The mediastinum is unremarkable. The lungs are clear. No acute disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        \n",
              "3633   Streaky bibasilar opacities most suggestive of atelectasis with hypoventilation. Heart size mildly enlarged for technique, stable mediastinal contours. No definite pleural effusion seen, no typical findings of pulmonary edema.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         \n",
              "2377  Density in the left upper lung on PA XXXX XXXX represents superimposed bony and vascular structures. There is calcification of the first rib costicartilage junction which XXXX contributes to this appearance. The lungs otherwise appear clear. The heart and pulmonary XXXX appear normal. In the pleural spaces are clear. The mediastinal contour is normal. There are degenerative changes of thoracic spine. There is an electronic cardiac device overlying the left chest wall with intact distal leads in the right heart. 1. Irregular density in the left upper lung on PA XXXX, XXXX artifact related to superimposed vascular bony structures. Chest fluoroscopy or XXXX would confirm this 2. Otherwise, no acute cardiopulmonary disease.   \n",
              "\n",
              "         label  \n",
              "1598  abnormal  \n",
              "2220  normal    \n",
              "2362  normal    \n",
              "3633  abnormal  \n",
              "2377  abnormal  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ec18a85a-b4d4-4ed9-bda3-fecf0158e086\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>findings</th>\n",
              "      <th>impression</th>\n",
              "      <th>full-text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1598</th>\n",
              "      <td>1865</td>\n",
              "      <td>Heart size is normal. The lungs are clear. There are no focal air space consolidations. No pleural effusions or pneumothoraces. Calcified right upper lobe pulmonary granuloma and calcified right hilar lymph XXXX. The hilar and mediastinal contours are normal. Normal pulmonary vascularity.</td>\n",
              "      <td>No acute abnormality. .</td>\n",
              "      <td>Heart size is normal. The lungs are clear. There are no focal air space consolidations. No pleural effusions or pneumothoraces. Calcified right upper lobe pulmonary granuloma and calcified right hilar lymph XXXX. The hilar and mediastinal contours are normal. Normal pulmonary vascularity. No acute abnormality. .</td>\n",
              "      <td>abnormal</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2220</th>\n",
              "      <td>8</td>\n",
              "      <td>The heart, pulmonary XXXX and mediastinum are within normal limits. There is no pleural effusion or pneumothorax. There is no focal air space opacity to suggest a pneumonia. There is an interim XXXX cervical spinal fusion partly evaluated.</td>\n",
              "      <td>No acute cardiopulmonary disease.</td>\n",
              "      <td>The heart, pulmonary XXXX and mediastinum are within normal limits. There is no pleural effusion or pneumothorax. There is no focal air space opacity to suggest a pneumonia. There is an interim XXXX cervical spinal fusion partly evaluated. No acute cardiopulmonary disease.</td>\n",
              "      <td>normal</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2362</th>\n",
              "      <td>3112</td>\n",
              "      <td>The heart is normal in size. The mediastinum is unremarkable. The lungs are clear.</td>\n",
              "      <td>No acute disease.</td>\n",
              "      <td>The heart is normal in size. The mediastinum is unremarkable. The lungs are clear. No acute disease.</td>\n",
              "      <td>normal</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3633</th>\n",
              "      <td>1906</td>\n",
              "      <td></td>\n",
              "      <td>Streaky bibasilar opacities most suggestive of atelectasis with hypoventilation. Heart size mildly enlarged for technique, stable mediastinal contours. No definite pleural effusion seen, no typical findings of pulmonary edema.</td>\n",
              "      <td>Streaky bibasilar opacities most suggestive of atelectasis with hypoventilation. Heart size mildly enlarged for technique, stable mediastinal contours. No definite pleural effusion seen, no typical findings of pulmonary edema.</td>\n",
              "      <td>abnormal</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2377</th>\n",
              "      <td>3667</td>\n",
              "      <td>Density in the left upper lung on PA XXXX XXXX represents superimposed bony and vascular structures. There is calcification of the first rib costicartilage junction which XXXX contributes to this appearance. The lungs otherwise appear clear. The heart and pulmonary XXXX appear normal. In the pleural spaces are clear. The mediastinal contour is normal. There are degenerative changes of thoracic spine. There is an electronic cardiac device overlying the left chest wall with intact distal leads in the right heart.</td>\n",
              "      <td>1. Irregular density in the left upper lung on PA XXXX, XXXX artifact related to superimposed vascular bony structures. Chest fluoroscopy or XXXX would confirm this 2. Otherwise, no acute cardiopulmonary disease.</td>\n",
              "      <td>Density in the left upper lung on PA XXXX XXXX represents superimposed bony and vascular structures. There is calcification of the first rib costicartilage junction which XXXX contributes to this appearance. The lungs otherwise appear clear. The heart and pulmonary XXXX appear normal. In the pleural spaces are clear. The mediastinal contour is normal. There are degenerative changes of thoracic spine. There is an electronic cardiac device overlying the left chest wall with intact distal leads in the right heart. 1. Irregular density in the left upper lung on PA XXXX, XXXX artifact related to superimposed vascular bony structures. Chest fluoroscopy or XXXX would confirm this 2. Otherwise, no acute cardiopulmonary disease.</td>\n",
              "      <td>abnormal</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ec18a85a-b4d4-4ed9-bda3-fecf0158e086')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-ec18a85a-b4d4-4ed9-bda3-fecf0158e086 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-ec18a85a-b4d4-4ed9-bda3-fecf0158e086');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L7TwCTMm9cqB"
      },
      "source": [
        "# Some characteristics of the data\n",
        "\n",
        "We previously already noticed that not all reports had labels associated with them and kept track of a list of such reports in reports_unlabeled. Through further examination of the data we see that some reports (e.g. id 68) only have an impression section but no findings section. THere are also numerous XXXX in reports which likely represent placeholder text after PHI removal (names, dates, etc). Even for normal labeled reports, there is quite a large amount of variation in the text of the reports."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IrbIaKT--Vvs"
      },
      "source": [
        "## Distribution of labels\n",
        "\n",
        "For any data, it is important to examine the distribution of classes (labels) of the data. Imbalanced datasets can have large impacts in terms of how a model behaves after training. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 357
        },
        "id": "GZSWHhOu-ci0",
        "outputId": "c540b967-3f67-49a6-93d0-17bc47319b52"
      },
      "source": [
        "# code from Wiggins et. al.\n",
        "\n",
        "label_counts = reports_df.label.value_counts()\n",
        "print('Abnormal:', label_counts['abnormal'])\n",
        "print('Normal:', label_counts['normal'])\n",
        "print()\n",
        "label_counts.plot.bar();"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Abnormal: 2469\n",
            "Normal: 1486\n",
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEeCAYAAABmGcWlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAARHElEQVR4nO3df6zddX3H8edrBTVRJkUqY6WxDKsJ6gR2RRLNxJHxy2ToXAiYSHW4+kfJcPOfauJADBlLJmZkSlZnYzUqI/EHzezEjhjRGZRb0gAFOm6QhjYIV0FEERR874/zveO03tt7296ebz2f5yM5Od/z/n6/575POLzu936+n++3qSokSW34vb4bkCSNjqEvSQ0x9CWpIYa+JDXE0Jekhhj6ktSQI/puYF+OPfbYWrlyZd9tSNLvlK1bt/64qpbNtu6wDv2VK1cyOTnZdxuS9Dslyc651jm8I0kNMfQlqSHzhn6SFUm+leSeJNuTXN7Vr0yyO8m27nH+0D4fSjKVZEeSc4bq53a1qSTrDs1HkiTNZSFj+s8CH6yqO5IcBWxNsqVb94mq+ufhjZOcDFwEvAb4Q+C/k7yqW/1J4M+BXcDtSTZV1T2L8UEkSfObN/Sr6mHg4W75yST3Asv3scsFwA1V9QzwwyRTwOnduqmqegAgyQ3dtoa+JI3Ifo3pJ1kJnAp8vytdluTOJBuSLO1qy4GHhnbb1dXmqkuSRmTBoZ/kJcCXgQ9U1c+A64GTgFMY/CXw8cVoKMmaJJNJJqenpxfjLSVJnQWFfpIjGQT+F6rqKwBV9UhVPVdVvwE+zfNDOLuBFUO7n9DV5qrvoarWV9VEVU0sWzbrtQWSpAM075h+kgCfAe6tqmuH6sd34/0A7wDu7pY3AV9Mci2DE7mrgB8AAVYlOZFB2F8EvGuxPkifVq77et8tjJUHr3lb3y1IY2shs3feBLwbuCvJtq72YeDiJKcABTwIvB+gqrYnuZHBCdpngbVV9RxAksuAm4ElwIaq2r6In0WSNI+FzN75LoOj9L1t3sc+VwNXz1LfvK/9JEmHllfkSlJDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQ+YN/SQrknwryT1Jtie5vKsfk2RLkvu756VdPUmuSzKV5M4kpw291+pu+/uTrD50H0uSNJuFHOk/C3ywqk4GzgDWJjkZWAfcUlWrgFu61wDnAau6xxrgehj8kgCuAN4InA5cMfOLQpI0GvOGflU9XFV3dMtPAvcCy4ELgI3dZhuBt3fLFwCfq4HbgKOTHA+cA2ypqseq6nFgC3Duon4aSdI+7deYfpKVwKnA94HjqurhbtWPgOO65eXAQ0O77epqc9UlSSOy4NBP8hLgy8AHqupnw+uqqoBajIaSrEkymWRyenp6Md5SktRZUOgnOZJB4H+hqr7SlR/phm3onh/t6ruBFUO7n9DV5qrvoarWV9VEVU0sW7Zsfz6LJGkeC5m9E+AzwL1Vde3Qqk3AzAyc1cBNQ/VLulk8ZwBPdMNANwNnJ1nancA9u6tJkkbkiAVs8ybg3cBdSbZ1tQ8D1wA3JrkU2Alc2K3bDJwPTAFPAe8FqKrHknwMuL3b7qqqemxRPoUkaUHmDf2q+i6QOVafNcv2Bayd4702ABv2p0FJ0uLxilxJaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWrIvKGfZEOSR5PcPVS7MsnuJNu6x/lD6z6UZCrJjiTnDNXP7WpTSdYt/keRJM1nIUf6nwXOnaX+iao6pXtsBkhyMnAR8Jpun08lWZJkCfBJ4DzgZODibltJ0ggdMd8GVXVrkpULfL8LgBuq6hngh0mmgNO7dVNV9QBAkhu6be/Z744lSQfsYMb0L0tyZzf8s7SrLQceGtpmV1ebqy5JGqEDDf3rgZOAU4CHgY8vVkNJ1iSZTDI5PT29WG8rSWIBwzuzqapHZpaTfBr4z+7lbmDF0KYndDX2Ud/7vdcD6wEmJibqQPqT9LyV677edwtj48Fr3tZ3CwftgI70kxw/9PIdwMzMnk3ARUlemOREYBXwA+B2YFWSE5O8gMHJ3k0H3rYk6UDMe6Sf5EvAmcCxSXYBVwBnJjkFKOBB4P0AVbU9yY0MTtA+C6ytque697kMuBlYAmyoqu2L/mkkSfu0kNk7F89S/sw+tr8auHqW+mZg8351J0laVF6RK0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDZk39JNsSPJokruHasck2ZLk/u55aVdPkuuSTCW5M8lpQ/us7ra/P8nqQ/NxJEn7spAj/c8C5+5VWwfcUlWrgFu61wDnAau6xxrgehj8kgCuAN4InA5cMfOLQpI0OvOGflXdCjy2V/kCYGO3vBF4+1D9czVwG3B0kuOBc4AtVfVYVT0ObOG3f5FIkg6xAx3TP66qHu6WfwQc1y0vBx4a2m5XV5urLkkaoYM+kVtVBdQi9AJAkjVJJpNMTk9PL9bbSpI48NB/pBu2oXt+tKvvBlYMbXdCV5ur/luqan1VTVTVxLJlyw6wPUnSbA409DcBMzNwVgM3DdUv6WbxnAE80Q0D3QycnWRpdwL37K4mSRqhI+bbIMmXgDOBY5PsYjAL5xrgxiSXAjuBC7vNNwPnA1PAU8B7AarqsSQfA27vtruqqvY+OSxJOsTmDf2quniOVWfNsm0Ba+d4nw3Ahv3qTpK0qLwiV5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGnJQoZ/kwSR3JdmWZLKrHZNkS5L7u+elXT1JrksyleTOJKctxgeQJC3cYhzpv7WqTqmqie71OuCWqloF3NK9BjgPWNU91gDXL8LPliTth0MxvHMBsLFb3gi8faj+uRq4DTg6yfGH4OdLkuZwsKFfwDeTbE2ypqsdV1UPd8s/Ao7rlpcDDw3tu6ur7SHJmiSTSSanp6cPsj1J0rAjDnL/N1fV7iQvB7YkuW94ZVVVktqfN6yq9cB6gImJif3aV5K0bwd1pF9Vu7vnR4GvAqcDj8wM23TPj3ab7wZWDO1+QleTJI3IAYd+khcnOWpmGTgbuBvYBKzuNlsN3NQtbwIu6WbxnAE8MTQMJEkagYMZ3jkO+GqSmff5YlV9I8ntwI1JLgV2Ahd2228GzgemgKeA9x7Ez5YkHYADDv2qegB4/Sz1nwBnzVIvYO2B/jxJ0sHzilxJaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWrIyEM/yblJdiSZSrJu1D9fklo20tBPsgT4JHAecDJwcZKTR9mDJLVs1Ef6pwNTVfVAVf0KuAG4YMQ9SFKzjhjxz1sOPDT0ehfwxuENkqwB1nQvf55kx4h6a8GxwI/7bmI++ae+O1BPDvvv5+/Qd/MVc60YdejPq6rWA+v77mMcJZmsqom++5Bm4/dzNEY9vLMbWDH0+oSuJkkagVGH/u3AqiQnJnkBcBGwacQ9SFKzRjq8U1XPJrkMuBlYAmyoqu2j7KFxDpvpcOb3cwRSVX33IEkaEa/IlaSGGPqS1BBDX5IaYuhLUkMOu4uztDiS/P2+1lfVtaPqRRqW5Jh9ra+qx0bVS4sM/fF1VN8NSHPYChSQWdYV8EejbactTtmUpIZ4pD/mkrwIuBR4DfCimXpV/XVvTUmdJEuBVez53by1v47Gnydyx9/ngT8AzgG+zeB+R0/22pEEJHkfcCuDK/Q/2j1f2WdPLTD0x98rq+ojwC+qaiPwNva6nbXUk8uBNwA7q+qtwKnAT/ttafwZ+uPv193zT5O8Fngp8PIe+5FmPF1VTwMkeWFV3Qe8uueexp5j+uNvfTdu+hEGdzR9CfAP/bYkAbArydHA14AtSR4Hdvbc09hz9o6k3iV5C4O/Qr/R/VOqOkQM/THXHUldAqxk6C+7qvrbvnqSZnR/ha5gz+/mHf11NP4c3hl/m4HbgLuA3/Tci/T/knwMeA/wAM9/Nwv4s756aoFH+mMuyR1VdVrffUh7S7IDeJ3DOaPl7J3x9/kkf5Pk+CTHzDz6bkoC7gaO7ruJ1nikP+aSrAWuZjD/eeY/dlWV9zdRr5JMADcxCP9nZupV9Re9NdUAQ3/MJXkAOL2qftx3L9KwJNuBf2Ov801V9e3emmqAJ3LH3xTwVN9NSLN4qqqu67uJ1hj64+8XwLYk32LPP6Gdsqm+fSfJPzK4aHD4u+mUzUPI0B9/X+se0uHm1O75jKGaUzYPMUN/jCVZArynu5mVdNjovpubquoTfffSGqdsjrGqeg74TZKX9t2LNKz7bl7cdx8t8kh//P0cuCvJFgbj+4Bj+jos/E+SfwX+gz2/m47pH0JO2RxzSVbPVu/urS/1pptcsLeqKsf0DyFDvwFJXgC8qnu5o6p+va/tJY0vQ3/MJTkT2Ag8CITBHQ1X+++Qqm/duaYrgD/tSt8GrqqqJ/rravwZ+mMuyVbgXVW1o3v9KuBLVfUn/Xam1iX5MoNbMMwMNb4beH1V/WV/XY0/T+SOvyNnAh+gqv43yZF9NiR1Tqqqdw69/miSbb110winbI6/yST/nuTM7vFpYLLvpiTgl0nePPMiyZuAX/bYTxMc3hlzSV4IrAVm/uf6DvCpqnpm7r2kQy/JKQyGdmauI3mcwfmmO/vravwZ+pJ60R2Q/BVwEoP76j/BYMrmVb02NuYc0x9z3Z/MVwKvYM9/h9T76atvNzH4dx7uAHb33EszPNIfc0nuA/4O2Ao8N1Ovqp/01pQEJLm7ql7bdx+t8Uh//D1RVf/VdxPSLL6X5HVVdVffjbTEI/0xl+QaYAnwFbxnuQ4jSe4BXgn8kMF3MwzG9P+418bGnKE/5ry/iQ5XSV4xW72qdo66l5YY+pLUEC/OGnNJXpbkuiR3JNma5F+SvKzvviT1w9AffzcA08A7GcyJnmZw/3JJDXJ4Z8zNNi0uyV1V9bq+epLUH4/0x983k1yU5Pe6x4XAzX03JakfHumPqSRPAsVgGtyLef7CrCXAz6vq9/vqTVJ/vDhrTFXVUTPLSY4BVgEv6q8jSYcDQ3/MJXkfcDlwArANOAP4HnBWn31J6odj+uPvcuANwM6qeitwKoO7GUpqkKE//p6uqqdhcCvbqroPeHXPPUnqicM7429XkqOBrwFbkjwOeJm71Chn7zQkyVsY/CtF36iqX/Xdj6TRM/QlqSGO6UtSQwx9SWqIoS9JDTH0Jakhhr4kNeT/APU4ar7H2egEAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2gkDEelM-2he"
      },
      "source": [
        "# Training and test set split\n",
        "\n",
        "We will carve the data into two separate data sets - a training set to train the model and a test set to evaluate the model performance. It is necessary to use a separate dataset on which the model is not trained to evaluate the model because models can overfit onto training data and perform very well on that particular dataset but fail to generalize to examples it has not seen before.\n",
        "\n",
        "Similarly, a third validation set is sometimes employed as part of the model development process for hyperparameter tuning which is  part of the training process.\n",
        "\n",
        "For simplicity, we will use a single stratified train-test split, but a cross-validation approach with multiple train-test splits can offer better insight into how a model performs.\n",
        "\n",
        "The stratification refers to how the sampling is done when the data is split into separate data sets. This is important when there is class imbalance in the dataset because you could conceivably end up with one dataset only having members of one class. \n",
        "\n",
        "The stratify option in the train_test_split function takes the labels of the dataset as input. When this option is enabled, the sampler will attempt to preserve class proportions of the input dataset (aka the input dataset labels) in the split datasets via stratified sampling. Note this does not solve the class imbalance problem on its own."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lGzyqyLg-1fY"
      },
      "source": [
        "import numpy as np\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "# fix seed for reproducibility\n",
        "np.random.seed(2021)\n",
        "\n",
        "train_df, test_df = train_test_split(reports_df, test_size=0.15, stratify=reports_df.label.values)"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"train_df shape: \", train_df.shape)\n",
        "print(\"test_df shape: \", test_df.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xofRltUQ5Maq",
        "outputId": "ee746f78-f137-4c68-fc3d-39a94d5b64ec"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "train_df shape:  (3361, 5)\n",
            "test_df shape:  (594, 5)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G1UzvyxVF_f0"
      },
      "source": [
        "If we examine the dataset labels of the test set, we can see that we have similar proportions of abnormal and normal labels compared to the original full dataset (approx 62% abnormal)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        },
        "id": "6sXQ7JaPCGCB",
        "outputId": "7d98a288-b955-49bf-abcd-8ba8c594e08e"
      },
      "source": [
        "# code from Wiggins et. al.\n",
        "\n",
        "print('Test set size:', test_df.shape[0])\n",
        "test_counts = test_df.label.value_counts()\n",
        "print('Abnormal:', test_counts['abnormal'])\n",
        "print('Normal:', test_counts['normal'])\n",
        "print()\n",
        "test_counts.plot.bar();"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Test set size: 594\n",
            "Abnormal: 371\n",
            "Normal: 223\n",
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEeCAYAAABxO1VsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAASO0lEQVR4nO3df6xfdX3H8efLgmCmsyB3rGurZVo1qLO4K7K4TIRsIiYrTkdgiVbH1i3BDDezDE2cuo0MkykZ2SSrw1nNJhJ/0Sj+6BjROSN4yzp+s3X8CG0qXBUYDGGjvPfH99z5bb3t/fm9p3zu85GcfM/5nM/5ft83/fZ1z/18P+d8U1VIktrytL4LkCQtPsNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBR/RdAMBxxx1X69at67sMSXpK2bFjx/eqamy6fYdFuK9bt46JiYm+y5Ckp5Qk9xxsn8MyktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYdFhcxPVWsu/BLfZfQlLsvfkPfJUjN8sxdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGzRjuSY5Ocn2Sf0tyS5IPdO0fT3JXkp3dsqFrT5JLk+xKcmOSV4z6h5Ak7W82FzE9DpxWVY8kORL4ZpIvd/v+sKo+c0D/1wPru+VVwGXdoyRpicx45l4Dj3SbR3ZLHeKQjcAnuuO+DaxMsmrhpUqSZmtWY+5JViTZCdwPbK+q67pdF3VDL5ckOaprWw3cO3T47q5NkrREZhXuVbWvqjYAa4CTk7wUeDfwYuCVwLHAH83lhZNsTjKRZGJycnKOZUuSDmVOs2Wq6kHgWuCMqtrbDb08DvwdcHLXbQ+wduiwNV3bgc+1parGq2p8bGxsftVLkqY1m9kyY0lWduvPAH4ZuH1qHD1JgLOAm7tDtgFv7WbNnAI8VFV7R1K9JGlas5ktswrYmmQFg18GV1bVF5P8U5IxIMBO4He7/lcDZwK7gEeBty9+2ZKkQ5kx3KvqRuCkadpPO0j/As5feGmSpPnyClVJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQTOGe5Kjk1yf5N+S3JLkA137CUmuS7IryaeTPL1rP6rb3tXtXzfaH0GSdKDZnLk/DpxWVS8HNgBnJDkF+CBwSVW9AHgAOK/rfx7wQNd+SddPkrSEZgz3Gnik2zyyWwo4DfhM174VOKtb39ht0+0/PUkWrWJJ0oxmNeaeZEWSncD9wHbgP4EHq+qJrstuYHW3vhq4F6Db/xDwnMUsWpJ0aLMK96raV1UbgDXAycCLF/rCSTYnmUgyMTk5udCnkyQNmdNsmap6ELgW+AVgZZIjul1rgD3d+h5gLUC3/9nA96d5ri1VNV5V42NjY/MsX5I0ndnMlhlLsrJbfwbwy8BtDEL+zV23TcBV3fq2bptu/z9VVS1m0ZKkQzti5i6sArYmWcHgl8GVVfXFJLcCVyT5M+Bfgcu7/pcDn0yyC/gBcM4I6pYkHcKM4V5VNwInTdN+J4Px9wPbHwN+fVGqkyTNi1eoSlKDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ2aMdyTrE1ybZJbk9yS5IKu/f1J9iTZ2S1nDh3z7iS7ktyR5HWj/AEkST9uxi/IBp4A3lVVNyR5FrAjyfZu3yVV9RfDnZOcCJwDvAT4GeAfk7ywqvYtZuGSpIOb8cy9qvZW1Q3d+sPAbcDqQxyyEbiiqh6vqruAXcDJi1GsJGl25jTmnmQdcBJwXdf0jiQ3JvlYkmO6ttXAvUOH7ebQvwwkSYts1uGe5JnAZ4F3VtV/AZcBzwc2AHuBD83lhZNsTjKRZGJycnIuh0qSZjCrcE9yJINg//uq+hxAVd1XVfuq6kngo/xo6GUPsHbo8DVd236qaktVjVfV+NjY2EJ+BknSAWYzWybA5cBtVfXhofZVQ93eCNzcrW8DzklyVJITgPXA9YtXsiRpJrOZLfNq4C3ATUl2dm3vAc5NsgEo4G7gdwCq6pYkVwK3Mphpc74zZSRpac0Y7lX1TSDT7Lr6EMdcBFy0gLokSQvgFaqS1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGjSbe8tIOsytu/BLfZfQlLsvfkPfJSyYZ+6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBs0Y7knWJrk2ya1JbklyQdd+bJLtSf6jezyma0+SS5PsSnJjkleM+oeQJO1vNmfuTwDvqqoTgVOA85OcCFwIXFNV64Frum2A1wPru2UzcNmiVy1JOqQZw72q9lbVDd36w8BtwGpgI7C167YVOKtb3wh8oga+DaxMsmrRK5ckHdScxtyTrANOAq4Djq+qvd2u7wLHd+urgXuHDtvdtUmSlsiswz3JM4HPAu+sqv8a3ldVBdRcXjjJ5iQTSSYmJyfncqgkaQazCvckRzII9r+vqs91zfdNDbd0j/d37XuAtUOHr+na9lNVW6pqvKrGx8bG5lu/JGkas5ktE+By4Laq+vDQrm3Apm59E3DVUPtbu1kzpwAPDQ3fSJKWwGzu5/5q4C3ATUl2dm3vAS4GrkxyHnAPcHa372rgTGAX8Cjw9kWtWJI0oxnDvaq+CeQgu0+fpn8B5y+wLknSAniFqiQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktSgGcM9yceS3J/k5qG29yfZk2Rnt5w5tO/dSXYluSPJ60ZVuCTp4GZz5v5x4Ixp2i+pqg3dcjVAkhOBc4CXdMd8JMmKxSpWkjQ7M4Z7VX0D+MEsn28jcEVVPV5VdwG7gJMXUJ8kaR4WMub+jiQ3dsM2x3Rtq4F7h/rs7tokSUtovuF+GfB8YAOwF/jQXJ8gyeYkE0kmJicn51mGJGk68wr3qrqvqvZV1ZPAR/nR0MseYO1Q1zVd23TPsaWqxqtqfGxsbD5lSJIOYl7hnmTV0OYbgamZNNuAc5IcleQEYD1w/cJKlCTN1REzdUjyKeBU4Lgku4H3Aacm2QAUcDfwOwBVdUuSK4FbgSeA86tq32hKlyQdzIzhXlXnTtN8+SH6XwRctJCiJEkL4xWqktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoNmDPckH0tyf5Kbh9qOTbI9yX90j8d07UlyaZJdSW5M8opRFi9Jmt5sztw/DpxxQNuFwDVVtR64ptsGeD2wvls2A5ctTpmSpLmYMdyr6hvADw5o3ghs7da3AmcNtX+iBr4NrEyyarGKlSTNznzH3I+vqr3d+neB47v11cC9Q/12d22SpCW04A9Uq6qAmutxSTYnmUgyMTk5udAyJElD5hvu900Nt3SP93fte4C1Q/3WdG0/pqq2VNV4VY2PjY3NswxJ0nTmG+7bgE3d+ibgqqH2t3azZk4BHhoavpEkLZEjZuqQ5FPAqcBxSXYD7wMuBq5Mch5wD3B21/1q4ExgF/Ao8PYR1CxJmsGM4V5V5x5k1+nT9C3g/IUWJUlaGK9QlaQGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkho043eoHkqSu4GHgX3AE1U1nuRY4NPAOuBu4OyqemBhZUqS5mIxztxfW1Ubqmq8274QuKaq1gPXdNuSpCU0imGZjcDWbn0rcNYIXkOSdAgLDfcCvpZkR5LNXdvxVbW3W/8ucPwCX0OSNEcLGnMHfrGq9iT5KWB7ktuHd1ZVJanpDux+GWwGeO5zn7vAMiRJwxZ05l5Ve7rH+4HPAycD9yVZBdA93n+QY7dU1XhVjY+NjS2kDEnSAeYd7kl+IsmzptaBXwFuBrYBm7pum4CrFlqkJGluFjIsczzw+SRTz/MPVfWVJN8BrkxyHnAPcPbCy5QkzcW8w72q7gRePk3794HTF1KUJGlhvEJVkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJatDIwj3JGUnuSLIryYWjeh1J0o8bSbgnWQH8NfB64ETg3CQnjuK1JEk/blRn7icDu6rqzqr6H+AKYOOIXkuSdIAjRvS8q4F7h7Z3A68a7pBkM7C523wkyR0jqmU5Og74Xt9FzCQf7LsC9cD35uJ63sF2jCrcZ1RVW4Atfb1+y5JMVNV433VIB/K9uXRGNSyzB1g7tL2ma5MkLYFRhft3gPVJTkjydOAcYNuIXkuSdICRDMtU1RNJ3gF8FVgBfKyqbhnFa2laDnfpcOV7c4mkqvquQZK0yLxCVZIaZLhLUoMMd0lqkOEuSQ3q7SImLY4kf3Co/VX14aWqRRqW5NhD7a+qHyxVLcuR4f7U96y+C5AOYgdQQKbZV8DPLm05y4tTISWpQZ65NyLJ0cB5wEuAo6faq+o3eytK6iQ5BljP/u/Nb/RXUfv8QLUdnwR+Gngd8HUG9/N5uNeKJCDJbwHfYHDF+ge6x/f3WdNyYLi34wVV9V7gv6tqK/AGDrjNstSTC4BXAvdU1WuBk4AH+y2pfYZ7O/63e3wwyUuBZwM/1WM90pTHquoxgCRHVdXtwIt6rql5jrm3Y0s3rvleBnfgfCbwx/2WJAGwO8lK4AvA9iQPAPf0XFPznC0jackkeQ2Dvyq/0n0Fp0bEcG9Ed2b0VmAdQ3+RVdXv9VWTNKX7q3It+783b+ivovY5LNOOq4FvAzcBT/Zci/T/kvwp8DbgTn703izgtL5qWg48c29Ekhuq6hV91yEdKMkdwMschllazpZpxyeT/HaSVUmOnVr6LkoCbgZW9l3EcuOZeyOSnA9cxGD+8NQ/alWV9+9Qr5KMA1cxCPnHp9qr6ld7K2oZMNwbkeRO4OSq+l7ftUjDktwC/A0HfB5UVV/vrahlwA9U27ELeLTvIqRpPFpVl/ZdxHJjuLfjv4GdSa5l/z99nQqpvv1zkj9ncHHd8HvTqZAjZLi34wvdIh1uTuoeTxlqcyrkiBnuDUiyAnhbd1Mm6bDRvTe3VdUlfdey3DgVsgFVtQ94Msmz+65FGta9N8/tu47lyDP3djwC3JRkO4Pxd8Axdx0W/iXJXwGfZv/3pmPuI+RUyEYk2TRde3dvd6k33Yf8B6qqcsx9hAz3hiR5OvDCbvOOqvrfQ/WX1C7DvRFJTgW2Ancz+Lb5tcAmv6dSfes+C3of8Etd09eBP6mqh/qrqn2GeyOS7AB+o6ru6LZfCHyqqn6+38q03CX5LINbD0wNEb4FeHlV/Vp/VbXPD1TbceRUsANU1b8nObLPgqTO86vqTUPbH0iys7dqlgmnQrZjIsnfJjm1Wz4KTPRdlAT8MMkvTm0keTXwwx7rWRYclmlEkqOA84Gp/0T/DHykqh4/+FHS6CXZwGBIZuo6jAcYfB50Y39Vtc9wlzRS3YnHm4HnM7iv+0MMpkL+Sa+FNc4x90Z0f+q+H3ge+39PpfdzV9+uYvA9AzcAe3quZdnwzL0RSW4Hfh/YAeybaq+q7/dWlAQkubmqXtp3HcuNZ+7teKiqvtx3EdI0vpXkZVV1U9+FLCeeuTciycXACuBzeM9sHUaS3Aq8ALiLwXszDMbcf67XwhpnuDfC+3focJXkedO1V9U9S13LcmK4S1KDvIipEUmek+TSJDck2ZHkL5M8p++6JPXDcG/HFcAk8CYGc4onGdw/W9Iy5LBMI6abbpbkpqp6WV81SeqPZ+7t+FqSc5I8rVvOBr7ad1GS+uGZ+1NckocZfJN8gJ/gRxcwrQAeqaqf7Ks2Sf3xIqanuKp61tR6kmOB9cDR/VUk6XBguDciyW8BFwBrgJ3AKcC3gNP7rEtSPxxzb8cFwCuBe6rqtcBJDO6+J2kZMtzb8VhVPQaDW6xW1e3Ai3quSVJPHJZpx+4kK4EvANuTPAB4ebe0TDlbpkFJXsPgW2++UlX/03c9kpae4S5JDXLMXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQf8Hde7biSOq6ucAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AT-jzMJb2ivw"
      },
      "source": [
        "# Preprocessing\n",
        "\n",
        "Most models will not understand a stream of free text as is. Before we can use our data as input into a model, we need to convert it into a form that can be understood by an algorithm. Often this is done by creating a \"vocabulary\" of words such that every unique word has a unique index and thus we can input a word's index as input into a model (or use the index to look up some numerical representation of the word).\n",
        "\n",
        "The following sections will examine pieces of a preprocessing pipeline in isolation using independent sections of code but then put these pieces together as one function at the end."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ngfIeq-ep7-"
      },
      "source": [
        "## Regular expressions (regex)\n",
        "The data used in this tutorial was already nicely packaged into distinct Findings and Impression sections. Most of the time when working with radiology reports you'll have to divide the report into these separate entities yourself. One way to do this is with regular expressions which are rule based expressions that can match patterns of text. Regular expressions can be a powerful way to automatically segment reports or identify sections of text if pre-existing knowledge about their format is known.\n",
        "\n",
        "[Cheat sheet](http://stanford.edu/~wpmarble/webscraping_tutorial/regex_cheatsheet.pdf) for regex syntax\n",
        "\n",
        "We will not extensively cover regular expression nuances in this tutorial but just show a simple example of how they are used with respect to the Python re package. Regular expressions are a general framework thus you may also see packages outside of the re package which interface with regular expressions.\n",
        "\n",
        "More about regular expressions can be found [here](https://www.regular-expressions.info/refflavors.html).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wvgJFJX-uRZJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "72773755-6135-468b-b914-f5ffbb622932"
      },
      "source": [
        "#@title\n",
        "import re\n",
        "\n",
        "regex_test_string = \"\"\"\n",
        "123-456-7890\n",
        "(123)456-7890\n",
        "(123)-456-7890\n",
        "123 456 7890\n",
        "123.456.7890\n",
        "+1-123-456-7890\n",
        "+1 (321) 456-7890\n",
        "+91 123-456-7890\n",
        "\"\"\"\n",
        "\n",
        "regex_pattern = \"\\+?(\\d{1,3})?[-. (]?[-. (]?(\\d{3})[-. )]?[-. )]?\\d{3}[-. ]?\\d{4}\"#@param {type:\"string\"}\n",
        "m = re.finditer(regex_pattern,regex_test_string)\n",
        "\n",
        "for match in m:\n",
        "  print(match.group())\n",
        "  print(f\"Area code is {match.group(2)}\\n\")"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "123-456-7890\n",
            "Area code is 123\n",
            "\n",
            "(123)456-7890\n",
            "Area code is 123\n",
            "\n",
            "(123)-456-7890\n",
            "Area code is 123\n",
            "\n",
            "123 456 7890\n",
            "Area code is 123\n",
            "\n",
            "123.456.7890\n",
            "Area code is 123\n",
            "\n",
            "+1-123-456-7890\n",
            "Area code is 123\n",
            "\n",
            "+1 (321) 456-7890\n",
            "Area code is 321\n",
            "\n",
            "+91 123-456-7890\n",
            "Area code is 123\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z2kgbwY9T82E"
      },
      "source": [
        "Note by default regex does greedy matching\n",
        "Greedy matching - match as much as text as possible to fulfill the pattern even if a smaller subsection of the text would have already fulfilled the pattern.\n",
        "\n",
        "In the following example we look for a string that begins with \"h\", has at least one character of anything except line breaks, and then an \"l\".\n",
        "\n",
        "If our test string is \"hello\" , then the substring \"hel\" matches our objective but so does \"hell\" if we go one character further. The default behavior is to return the longer version (hence greedy). If we wanted the non-greedy version, adding a \"?\" after a quantifier makes it non-greedy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Qys-4is5T76_",
        "outputId": "cb1db49c-b8da-4ce3-d77f-efc988a65ef4"
      },
      "source": [
        "#@title\n",
        "regex_pattern = \"h.+l\"\n",
        "m = re.finditer(regex_pattern,\"hello\")\n",
        "for match in m:\n",
        "  print(match.group())"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "hell\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a7CnEewdUKJf",
        "outputId": "9c8ba0a1-ee41-41c8-cc34-13679f97360e"
      },
      "source": [
        "#@title\n",
        "regex_pattern = \"h.+?l\"\n",
        "m = re.finditer(regex_pattern,\"hello\")\n",
        "for match in m:\n",
        "  print(match.group())"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "hel\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NXLQtF-d2eYv"
      },
      "source": [
        "## Word Tokenization\n",
        "\n",
        "The first step in processing text is often tokenization. This is a process in which text is split into discrete sections which are known as tokens. A natural place to create these sections (for English) is dividing at spaces so that each word represents a token. However, tokens could also be defined at a phrase level (multiple words), sentence level, or character level. For simplicity here, we will tokenize at the single word level."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zelqyCW7HHZE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "50dd077c-d2c1-42a0-97c1-2b8d2ae0ece4"
      },
      "source": [
        "import nltk\n",
        "nltk.download('punkt')\n",
        "text = \"Heart size upper limits of normal. Mediastinal contours are maintained. The patient is mildly rotated. There is a small to moderate sized right apical pneumothorax which measures approximately 2.0 cm. No focal airspace consolidation is seen. Left chest is clear. No definite displaced bony injury is seen. Results called XXXX. XXXX XXXX p.m. XXXX, XXXX. Small to moderate right apical pneumothorax.\"\n",
        "text_tokens = nltk.tokenize.word_tokenize(text)\n",
        "print(text_tokens)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['Heart', 'size', 'upper', 'limits', 'of', 'normal', '.', 'Mediastinal', 'contours', 'are', 'maintained', '.', 'The', 'patient', 'is', 'mildly', 'rotated', '.', 'There', 'is', 'a', 'small', 'to', 'moderate', 'sized', 'right', 'apical', 'pneumothorax', 'which', 'measures', 'approximately', '2.0', 'cm', '.', 'No', 'focal', 'airspace', 'consolidation', 'is', 'seen', '.', 'Left', 'chest', 'is', 'clear', '.', 'No', 'definite', 'displaced', 'bony', 'injury', 'is', 'seen', '.', 'Results', 'called', 'XXXX', '.', 'XXXX', 'XXXX', 'p.m.', 'XXXX', ',', 'XXXX', '.', 'Small', 'to', 'moderate', 'right', 'apical', 'pneumothorax', '.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o4fhncsA205b"
      },
      "source": [
        "## Stopword removal\n",
        "\n",
        "Stop words are high-frequeny words in language that tend to appear in any context due to their grammatical role (e.g. \"a\", \"the\", \"of\"). They generally do not contain much information, so it is customary to remove these words as part of the prprocessing step. In general this works well, but phrases that rely on stopwords can lose meaning (e.g. \"to be or not to be\"). \n",
        "\n",
        "In this case, XXXX should be removed as well.\n",
        "\n",
        "Removal of stopwords is as simple as defining a set of words that you want to remove and then removing these words by scanning through input text and filtering out words that are within the set.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c1hZAWWsHMpg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c11b9d86-7446-4cd3-ed68-cc29d72aa3ba"
      },
      "source": [
        "import nltk\n",
        "nltk.download('stopwords')\n",
        "from nltk.corpus import stopwords\n",
        "print(stopwords.words('english'))"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn', \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\"]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gkTIN1FPRUbw"
      },
      "source": [
        "The imported stopwords exist originally as a list structure, but we should convert to a set for better performance. We can also add custom words to expand this set. Notice our set of stopwords is also all lower case. We will lowercase our tokens prior to matching to the stopword set.\n",
        "\n",
        "In addition to stopwords, numbers are often removed as well (e.g. list numberings) but in the context of radiology these numbers can be very important (e.g. size of a lung nodule)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "j28XvnypRTSq",
        "outputId": "ef421f7a-f370-48d3-8e10-0ca3c1921bbf"
      },
      "source": [
        "import string\n",
        "stop = set(stopwords.words('english')+list(string.punctuation))\n",
        "stop.add(\"xxxx\")\n",
        "\n",
        "filtered_tokens = [i.lower() for i in text_tokens if i.lower() not in stop ]\n",
        "print(filtered_tokens)"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['heart', 'size', 'upper', 'limits', 'normal', 'mediastinal', 'contours', 'maintained', 'patient', 'mildly', 'rotated', 'small', 'moderate', 'sized', 'right', 'apical', 'pneumothorax', 'measures', 'approximately', '2.0', 'cm', 'focal', 'airspace', 'consolidation', 'seen', 'left', 'chest', 'clear', 'definite', 'displaced', 'bony', 'injury', 'seen', 'results', 'called', 'p.m.', 'small', 'moderate', 'right', 'apical', 'pneumothorax']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iqPsXyKZ2bf7"
      },
      "source": [
        "## Word normalization - stemming and lemmatization\n",
        "\n",
        "Word (or text) normalization is reducing text to a standardized form before processing. One way to do this is word stemming and/or lemmatization. In word stemming, the suffix of a word is removed, often leaving behind a stem that is not a full word. In lemmatization, words are transformed into a base word. For example, it is common to remove verb conjugations to reduce different forms of a verb to a single word/token - transforming flying, flown, flew -> fly. \n",
        "\n",
        "We will introduce the concept here, but it will not be included within the preprocessing pipeline. Medical stemming is a difficult topic due to many words sharing roots while still having different meanings (e.g. pneumonia, pneumonitis, pneumonic, pneumothorax). Most available stemming packages are not designed for medical stemming and could introduce problems by truncating useful parts of words (\"over stemming\"). \n",
        "\n",
        "In addition to stemming/lemmatizing words which are based on word roots, one can normalize words by mapping words to a controlled vocabulary. This is especially useful in medicine where a number of terms could refer to the same entity (e.g. hepatorenal recess vs subhepatic recess vs pouch of Morison vs Morison's pouch). Examples of efforts to create standardized medical vocabulary include [UMLS](https://www.nlm.nih.gov/research/umls/index.html), [SNOMED-CT](https://www.nlm.nih.gov/healthit/snomedct/index.html), and [RadLex](http://radlex.org/)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l3f1ZJUlLRB3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e32d5980-1bef-4679-da54-c2d515da96a5"
      },
      "source": [
        "import nltk\n",
        "nltk.download(\"wordnet\")\n",
        "from nltk.stem import WordNetLemmatizer\n",
        "from nltk.stem.porter import PorterStemmer"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "umebJGsHLX5w"
      },
      "source": [
        "The [Porter Stemmer](http://snowball.tartarus.org/algorithms/porter/stemmer.html) is a commonly used stemmer which uses an extensive rule based approach to stem words. The stems generated by stemming may not be \"real\" words which is ok as the point is to normalize similar words into the same token."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "wzAepJ9VJMG7",
        "outputId": "75844cf1-e704-42ea-fb05-bc7a6ecb6ee7"
      },
      "source": [
        "stemmer = PorterStemmer()\n",
        "stemmer.stem(\"flying\")"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'fli'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Er980_PYKNId",
        "outputId": "a25cf9ea-2931-47bb-e3ca-9b7c4815f822"
      },
      "source": [
        "print(stemmer.stem(\"pneumonic\"))\n",
        "print(stemmer.stem(\"pneumonitis\"))"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "pneumon\n",
            "pneumon\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cXckjAH3KlxY"
      },
      "source": [
        "For lemmatization demonstration, we will use the NLTK default lemmatizer which is based on [WordNet](https://wordnet.princeton.edu/). In addition to the target text, the [lemmatizer](https://www.nltk.org/_modules/nltk/stem/wordnet.html) will generally require a part of speech parameter to be fully functional (default is noun)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "3eSFqvCmJwy1",
        "outputId": "f96afeec-aee8-4b92-94de-d12db6c54a3b"
      },
      "source": [
        "lemmatizer = WordNetLemmatizer()\n",
        "print(lemmatizer.lemmatize(\"flown\",pos=\"v\"))"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "error",
          "ename": "LookupError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mLookupError\u001b[0m                               Traceback (most recent call last)",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__load\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     83\u001b[0m                 \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 84\u001b[0;31m                     \u001b[0mroot\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"{self.subdir}/{zip_name}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     85\u001b[0m                 \u001b[0;32mexcept\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/nltk/data.py\u001b[0m in \u001b[0;36mfind\u001b[0;34m(resource_name, paths)\u001b[0m\n\u001b[1;32m    582\u001b[0m     \u001b[0mresource_not_found\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34mf\"\\n{sep}\\n{msg}\\n{sep}\\n\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 583\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_not_found\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    584\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mLookupError\u001b[0m: \n**********************************************************************\n  Resource \u001b[93momw-1.4\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('omw-1.4')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mcorpora/omw-1.4.zip/omw-1.4/\u001b[0m\n\n  Searched in:\n    - '/root/nltk_data'\n    - '/usr/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n**********************************************************************\n",
            "\nDuring handling of the above exception, another exception occurred:\n",
            "\u001b[0;31mLookupError\u001b[0m                               Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-25-2be660779195>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mlemmatizer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mWordNetLemmatizer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlemmatizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlemmatize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"flown\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mpos\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"v\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/nltk/stem/wordnet.py\u001b[0m in \u001b[0;36mlemmatize\u001b[0;34m(self, word, pos)\u001b[0m\n\u001b[1;32m     43\u001b[0m         \u001b[0;34m:\u001b[0m\u001b[0;32mreturn\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mThe\u001b[0m \u001b[0mlemma\u001b[0m \u001b[0mof\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mword\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mgiven\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mpos\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     44\u001b[0m         \"\"\"\n\u001b[0;32m---> 45\u001b[0;31m         \u001b[0mlemmas\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mwn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_morphy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mword\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpos\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     46\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mmin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlemmas\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mlemmas\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0mword\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     47\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, attr)\u001b[0m\n\u001b[1;32m    119\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"LazyCorpusLoader object has no attribute '__bases__'\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    120\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 121\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__load\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    122\u001b[0m         \u001b[0;31m# This looks circular, but its not, since __load() changes our\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    123\u001b[0m         \u001b[0;31m# __class__ to something new:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__load\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     87\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     88\u001b[0m         \u001b[0;31m# Load the corpus.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 89\u001b[0;31m         \u001b[0mcorpus\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__reader_cls\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mroot\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     90\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     91\u001b[0m         \u001b[0;31m# This is where the magic happens!  Transform ourselves into\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/nltk/corpus/reader/wordnet.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, root, omw_reader)\u001b[0m\n\u001b[1;32m   1174\u001b[0m             )\n\u001b[1;32m   1175\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1176\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprovenances\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0momw_prov\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1177\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1178\u001b[0m         \u001b[0;31m# A cache to store the wordnet data of multiple languages\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/nltk/corpus/reader/wordnet.py\u001b[0m in \u001b[0;36momw_prov\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1283\u001b[0m         \u001b[0mprovdict\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1284\u001b[0m         \u001b[0mprovdict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"eng\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1285\u001b[0;31m         \u001b[0mfileids\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_omw_reader\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfileids\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1286\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mfileid\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mfileids\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1287\u001b[0m             \u001b[0mprov\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlangfile\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfileid\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, attr)\u001b[0m\n\u001b[1;32m    119\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"LazyCorpusLoader object has no attribute '__bases__'\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    120\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 121\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__load\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    122\u001b[0m         \u001b[0;31m# This looks circular, but its not, since __load() changes our\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    123\u001b[0m         \u001b[0;31m# __class__ to something new:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__load\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     84\u001b[0m                     \u001b[0mroot\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"{self.subdir}/{zip_name}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     85\u001b[0m                 \u001b[0;32mexcept\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 86\u001b[0;31m                     \u001b[0;32mraise\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     87\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     88\u001b[0m         \u001b[0;31m# Load the corpus.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__load\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     79\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     80\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 81\u001b[0;31m                 \u001b[0mroot\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"{self.subdir}/{self.__name}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     82\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mLookupError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     83\u001b[0m                 \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/nltk/data.py\u001b[0m in \u001b[0;36mfind\u001b[0;34m(resource_name, paths)\u001b[0m\n\u001b[1;32m    581\u001b[0m     \u001b[0msep\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"*\"\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m70\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    582\u001b[0m     \u001b[0mresource_not_found\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34mf\"\\n{sep}\\n{msg}\\n{sep}\\n\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 583\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_not_found\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    584\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    585\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mLookupError\u001b[0m: \n**********************************************************************\n  Resource \u001b[93momw-1.4\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('omw-1.4')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mcorpora/omw-1.4\u001b[0m\n\n  Searched in:\n    - '/root/nltk_data'\n    - '/usr/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n**********************************************************************\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3y_nsmWpHeas"
      },
      "source": [
        "## Negation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LhvODUGKYSa3"
      },
      "source": [
        "Dealing with negation is important especially when using a bag of word model because it ignores word ordering. The presence of a word in a document does not mean we necessarily want it to always be used as the same feature especially if the word is qualified by \"no\" (e.g. no nodule). One way to deal with this is to detect such negated words/phrases and create \"new\" vocab words out of negated entities by appending some text to it (for example append \"NEGEX_\" to negated entities so that these entities are counted as separated words compared to their non-negated counterpart)\n",
        "\n",
        "spacy comes with pre-trained [models](https://github.com/explosion/spacy-models) for purposes of preprocessing (e.g. named entity recognition) and also [negation](https://github.com/jenojp/negspacy). Here we will use a default implementation of negspacy for negation detection."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y_WKbR3ywkXO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "79281f62-50a5-438f-f50e-685d1424ff40"
      },
      "source": [
        "import scispacy\n",
        "import spacy\n",
        "from negspacy.negation import Negex\n",
        "import nltk\n",
        "nltk.download('stopwords')\n",
        "nltk.download('punkt')\n",
        "import string\n",
        "\n",
        "nlp = spacy.load(\"en_core_sci_md\")\n",
        "nlp.add_pipe(\"negex\")"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<negspacy.negation.Negex at 0x7f0fafb82e10>"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kIuHzg8sstac"
      },
      "source": [
        "What kind of phrases is negex recognizing as negation? We can print the default patterns from negspaCy to see. The \"en_clinical\" dataset is used by default. Default termsets also include \"en\" and \"en_clinical_sensitive\". The official documentation covers how to change term sets and also how to add/remove custom phrases to the sets.\n",
        "\n",
        "From the documentation, negspaCy uses four patterns:\n",
        "\n",
        "**pseudo_negations** - phrases that are false triggers, ambiguous negations, or double negatives\n",
        "\n",
        "**preceding_negations** - negation phrases that precede an entity\n",
        "\n",
        "**following_negations** - negation phrases that follow an entity\n",
        "\n",
        "**termination** - phrases that cut a sentence in parts, for purposes of negation detection (.e.g., \"but\")"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zl3g4Djkr2Be",
        "outputId": "f0d81a9c-8b5d-4855-8fa5-fb588d7a5048"
      },
      "source": [
        "from negspacy.termsets import termset\n",
        "\n",
        "ts = termset(\"en_clinical\")\n",
        "ts_dict = ts.get_patterns()\n",
        "\n",
        "for key,value in ts_dict.items():\n",
        "  print(key,value)\n",
        "  print(\"\\n\")"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "pseudo_negations ['no further', 'not able to be', 'not certain if', 'not certain whether', 'not necessarily', 'without any further', 'without difficulty', 'without further', 'might not', 'not only', 'no increase', 'no significant change', 'no change', 'no definite change', 'not extend', 'not cause', 'gram negative', 'not rule out', 'not ruled out', 'not been ruled out', 'not drain', 'no suspicious change', 'no interval change', 'no significant interval change']\n",
            "\n",
            "\n",
            "preceding_negations ['absence of', 'declined', 'denied', 'denies', 'denying', 'no sign of', 'no signs of', 'not', 'not demonstrate', 'symptoms atypical', 'doubt', 'negative for', 'no', 'versus', 'without', \"doesn't\", 'doesnt', \"don't\", 'dont', \"didn't\", 'didnt', \"wasn't\", 'wasnt', \"weren't\", 'werent', \"isn't\", 'isnt', \"aren't\", 'arent', 'cannot', \"can't\", 'cant', \"couldn't\", 'couldnt', 'never', 'patient was not', 'without indication of', 'without sign of', 'without signs of', 'without any reactions or signs of', 'no complaints of', 'no evidence of', 'no cause of', 'evaluate for', 'fails to reveal', 'free of', 'never developed', 'never had', 'did not exhibit', 'rules out', 'rule out', 'rule him out', 'rule her out', 'rule patient out', 'rule the patient out', 'ruled out', 'ruled him out', 'ruled her out', 'ruled patient out', 'ruled the patient out', 'r/o', 'ro']\n",
            "\n",
            "\n",
            "following_negations ['declined', 'unlikely', 'was not', 'were not', \"wasn't\", 'wasnt', \"weren't\", 'werent', 'was ruled out', 'were ruled out', 'free']\n",
            "\n",
            "\n",
            "termination ['although', 'apart from', 'as there are', 'aside from', 'but', 'except', 'however', 'involving', 'nevertheless', 'still', 'though', 'which', 'yet', 'cause for', 'cause of', 'causes for', 'causes of', 'etiology for', 'etiology of', 'origin for', 'origin of', 'origins for', 'origins of', 'other possibilities of', 'reason for', 'reason of', 'reasons for', 'reasons of', 'secondary to', 'source for', 'source of', 'sources for', 'sources of', 'trigger event for']\n",
            "\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AqDHio7AFHw9"
      },
      "source": [
        "## Putting it all together"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0OH7aKTxmylY"
      },
      "source": [
        "from nltk.corpus import stopwords\n",
        "\n",
        "def preprocess(nlp_model,input_text):\n",
        "  input_text = input_text.strip()\n",
        "  doc = nlp_model(input_text)\n",
        "  negation_list = [0]*len(doc)\n",
        "  tokens = list()\n",
        "  stop = set(stopwords.words('english')+list(string.punctuation))\n",
        "  stop.add(\"XXXX\")\n",
        "\n",
        "  for ent in doc.ents:\n",
        "    if ent._.negex:\n",
        "      index = ent.start\n",
        "      while index < ent.end:\n",
        "        negation_list[index] = 1\n",
        "        index += 1\n",
        "        \n",
        "  for i,token in enumerate(doc):\n",
        "    if str(token).lower() not in stop:\n",
        "      if negation_list[i] == 1:\n",
        "        tokens.append((\"NEGEX_\"+str(token).lower()))\n",
        "      else:\n",
        "        tokens.append(str(token).lower())\n",
        "\n",
        "  return tokens\n"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bggt3sfZeYJg"
      },
      "source": [
        "## Creating a vocabulary\n",
        "After preprocessing the text, it is time to scan through the entire training corpus (aka all of the text available) and create a vocabulary which is essentially an index of all tokens encountered.\n",
        "\n",
        "We will use the gensim [Dictionary](https://radimrehurek.com/gensim/corpora/dictionary.html) class for this as it has some useful methods.\n",
        "\n",
        "We can also set a threshold to filter out tokens using the gensim dict method filter_extremes() that occur below a specified frequency. This could be useful for removing misspelled words."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2LqwYpEjXDjs",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "29a14f63-4432-41d5-9b76-d15194d1e5d4"
      },
      "source": [
        "from gensim.corpora import Dictionary\n",
        "\n",
        "train_text = train_df[\"full-text\"].to_list()\n",
        "\n",
        "def token_generator(text_list):\n",
        "  for text in text_list:\n",
        "    yield preprocess(nlp,text)\n",
        "\n",
        "train_tokens = token_generator(train_text)\n",
        "train_vocab_dict = Dictionary(train_tokens)\n",
        "\n",
        "# gensim Dictionary does not create an id2token dictionary  until dictionary is called\n",
        "train_vocab_dict[0]"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'NEGEX_abnormalities'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SKHBh7yKcw0N",
        "outputId": "15087c0e-5d6f-42dc-c14e-86fdf2fa4658"
      },
      "source": [
        "len(train_vocab_dict.token2id)"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2339"
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V2tEqwONfclV"
      },
      "source": [
        "We will also need to process the test set for later use. Note we don't create a dictionary from the test tokens as it is may contain \"out of vocabulary words\" that would not have been seen in training. We need to simulate how our model pipeline behaves when it encounters unseen text/words."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0FSV_cs2fU9d"
      },
      "source": [
        "test_text = test_df[\"full-text\"].to_list()\n",
        "test_tokens = token_generator(test_text)"
      ],
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nx0MthQ91wAh"
      },
      "source": [
        "# Classification model - introducing Naive Bayes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZS3gRR_Y2Cd8"
      },
      "source": [
        "## Theory"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V_zv9jPL3SW7"
      },
      "source": [
        "Naive Bayes (NB) derives its name from Bayes' theorem.\n",
        "\n",
        "Our objective is to classifiy a document d with the best possible class prediction, $\\hat{c}$, where $c$ is the true class of document $d$\n",
        "\n",
        "$ \\hat{c} = \\underset{c \\in C}{argmax} \\: P(c|d) = \\underset{c \\in C}{argmax} \\frac{P(d|c)P(c)}{P(d)} $\n",
        "\n",
        "$P(d)$ is a constant so we can just find the class c that maximizes the numerator of the expression\n",
        "\n",
        "$ \\hat{c} = \\underset{c \\in C}{argmax} \\: P(d|c)P(c) $\n",
        "\n",
        "$P(c)$ is readily determined - it is the frequency of class c in the training set.\n",
        "\n",
        "For $P(d|c)$, we let the words (tokens) of document d be the features that define it. We assume the order of these features does not matter (bag of words) and that the probability of a feature occurring in document d is independent of other words already inside of document d (conditional independence between every pair of tokens).\n",
        "\n",
        "Based on this $P(d|c) = P(x_{1}, x_{2},...,x_{n}| c)$ where $x_i$ are the tokens x in d. If we assume independence among these words then this probability further breaks down into:\n",
        "\n",
        "$\\prod_{i = 1}^{n}{P(x_{i}|c)}$\n",
        "\n",
        "The different variations of Naive Bayes come from the different assumptions of what we assume the probability distribution of $P(x_{i}|c)$ looks like. \n",
        "\n",
        "If we assume a multinomial distribution, then we can estimate the probability $P(x_{i}|c)$ as $\\theta_{c,i}$ where the maximum likelihood estimator for theta for a multinomial is:\n",
        "\n",
        "$\\theta_{c,i} = \\frac{N_{c,i}}{N_{c}}$\n",
        "\n",
        "where $N_{c,i}$ is the total count of the token i occurring in class c and $N_{c}$ is the number of total tokens in class c. In practice, we use an additional smoothing term to this estimator by giving an extra count of $\\alpha$ to all words. This is to account for cases in which a token does not appear in class c to avoid a 0 in our product.\n",
        "\n",
        "$\\theta_{c,i} = \\frac{N_{c,i}+\\alpha}{N_{c}+\\alpha n}$ where n is the number of unique tokens\n",
        "\n",
        "See [here](https://scikit-learn.org/stable/modules/naive_bayes.html) for more information"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "STdw1x_E16bC"
      },
      "source": [
        "## Code"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zW90rq3WeHj5"
      },
      "source": [
        "y_train = list(map(int,(train_df['label'] != \"normal\").to_list()))\n",
        "y_test = list(map(int,(test_df['label'] != \"normal\").to_list()))"
      ],
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C2QWpczbsgpU"
      },
      "source": [
        "The sklearn naive bayes implementation does not take compressed representations of a sparse matrix as input, so we will have to create a sparse vector representation of our bag of words format"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zQ0q-IfaesbM"
      },
      "source": [
        "def sparse_vector_create(tuple_list,vocab_len):\n",
        "#tuple_list will have data structure akin to gensim dictionary doc2bow output \n",
        "  sparse_vector = np.zeros(vocab_len)\n",
        "  for id,freq in tuple_list:\n",
        "    sparse_vector[id] = freq\n",
        "  return sparse_vector\n",
        "\n",
        "def sparse_vector_generator(tokens,vocab_dict,vocab_len):\n",
        "  for token in tokens:\n",
        "    yield sparse_vector_create(vocab_dict.doc2bow(token),vocab_len)\n",
        "\n",
        "# recreate generator object to reset it , otherwise will output empty result\n",
        "train_tokens = token_generator(train_text)\n",
        "x_train_sparse = [sparse_vector for sparse_vector in sparse_vector_generator(train_tokens,train_vocab_dict,len(train_vocab_dict))]\n"
      ],
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R2wfh_v1dxA_",
        "outputId": "0efb4453-501d-402f-cb80-317849796034"
      },
      "source": [
        "from sklearn import naive_bayes\n",
        "\n",
        "nb_classifier = naive_bayes.MultinomialNB(alpha=1.0)\n",
        "\n",
        "nb_classifier.fit(x_train_sparse,y_train)"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "MultinomialNB()"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1rYR3aLUs0Vk"
      },
      "source": [
        "Let's try running our model on the test set. The test set may have words that are not included in the training vocabulary, so we will need 1 additional preprocessing step to handle such out-of-vocabulary words. The simplest approach is to simply ignore such words. The gensim dictionary doc2bow function automatically ignores out of vocabulary words.\n",
        "\n",
        "Other tokenization techniques (especially sub-word or character level) may avoid this issue all together by ensuring any string of text encontered is able to be tokenized and thus included into the vocabulary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tL5JHCiVh8E_",
        "outputId": "021615c9-ea76-436d-f688-b7da15f368cc"
      },
      "source": [
        "# doc2bow ignores out of vocabulary words\n",
        "train_vocab_dict.doc2bow([\"madeupword\"])"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[]"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZUywn0YpszSh"
      },
      "source": [
        "# basically the same code as the training version above but we're not storing each iteration of the for loop into an array\n",
        "\n",
        "test_tokens = token_generator(test_text)\n",
        "nb_predictions = list()\n",
        "\n",
        "for token in test_tokens:\n",
        "  test_sparse_vector = sparse_vector_create(train_vocab_dict.doc2bow(token),len(train_vocab_dict))\n",
        "  nb_predictions.append(nb_classifier.predict(test_sparse_vector.reshape(1,-1))[0])\n",
        " "
      ],
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J16it1OP_ydm"
      },
      "source": [
        "\\# More on sparse word vectors - term frequency inverse document frequency (tf-idf)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JfjWcj7SAnJR"
      },
      "source": [
        "## Theory"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tn0jAUfHW5ZU"
      },
      "source": [
        "### tfidf\n",
        "\n",
        "Earlier we created vectors of raw term counts with respect to each document. These were called sparse vectors because they were largely 0 as many unique words tend to not show up in may documents and thus the counts of those words was 0 for most documents - represented as a 0 in the term vector in the corresponding index for that word.\n",
        "\n",
        "There is a related concept called term frequency-inverse document frequency. We start with related quantity to term count: the term frequency. For a given document d and term t, the term frequency(tf) is is $\\frac{f_{t,d}}{\\sum_{t' \\in d}{f_{t',d}}}$ where the denominator is just the number of words/terms in the document. Sometimes we prefer using the log of the tf term as importance of a term does not increase linearly but rather \"peaks\" after a certain point.\n",
        "\n",
        "We then multiply this \"tf\" by another term - the inverse document frequency (idf) which is usually $log{\\frac{N}{n_t}}$ where N is number of total documents and $n_t$ is number of documents that contain the term t.\n",
        "\n",
        "For an unseen document, the tf is readily calculated. The idf for the terms in the document however uses the weighting learned from the training data.\n",
        "\n",
        "More on sklearn [tfidf](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NmC0HOIUeHL8"
      },
      "source": [
        "### Support Vector Machine (SVM)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SAcgfh9Dcqjk"
      },
      "source": [
        "The theory behind SVM is outside of the scope of this tutorial. However intuitively SVM classifiers classify data by finding a \"hyper-plane\" that separates two classes of data. One can think of a hyper plane as the multidimensional generalization of a line which the equivalent of a hyperplane for 2D space, or a plane which is the hyperplane for 3D space. The hyperplane is basically a linear \"plane\" in N dimensional space that separates the space into two sections (hence the plane being N-1 dimensional). The optimization objective for SVM is set such that it will choose a hyperplane by finding the plane that has the maximum \"margin\" between the two classes i.e. the largest distance between the plane and the closest data point of each class."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ECqSWs6q_0oV"
      },
      "source": [
        "## Code"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LJkXCj8t52UZ"
      },
      "source": [
        "The sklearn library readily comes with implementation of [tfidf](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html) and [SVM](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) which we will use here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QgD87-CM7TQ9"
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfTransformer\n",
        "tfidf = TfidfTransformer(use_idf=True, smooth_idf = True,sublinear_tf = True)\n",
        "tfidf.fit(x_train_sparse)\n",
        "x_train_tfidf = tfidf.transform(x_train_sparse)"
      ],
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(sum(y_train))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gmemzruMih41",
        "outputId": "56b4cf65-652b-46d9-8e0d-e75ee30b20fb"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2098\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install nlpaug"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UBmgN4oQCQZC",
        "outputId": "1b06d08f-756f-4925-ce87-0a7137a96f8f"
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting nlpaug\n",
            "  Downloading nlpaug-1.1.11-py3-none-any.whl (410 kB)\n",
            "\u001b[K     |████████████████████████████████| 410 kB 18.0 MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.16.2 in /usr/local/lib/python3.7/dist-packages (from nlpaug) (1.21.6)\n",
            "Requirement already satisfied: gdown>=4.0.0 in /usr/local/lib/python3.7/dist-packages (from nlpaug) (4.4.0)\n",
            "Requirement already satisfied: pandas>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from nlpaug) (1.3.5)\n",
            "Requirement already satisfied: requests>=2.22.0 in /usr/local/lib/python3.7/dist-packages (from nlpaug) (2.23.0)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from gdown>=4.0.0->nlpaug) (1.15.0)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.7/dist-packages (from gdown>=4.0.0->nlpaug) (4.6.3)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from gdown>=4.0.0->nlpaug) (4.64.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from gdown>=4.0.0->nlpaug) (3.8.0)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.2.0->nlpaug) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.2.0->nlpaug) (2022.6)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.22.0->nlpaug) (2022.9.24)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.22.0->nlpaug) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.22.0->nlpaug) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.22.0->nlpaug) (2.10)\n",
            "Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /usr/local/lib/python3.7/dist-packages (from requests>=2.22.0->nlpaug) (1.7.1)\n",
            "Installing collected packages: nlpaug\n",
            "Successfully installed nlpaug-1.1.11\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import nlpaug.augmenter.word as naw\n",
        "\n",
        "#Data Augmentation: \n",
        "#Approach #1: Trimming out Abnormal Cases\n",
        "indices_1 = []\n",
        "abnormal = 1263\n",
        "normal = 1263\n",
        "for i in range(3361):\n",
        "  if y_train[i] == 1 and abnormal >= 1:\n",
        "    #Abnormal\n",
        "    abnormal -= 1\n",
        "    indices_1.append(i)\n",
        "  elif y_train[i] == 0 and normal >= 1:\n",
        "    normal -= 1\n",
        "    indices_1.append(i)\n",
        "\n",
        "x_train_tfidf_1 = x_train_tfidf[indices_1, :]\n",
        "y_train_1 = [y_train[i] for i in indices_1]\n",
        "\n",
        "#Approach #2: Repeating Beginning Normal Cases\n",
        "indices_2 = [i for i in range(3361)]\n",
        "normal_indices = []\n",
        "for i in range(3361):\n",
        "  if y_train[i] == 0:\n",
        "    normal_indices.append(i)\n",
        "\n",
        "indices_2.extend(normal_indices[0:834])\n",
        "\n",
        "x_train_tfidf_2 = x_train_tfidf[indices_2, :]\n",
        "y_train_2 = [y_train[i] for i in indices_2]\n",
        "\n",
        "#Approach #3: Masking \n",
        "aug = naw.RandomWordAug()\n",
        "train_text_del = aug.augment(train_text)\n",
        "\n",
        "#New Library based on the new deleted text\n",
        "train_tokens_del = token_generator(train_text_del)\n",
        "train_vocab_dict_del = Dictionary(train_tokens_del)\n",
        "\n",
        "train_vocab_dict_del[0]\n",
        "\n",
        "train_tokens_del = token_generator(train_text_del)\n",
        "x_train_sparse_del = [sparse_vector for sparse_vector in sparse_vector_generator(train_tokens_del, train_vocab_dict_del ,len(train_vocab_dict_del))]\n",
        "\n",
        "#Creating New tfidf transformer\n",
        "tfidf_del = TfidfTransformer(use_idf=True, smooth_idf = True,sublinear_tf = True)\n",
        "tfidf_del.fit(x_train_sparse_del)\n",
        "tfidf_del.fit(x_train_sparse_del)\n",
        "x_train_tfidf_del = tfidf_del.transform(x_train_sparse_del)\n",
        "\n",
        "# Getting the test_text into the new tfidf\n",
        "test_tokens_del = token_generator(test_text)\n",
        "\n",
        "for token in test_tokens_del:\n",
        "  test_sparse_vector_del = sparse_vector_create(train_vocab_dict_del.doc2bow(token),len(train_vocab_dict_del))"
      ],
      "metadata": {
        "id": "HK0suZNwFAHw"
      },
      "execution_count": 44,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn import svm\n",
        "import sklearn.model_selection\n",
        "import sklearn.svm\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def cross_validation_f(data_train, target_train, parameters, folds):\n",
        "    kf = sklearn.model_selection.KFold(n_splits=folds)\n",
        "\n",
        "    average_training_sens = np.zeros(len(parameters))\n",
        "    average_validation_sens = np.zeros(len(parameters))\n",
        "    average_training_spec = np.zeros(len(parameters))\n",
        "    average_validation_spec = np.zeros(len(parameters))\n",
        "    \n",
        "    for i in range(len(parameters)):\n",
        "        parameter = parameters[i]\n",
        "        \n",
        "        training_sensitivity, training_specificity = [], []\n",
        "        validation_sentivity, validation_specificity = [], []\n",
        "\n",
        "        for train_index, test_index in kf.split(data_train):\n",
        "            #print(train_index.shape)\n",
        "            data_fold_train = data_train[train_index,:]\n",
        "            data_fold_test = [target_train[i] for i in train_index] #target_train[train_index]\n",
        "            validation_train, validation_test = data_train[test_index, :], [target_train[i] for i in test_index]#target_train[test_index]\n",
        "\n",
        "            clf = svm.SVC(C=parameter, kernel ='poly')\n",
        "            train = clf.fit(data_fold_train, data_fold_test)\n",
        "            predicted_train = train.predict(data_fold_train)\n",
        "            predicted_validation = train.predict(validation_train)\n",
        "            \n",
        "            #print(\"Parameter: \", parameter, \" Train Data Metrics:\")\n",
        "            nb_metric = metric_calc(data_fold_test, predicted_train)\n",
        "            training_sens = nb_metric.sens()\n",
        "            training_spec = nb_metric.spec()\n",
        "\n",
        "            #print(\"Parameter: \", parameter, \" Train Data Metrics:\")\n",
        "            nb_metric = metric_calc(validation_test, predicted_validation)\n",
        "            validation_sens = nb_metric.sens()\n",
        "            validation_spec = nb_metric.spec()\n",
        "\n",
        "            training_sensitivity.append(training_sens)\n",
        "            validation_sentivity.append(validation_sens)\n",
        "            training_specificity.append(training_spec)\n",
        "            validation_specificity.append(validation_spec)\n",
        "        \n",
        "        average_training_sens[i] = np.mean(np.array(training_sensitivity))\n",
        "        average_validation_sens[i] = np.mean(np.array(validation_sentivity))   \n",
        "        average_training_spec[i] = np.mean(np.array(training_specificity))\n",
        "        average_validation_spec[i] = np.mean(np.array(validation_specificity))           \n",
        "    return average_training_sens, average_validation_spec, average_training_spec, average_training_sens\n",
        "\n",
        "poly_parameters = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]\n",
        "training_sens, validation_sens, training_spec, validation_spec = cross_validation_f(x_train_tfidf, y_train, poly_parameters, 10)\n",
        "x = np.array(poly_parameters)\n",
        "\n",
        "plt.scatter(x, training_sens)\n",
        "plt.scatter(x, validation_sens)\n",
        "\n",
        "plt.legend([\"Training\" , \"Validation\"])\n",
        "plt.xlabel('Degree of Polynomial') \n",
        "plt.ylabel('Sensitivity')\n",
        "plt.show()\n",
        "\n",
        "plt.scatter(x, training_spec)\n",
        "plt.scatter(x, validation_spec)\n",
        "\n",
        "plt.legend([\"Training\" , \"Validation\"])\n",
        "plt.xlabel('Degree of Polynomial') \n",
        "plt.ylabel('Specificity')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "5CNL7MMPyi-R",
        "outputId": "8885d977-916c-4ca5-fec8-e971e5f7121e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 543
        }
      },
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxU1Z338c/XBoQIYgTiKK2CiYJElJYWFzSiJK4JRFwimgzEzGicGGcw6qNZhBc+TDKDk8VonMfEPY5I1PDgSIYorqNmpAFpRETRkNjg0mJAjCiLv/nj3oaiuHRXQ1dXdff3/Xr1q26du/2q0Purc8695ygiMDMzy7dLqQMwM7Py5ARhZmaZnCDMzCyTE4SZmWVygjAzs0ydSh1AS+ndu3f069ev1GGYmbUp8+bNeyci+mStazcJol+/ftTU1JQ6DDOzNkXSn7a3zk1MZmaWyQnCzMwyOUGYmVkmJwgzM8tUtAQh6VZJb0t6YTvrJel6Scsk1Uo6PGfdOEmvpH/jihWjmZltXzHvYroduAG4czvrTwUOTP+OBG4CjpS0JzARqAYCmCdpZkT8pYixltSMBSuYOnspK1evY589unHFyQP4clVfx+E4HEcZxtCR4ihagoiIJyX1a2ST0cCdkQwn+wdJe0jaGxgBPBwR7wJIehg4BbinGHGW+h96xoIVXP3AItZt2ATAitXruPqBRQCOw3E4jjKLoaPFUco+iL7A6znv69Ky7ZVvQ9KFkmok1dTX1zc7gIYveMXqdQRbvuAZC1Y0+1g7aurspZv/gRus27CJqbOXtloMjsNxtIU4yiGGjhZHm+6kjoibI6I6Iqr79Ml8ELBR5fAPvXL1umaVOw7H0VHjKIcYOlocpUwQK4B9c95XpmXbK29x5fAPvc8e3ZpV7jgcR0eNoxxi6GhxlDJBzAT+Nr2b6ShgTUS8AcwGTpL0SUmfBE5Ky1pcOfxDX3HyALp1rtiqrFvnCq44eUCrxeA4HEdbiKMcYuhocRStk1rSPSQdzr0l1ZHcmdQZICL+HZgFnAYsAz4Avp6ue1fStcDc9FCTGzqsW9oVJw/YqpMHWv8fuqEzqdR3RDgOx1HucZRDDB0tDrWXOamrq6tjRwbrK/VdTGZmpSRpXkRUZ61rN6O57qgvV/V1QjAzy9Cm72IyM7PicYIwM7NMThBmZpbJCcLMzDI5QZiZWSYnCDMzy+QEYWZmmZwgzMwskxOEmZllcoIwM7NMThBmZpbJCcLMzDI5QZiZWSYnCDMzy+QEYWZmmZwgzMwskxOEmZllcoIwM7NMRU0Qkk6RtFTSMklXZazfX9IcSbWSHpdUmbPuXyUtlrRE0vWSVMxYzcxsa0VLEJIqgBuBU4FBwFhJg/I2uw64MyIOBSYDP0z3PQYYDhwKHAIcARxfrFjNzGxbxaxBDAOWRcRrEbEemAaMzttmEPBouvxYzvoAugJdgF2BzsBbRYzVzMzyFDNB9AVez3lfl5blWgiMSZfPAHpI6hURz5IkjDfSv9kRsaSIsZqZWZ5Sd1JfDhwvaQFJE9IKYJOkzwAHA5UkSeVEScfl7yzpQkk1kmrq6+tbM24zs3avmAliBbBvzvvKtGyziFgZEWMiogr4Xlq2mqQ28YeIeD8i3gd+Bxydf4KIuDkiqiOiuk+fPsX6HGZmHVIxE8Rc4EBJ/SV1Ac4FZuZuIKm3pIYYrgZuTZf/TFKz6CSpM0ntwk1MZmatqGgJIiI2ApcAs0ku7tMjYrGkyZJGpZuNAJZKehnYC5iSlt8HvAosIumnWBgRDxYrVjMz25YiotQxtIjq6uqoqakpdRhmZm2KpHkRUZ21rtSd1GZmVqacIMzMLJMThJmZZXKCMDOzTE4QZmaWyQnCzMwyOUGYmVkmJwgzM8vkBGFmZpmcIMzMLJMThJmZZXKCMDOzTE4QZmaWyQnCzMwyOUGYmVkmJwgzM8vkBGFmZpmcIMzMLJMThJmZZXKCMDOzTEVNEJJOkbRU0jJJV2Ws31/SHEm1kh6XVJmzbj9Jv5e0RNKLkvoVM1YzM9ta0RKEpArgRuBUYBAwVtKgvM2uA+6MiEOBycAPc9bdCUyNiIOBYcDbxYrVzMy2VcwaxDBgWUS8FhHrgWnA6LxtBgGPpsuPNaxPE0mniHgYICLej4gPihirmZnlKWaC6Au8nvO+Li3LtRAYky6fAfSQ1As4CFgt6QFJCyRNTWskW5F0oaQaSTX19fVF+AhmZh1XqTupLweOl7QAOB5YAWwCOgHHpeuPAA4AxufvHBE3R0R1RFT36dOn1YI2M+sIipkgVgD75ryvTMs2i4iVETEmIqqA76Vlq0lqG8+nzVMbgRnA4UWM1czM8hQzQcwFDpTUX1IX4FxgZu4GknpLaojhauDWnH33kNRQLTgReLGIsZqZWZ6iJYj0l/8lwGxgCTA9IhZLmixpVLrZCGCppJeBvYAp6b6bSJqX5khaBAj4ZbFiNTOzbSkiSh1Di6iuro6amppSh2Fm1qZImhcR1VnrSt1JbWZmZcoJwszMMjlBmJlZJicIMzPL5ARhZmaZnCDMzCyTE4SZmWVygjAzs0xOEGZmlskJwszMMjlBmJlZJicIMzPL5ARhZmaZnCDMzCxTQQkinRv69JzJfczMrJ0r9IL/C+A84BVJP5I0oIgxmZlZGSgoQUTEIxFxPsm80MuBRyQ9I+nrkjoXM0AzMyuNgpuMJPUCxgN/BywAfkaSMB4uSmRmZlZSnQrZSNJvgQHAXcCXIuKNdNW9kjzPp5lZO1RQggB+GRGzcgsk7RoRH21vLlMzM2vbCm1i+r8ZZc82tZOkUyQtlbRM0lUZ6/eXNEdSraTHJVXmrd9dUp2kGwqM08zMWkijNQhJfwP0BbpJqgKUrtod+EQT+1YANwJfAOqAuZJmRsSLOZtdB9wZEXdIOhH4IfC1nPXXAk824/OYWTuxYcMG6urq+PDDD0sdSrvQtWtXKisr6dy58PuKmmpiOpmkY7oS+HFO+Vrgu03sOwxYFhGvAUiaBowGchPEIOCydPkxYEbDCklDgb2A/wLcjGXWwdTV1dGjRw/69euHpKZ3sO2KCFatWkVdXR39+/cveL9Gm5gi4o6IOAEYHxEn5PyNiogHmjh2X+D1nPd1aVmuhcCYdPkMoIekXukDef8GXN7YCSRdKKlGUk19fX0T4ZhZW/Lhhx/Sq1cvJ4cWIIlevXo1uzbWVBPTVyPi10A/SZflr4+IH2fs1hyXAzdIGk/SlLQC2AT8AzArIuoa+48jIm4Gbgaorq6OnYzFzMqMk0PL2ZHvsqkmpt3S1+7NPnJysd83531lWrZZRKwkrUFI6g6cGRGrJR0NHCfpH9Jzd5H0fkRs09FtZlYMq1atYuTIkQC8+eabVFRU0KdPHwCee+45unTpst19a2pquPPOO7n++usbPccxxxzDM88803JBtzBFNP3DW1KfiGhWG46kTsDLwEiSxDAXOC8iFuds0xt4NyI+ljQF2BQR1+QdZzxQHRGXNHa+6urqqKnxIxlm7cWSJUs4+OCDSx0GAJMmTaJ79+5cfvmWVu+NGzfSqVOhTwqUh6zvVNK87T2uUOhtrk9L+r2kb0j6ZCE7RMRG4BJgNrAEmB4RiyVNljQq3WwEsFTSyyQd0lMKjMfMbCszFqxg+I8epf9VDzH8R48yY8GKpndqpvHjx/PNb36TI488kiuvvJLnnnuOo48+mqqqKo455hiWLl0KwOOPP84Xv/hFIEkuF1xwASNGjOCAAw7YqlbRvXv3zduPGDGCs846i4EDB3L++efT8ON91qxZDBw4kKFDh3LppZduPm5rKCj9RcRBkoYB5wLfk/QiMC3tn2hsv1nArLyya3KW7wPua+IYtwO3FxKnmXVMMxas4OoHFrFuwyYAVqxex9UPLALgy1X598bsnLq6Op555hkqKip47733eOqpp+jUqROPPPII3/3ud7n//vu32eell17iscceY+3atQwYMICLL754m9tNFyxYwOLFi9lnn30YPnw4Tz/9NNXV1Vx00UU8+eST9O/fn7Fjx7boZ2lKwWMxRcRzEXEZye2r7wJ3FC0qM7NmmDp76ebk0GDdhk1Mnb20xc919tlnU1FRAcCaNWs4++yzOeSQQ5gwYQKLFy/O3Of0009n1113pXfv3nzqU5/irbfe2mabYcOGUVlZyS677MKQIUNYvnw5L730EgcccMDmW1PLMkGkTzSPk/Q74BngDZJEYWZWcitXr2tW+c7YbbfdNi//4Ac/4IQTTuCFF17gwQcf3O5tpLvuuuvm5YqKCjZu3LhD27S2QmsQC4EhwOSIOCgi/k9EzCtiXGZmBdtnj27NKm8pa9asoW/fpAnr9ttvb/HjDxgwgNdee43ly5cDcO+997b4ORpTaII4ICImREST4y+ZmbW2K04eQLfOFVuVdetcwRUnF3dusyuvvJKrr76aqqqqovzi79atG7/4xS845ZRTGDp0KD169KBnz54tfp7tafQ2V0k/jYh/kvQgsM2GETEqY7eS8G2uZu1Lc29znbFgBVNnL2Xl6nXss0c3rjh5QIt3UJfC+++/T/fu3YkIvvWtb3HggQcyYcKEHTpWc29zbeouprvS1+t2KBozs1by5aq+7SIh5PvlL3/JHXfcwfr166mqquKiiy5qtXM3miBy+hmGRMTPctdJ+kfgiWIFZmZmMGHChB2uMeysQvsgxmWUjW/BOMzMrMw0NVjfWOA8oL+kmTmrepA8C2FmZu1UU30QDc889CYZfrvBWqC2WEGZmVnpNdUH8SfgT8DRrROOmZmVi0b7ICT9d/q6VtJ7OX9rJb3XOiGambW+E044gdmzZ29V9tOf/pSLL744c/sRI0bQcKv9aaedxurVq7fZZtKkSVx3XeM3hc6YMYMXX9wy8eY111zDI4880tzwW0RTM8odm772iIjdc/56RMTurROimVnrGzt2LNOmTduqbNq0aQWNhzRr1iz22GOPHTpvfoKYPHkyn//853foWDur0LGYPi1p13R5hKRLJe3YpzczK4ba6fCTQ2DSHslr7fSdOtxZZ53FQw89xPr16wFYvnw5K1eu5J577qG6uprPfvazTJw4MXPffv368c477wAwZcoUDjroII499tjNw4FD8nzDEUccwWGHHcaZZ57JBx98wDPPPMPMmTO54oorGDJkCK+++irjx4/nvvuSQa/nzJlDVVUVgwcP5oILLuCjjz7afL6JEydy+OGHM3jwYF566aWd+uwNCr3N9X5gk6TPkEzxuS/wHy0SgZnZzqqdDg9eCmteByJ5ffDSnUoSe+65J8OGDeN3v/sdkNQezjnnHKZMmUJNTQ21tbU88cQT1NZu/36defPmMW3aNJ5//nlmzZrF3LlzN68bM2YMc+fOZeHChRx88MHccsstHHPMMYwaNYqpU6fy/PPP8+lPf3rz9h9++CHjx4/n3nvvZdGiRWzcuJGbbrpp8/revXszf/58Lr744iabsQpVaIL4OJ0A6Azg5xFxBbB3i0RgZraz5kyGDXkjt25Yl5TvhNxmpobmpenTp3P44YdTVVXF4sWLt2oOyvfUU09xxhln8IlPfILdd9+dUaO2jE70wgsvcNxxxzF48GDuvvvu7Q4V3mDp0qX079+fgw46CIBx48bx5JNPbl4/ZswYAIYOHbp5cL+dVWiC2JA+EzEO+M+0rHMj25uZtZ41dc0rL9Do0aOZM2cO8+fP54MPPmDPPffkuuuuY86cOdTW1nL66advd4jvpowfP54bbriBRYsWMXHixB0+ToOG4cJbcqjwQhPE10ludZ0SEX+U1J8t4zSZmZVWz8rmlReoe/funHDCCVxwwQWMHTuW9957j912242ePXvy1ltvbW5+2p7Pfe5zzJgxg3Xr1rF27VoefPDBzevWrl3L3nvvzYYNG7j77rs3l/fo0YO1a9duc6wBAwawfPlyli1bBsBdd93F8ccfv1OfrykFJYiIeDEiLo2Ie9L3f4yIfylqZGZmhRp5DXTOm/uhc7ekfCeNHTuWhQsXMnbsWA477DCqqqoYOHAg5513HsOHD29038MPP5yvfOUrHHbYYZx66qkcccQRm9dde+21HHnkkQwfPpyBAwduLj/33HOZOnUqVVVVvPrqq5vLu3btym233cbZZ5/N4MGD2WWXXfjmN7+505+vMY0O9715I2k4MAnYn+ThOgEREQcUNbpm8HDfZu1Lc4f7pnZ60uewpi6pOYy8Bg49p3gBtkEtPdx3g1uACcA8YFMT2+ae+BTgZ0AF8KuI+FHe+v2BW4E+JGM7fTUi6iQNAW4Cdk/PNyUiWncqJTNrWw49xwmhhRWaINZEROONbXkkVQA3Al8A6oC5kmZGRG6X/3XAnRFxh6QTgR8CXwM+AP42Il6RtA8wT9LsiNj20UQzMyuKQhPEY5KmAg8AHzUURsT8RvYZBiyLiNcAJE0DRgO5CWIQcFnDOYAZ6XFfzjnHSklvk9QynCDMzFpJoQniyPQ1t50qgBMb2acv8HrO+7qc4zRYCIwhaYY6A+ghqVdErGrYQNIwoAvwat6+SLoQuBBgv/32K+iDmFnbERFIKnUY7UIh/c35CkoQEXFCs49cmMuBGySNB54EVpDTxyFpb5LbacdFxMcZcd1M8mQ31dXVzf/0Zla2unbtyqpVq+jVq5eTxE6KCFatWkXXrl2btV9BCULSXsA/A/tExKmSBgFHR8Qtjey2gmRIjgaVaVlu0CtJahBI6g6c2dDPIGl34CHgexHxhwI/j5m1E5WVldTV1VFfX1/qUNqFrl27UlnZvOdCCm1iuh24Dfhe+v5l4F6Su5u2Zy5wYPpQ3QrgXJLZ6TaT1Bt4N60dXE1yRxOSugC/JenAvq/AGM2sHencuTP9+/cvdRgdWqFPUveOiOnAxwDpuEyN3u6abnMJMBtYAkyPiMWSJktqGJBkBLBU0svAXsCUtPwc4HPAeEnPp39DmvG5zMxsJxVag/irpF4kHdNIOgpY09ROETELmJVXdk3O8n3ANjWEiPg18OsCYzMzsyIoNEFcBswEPi3paZJbTs8qWlRmZlZyTU05eoSkv0mfdzge+C7JcxC/J7lt1czM2qmm+iD+H7A+XT6GpJP6RuAvpLeXmplZ+9RUE1NFRLybLn8FuDki7gful/R8cUMzM7NSaqoGUSGpIYmMBB7NWVdo/4WZmbVBTV3k7wGekPQOsA54CiCdm7rJu5jMzKztajRBRMQUSXNI5p/+fWwZzGMX4NvFDs7MzEqnyWairGEuckdbNTOz9qnQJ6nNzKyDcYIwM7NMThBmZpbJCcLMzDI5QZiZWSYnCDMzy+QEYWZmmZwgzMwskxOEmZllcoIwM7NMThBmZpbJCcLMzDIVNUFIOkXSUknLJF2VsX5/SXMk1Up6XFJlzrpxkl5J/8YVM04zM9tW0RKEpAqS6UlPBQYBYyUNytvsOuDOiDgUmAz8MN13T2AicCQwDJgo6ZPFitXMzLZVzBrEMGBZRLwWEeuBacDovG0GsWWWusdy1p8MPBwR70bEX4CHgVOKGKuZmeUpZoLoC7ye874uLcu1EBiTLp8B9JDUq8B9kXShpBpJNfX19S0WuJmZlb6T+nLgeEkLgOOBFcCmQneOiJsjojoiqvv06VOsGM3MOqQmZ5TbCSuAfXPeV6Zlm0XEStIahKTuwJkRsVrSCmBE3r6PFzFWMzPLU8waxFzgQEn9JXUBzgVm5m4gqbekhhiuBm5Nl2cDJ0n6ZNo5fVJaZmZmraRoCSIiNgKXkFzYlwDTI2KxpMmSRqWbjQCWSnoZ2AuYku77LnAtSZKZC0xOy8zMrJUoIkodQ4uorq6OmpqaUodhZtamSJoXEdVZ60rdSW1mZmXKCcLMzDI5QZiZWSYnCDMzy+QEYWZmmZwgzMwskxOEmZllcoIwM7NMThBmZpbJCcLMzDI5QZiZWSYnCDMzy+QEYWZmmZwgzMwskxOEmZllcoIwM7NMThBmZpbJCcLMzDI5QZiZWaaiJghJp0haKmmZpKsy1u8n6TFJCyTVSjotLe8s6Q5JiyQtkXR1MeM0M7NtFS1BSKoAbgROBQYBYyUNytvs+8D0iKgCzgV+kZafDewaEYOBocBFkvoVK1YzM9tWMWsQw4BlEfFaRKwHpgGj87YJYPd0uSewMqd8N0mdgG7AeuC9IsZqZmZ5ipkg+gKv57yvS8tyTQK+KqkOmAV8Oy2/D/gr8AbwZ+C6iHi3iLGambU9tdPhJ4fApD2S19rpLXr4UndSjwVuj4hK4DTgLkm7kNQ+NgH7AP2B70g6IH9nSRdKqpFUU19f35pxm1lHVuQLc8ExPHgprHkdiOT1wUtbNJZiJogVwL457yvTslzfAKYDRMSzQFegN3Ae8F8RsSEi3gaeBqrzTxARN0dEdURU9+nTpwgfwczKSge5MBdkzmTYsG7rsg3rkvIWUswEMRc4UFJ/SV1IOqFn5m3zZ2AkgKSDSRJEfVp+Ylq+G3AU8FIRYzWzcteBLswFWVPXvPIdULQEEREbgUuA2cASkruVFkuaLGlUutl3gL+XtBC4BxgfEUFy91N3SYtJEs1tEVFbrFjNrA3oQBfmgvSsbF75DujUYkfKEBGzSDqfc8uuyVl+ERiesd/7JLe6mpklyunCvOb17PLWNPKapAaVmzQ7d0vKW0ipO6nNrCnl0O5eDnG0wi/mgoy8JrkQ52rhC3NBDj0HvnQ99NwXUPL6peuT8hZS1BqEWZtWOz1pvlhTl1yERl7Tov/zFRxD7q/EhnZ3aN1YyiGOVvjFXJCGz1vq/zYaYinieZU0+bd91dXVUVNTU+owrCWU44UZkotRC/9Ca9JPDtlOc8a+MOGFjhdHOfy30c5ImhcR29wlCq5BWLkph1+q0HiHaGvGUS7t7uUSR5F/MdvW3Adh5cV3qmytXNrdyyUOa1VOEFZefGHeWrl0iJZLHNaqnCBsi1LfpQK+MOdrhTtV2lQc1qrcSW2JcumULZc4GmJxh6i1c+6ktqaVS6dsB7qF0KzcOUGUg3L4pVoubf/gC7NZmXAfRKmVywBk5dL2b2Zlwwmi1Mrlts5y6ZQ1s7LhBFFq5dK047tUzCyP+yBKrVxGhgS3/ZvZVlyDKDU37ZhZmXKCKDU37ZhZmXITUzlw046ZlSHXIMzMLJMTRDmMP2RmVoY6dhNTucw9YGZWhopag5B0iqSlkpZJuipj/X6SHpO0QFKtpNNy1h0q6VlJiyUtktS1xQMsl4fUzMzKUNFqEJIqgBuBLwB1wFxJMyPixZzNvg9Mj4ibJA0CZgH9JHUCfg18LSIWSuoFbGjxIMvlITUzszJUzBrEMGBZRLwWEeuBacDovG0C2D1d7gmsTJdPAmojYiFARKyKiE0tHqHHHzIz265iJoi+QO4jwnVpWa5JwFcl1ZHUHr6dlh8EhKTZkuZLujLrBJIulFQjqaa+vr75EfohNTOz7Sr1XUxjgdsjohI4DbhL0i4kTV/HAuenr2dIGpm/c0TcHBHVEVHdp0+f5p/dD6mZmW1XMe9iWgHsm/O+Mi3L9Q3gFICIeDbtiO5NUtt4MiLeAZA0CzgcmNPiUfohNTOzTMWsQcwFDpTUX1IX4FxgZt42fwZGAkg6GOgK1AOzgcGSPpF2WB8PvIiZmbWaotUgImKjpEtILvYVwK0RsVjSZKAmImYC3wF+KWkCSYf1+Egmyf6LpB+TJJkAZkXEQ8WK1czMtqXketz2VVdXR01NTanDMDNrUyTNi4jqrHWl7qQ2M7My5QRhZmaZ2k0Tk6R64E+ljmMn9QbeKXUQZcTfx9b8fWzh72JrO/N97B8Rmc8JtJsE0R5IqtleW2BH5O9ja/4+tvB3sbVifR9uYjIzs0xOEGZmlskJorzcXOoAyoy/j635+9jC38XWivJ9uA/CzMwyuQZhZmaZnCDMzCyTE0QZkLRvOvXqi+kUq/9Y6phKTVJFOhXtf5Y6llKTtIek+yS9JGmJpKNLHVMpSZqQ/n/ygqR7ijIdcRmTdKuktyW9kFO2p6SHJb2Svn6yJc7lBFEeNgLfiYhBwFHAt9IpWDuyfwSWlDqIMvEz4L8iYiBwGB34e5HUF7gUqI6IQ0gGAj23tFG1uttJp0nIcRUwJyIOJJkW4aqWOJETRBmIiDciYn66vJbkApA/+16HIakSOB34ValjKTVJPYHPAbcARMT6iFhd2qhKrhPQLZ0K4BNsmaq4Q4iIJ4F384pHA3eky3cAX26JczlBlBlJ/YAq4H9KG0lJ/RS4Evi41IGUgf4kc6Tclja5/UrSbqUOqlQiYgVwHclcMm8AayLi96WNqizsFRFvpMtvAnu1xEGdIMqIpO7A/cA/RcR7pY6nFCR9EXg7IuaVOpYy0YlkNsWbIqIK+Cst1HzQFqVt66NJEuc+wG6SvlraqMpLOqdOizy/4ARRJiR1JkkOd0fEA6WOp4SGA6MkLQemASdK+nVpQyqpOqAuIhpqlPeRJIyO6vPAHyOiPiI2AA8Ax5Q4pnLwlqS9AdLXt1vioE4QZUCSSNqYl0TEj0sdTylFxNURURkR/Ug6Hx+NiA77CzEi3gRelzQgLRpJx55+98/AUel0xCL5Pjpsp32OmcC4dHkc8P9b4qBOEOVhOPA1kl/Lz6d/p5U6KCsb3wbullQLDAH+ucTxlExak7oPmA8sIrmGdahhNyTdAzwLDJBUJ+kbwI+AL0h6haSW9aMWOZeH2jAzsyyuQZiZWSYnCDMzy+QEYWZmmZwgzMwskxOEmZllcoKwsiRpU3q772JJCyV9R1JZ//cqqY+k/0mHxDgub93jkpamn+XpnOcaso7TL3ekztYkaZSkRp/UljRe0g2tFZOVTqdSB2C2HesiYgiApE8B/wHsDkzc2QNLqoiITTt7nAwjgUUR8XfbWX9+RNRIuhCYCowqQgw7JSJmkjx0ZeYahJW/iHgbuBC4RIkKSVMlzZVUK+kiAEm7SPpFOm/Cw5JmSTorXbdc0r9Img+cLekkSc9Kmi/pN+k4WEgaKukJSfMkzW4YviBX+gv/0fTccyTtJ2kI8K/A6LTm062Rj/Qk8Jn0s0xN5zVYJOkrGed6Mj12w/v/lnSYpEnpvACPS3pN0qU521yWHvMFSf+UE6VQlIsAAAM9SURBVPNLkm6X9LKkuyV9Pq3NvCJpWLrd5tqBpC/l1IgekdQiA8BZ2+EEYW1CRLxGMvb/p4BvkIzieQRwBPD3kvoDY4B+wCCSJ9PzJ9ZZFRGHA48A3wc+n76vAS5Lx8P6OXBWRAwFbgWmZITzc+COiDgUuBu4PiKeB64B7o2IIRGxrpGP8yWSp4DHkDwZfRjJ069TMxLSLcB4AEkHAV0jYmG6biBwMjAMmCips6ShwNeBI0nmFvl7SVXp9p8B/i3dbyBwHnAscDnw3Yw4/xs4Kh0kcBrJCLvWgbiJydqik4BDG2oHQE/gQJKL3W8i4mPgTUmP5e13b/p6FEkSeToZzocupEMXAIcAD6flFSRDSuc7muTiDnAXSc2hEHdLWgcsJxk+4zLgnrS56y1JT5AkvNqcfX4D/EDSFcAFJJPFNHgoIj4CPpL0NskQz8cCv42IvwJIegA4jqTZ6I8RsSgtX0wywUxIWkSSWPNVAvemSasL8McCP6e1E04Q1iZIOgDYRDJKpYBvR8TsvG2aGr/qrw2bAg9HxNi8/QcDiyOiWFN6nh8RNTnna3KHiPhA0sMkQ1yfAwzNWf1RzvImmv7/OXf7j3Pef7ydfX8O/DgiZkoaAUxqMmBrV9zEZGVPUh/g34Eb0rHuZwMXp01CSDpIySQ6TwNnpn0RewEjtnPIPwDDJX0m3X+3tPlmKdBH6ZzPaZPNZzP2f4Yt01yeDzy1gx/tKeAraZ9KH5KZ457L2O5XwPXA3Ij4SwHH/LKS0U53A87Yifh6AivS5XGNbWjtk2sQVq66SXoe6EwyZ/ddQMNQ6L8iaRKZr+RneD3JFIv3s2U47NdJRvxck3/giKiXNB64R9KuafH3I+LltNnqeiVTfXYimd1ucd4hvk0yw9sV6bm/voOf8bckzVULSSZ4uTIi3lQyq2BuvPMkvQfc1tQBI2K+pNvZkmh+FREL8o9ZoEnAbyT9BXiUZJIe60A8mqu1K5K6R8T7knqRXCSHp3MqtFmS9gEeBwam/StmrcI1CGtv/lPSHiSdqte2g+TwtyR3Ul3m5GCtzTUIMzPL5E5qMzPL5ARhZmaZnCDMzCyTE4SZmWVygjAzs0z/C4TVO2O4bVMCAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEICAYAAABI7RO5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxU1Z338c83gICCEIFklDYBEwWJIA0tKLigZkFNXHDtOBOIM3EZDRkz6kgW4UUeHpPgxISsD4n7EJBoYjDBQcUFozGhEWhARNGQ2GgUSUBUlCW/5497uy2apruudHVVd3/fr1e9qurce279bmnXj3POvecoIjAzM8vX+4odgJmZtS5OHGZmlokTh5mZZeLEYWZmmThxmJlZJk4cZmaWSUETh6SbJb0qaeUetkvSDElrJVVLGpazbbyk59LH+Jzy4ZJWpHVmSFIhz8HMzHalQt7HIel44A3g9og4ooHtpwJfBE4FRgLfi4iRkg4AqoAKIIAlwPCI+LukPwITgT8A84EZEXFfY3H07t07+vXr13wnZmbWDixZsuS1iOhTv7xjIT80IhZJ6tfILmeQJJUAnpTUU9KBwBjggYj4G4CkB4Cxkh4B9o+IJ9Py24EzgUYTR79+/aiqqtrLszEza18k/bmh8mKPcfQFXsx5X5OWNVZe00C5mZm1kGInjoKRdLGkKklVGzZsKHY4ZmZtRrETx3rg4Jz3ZWlZY+VlDZTvJiJmRkRFRFT06bNbF52Zmb1HxU4c84DPpVdXHQ1sjoiXgQXAJyW9X9L7gU8CC9Jtr0s6Or2a6nPAr4sWvZlZO1TQwXFJs0kGuntLqgEmA50AIuInJFdFnQqsBd4CPp9u+5ukbwCL00NNrR0oB/4duBXoSjIo3ujAuJmZNa+CXo5bKioqKqLVXlVVPRcWToXNNdCjDE6+DoacV+yozKyUNdPvhqQlEVFRv7zYXVXWmOq5cO9E2PwiEMnzvROT8mLEcuMRMKVn8txeY3AcjqPU42iB3w23OErZjUek//Hr6XEwXNngzfiFUfs/4vat75Z16gqfmdFyrZ9SiMFxOI7WEEcz/m64xdEaba7JVl4oC6fu+ocAyfuFU9tXDI7DcbSGOFrgd8OJo5T1KMtWXiilkMBKIQbH4ThaQxwt8LvhxLEnxe6nhGRAq1PXXcs6dU3KW1IpJLBSiMFxOI7WEEcL/G44cTSkVAalh5yX9I32OBhQ8tzSfbZQGgmsFGJwHI6jNcTRAr8bHhxvSKkMSpeSUrgsuBRicByOo7XE0Qz2NDjuxNGQKT1JZnOvTzBlU3OFZWZW0nxVVRal0E9pZlainDgaUgr9lGZmJcqJoyGlMihtZlaCCjrJYas25DwnCjOzBrjFYWZmmThxmJlZJk4cZmaWiROHmZll4sRhZmaZOHGYmVkmThxmZpaJE4eZmWXixGFmZpk4cZiZWSZOHGZmlokTh5mZZeLEYWZmmThxmJlZJk4cZmaWiROHmZll4sRhZmaZFDRxSBoraY2ktZKubWD7hyUtlFQt6RFJZTnbviVpZfo4P6f8Vkl/krQsfQwt5DmYmdmuCpY4JHUAfgicAgwCKiUNqrfbDcDtETEEmApcn9Y9DRgGDAVGAldJ2j+n3tURMTR9LCvUOZiZ2e4K2eIYAayNiBciYhswBzij3j6DgIfS1w/nbB8ELIqIHRHxJlANjC1grGZmlqdCJo6+wIs572vSslzLgXHp67OA7pJ6peVjJe0rqTdwInBwTr1paffWjZI6FyZ8MzNrSLEHx68CTpC0FDgBWA/sjIj7gfnAE8Bs4PfAzrTOJGAgcBRwAPBfDR1Y0sWSqiRVbdiwobBnYWbWjhQycaxn11ZCWVpWJyJeiohxEVEOfDUt25Q+T0vHMD4BCHg2LX85Eu8At5B0ie0mImZGREVEVPTp06e5z83MrN0qZOJYDBwqqb+kfYALgHm5O0jqLak2hknAzWl5h7TLCklDgCHA/en7A9NnAWcCKwt4DmZmVk/HQh04InZIugJYAHQAbo6IVZKmAlURMQ8YA1wvKYBFwOVp9U7AY0lu4HXgnyNiR7ptlqQ+JK2QZcClhToHMzPbnSKi2DEUXEVFRVRVVRU7DDOzVkXSkoioqF9e7MFxMzNrZZw4zMwsEycOMzPLxInDzMwyceIwM7NMnDjMzCwTJw4zM8vEicPMzDJx4jAzs0ycOMzMLBMnDjMzy8SJw8zMMnHiMDOzTJw4zMwsEycOMzPLxInDzMwyceIwM7NMnDjMzCwTJw4zM8vEicPMzDJx4jAzs0ycOMzMLBMnDjMzy8SJw8zMMnHiMDOzTJw4zMwsEycOMzPLxInDzMwyceIwM7NMCpo4JI2VtEbSWknXNrD9w5IWSqqW9Iikspxt35K0Mn2cn1PeX9If0mPeKWmfQp6DmZntqmCJQ1IH4IfAKcAgoFLSoHq73QDcHhFDgKnA9Wnd04BhwFBgJHCVpP3TOt8CboyIjwJ/B/61UOdgZma7K2SLYwSwNiJeiIhtwBzgjHr7DAIeSl8/nLN9ELAoInZExJtANTBWkoCTgLvS/W4DzizgOZiZWT2FTBx9gRdz3tekZbmWA+PS12cB3SX1SsvHStpXUm/gROBgoBewKSJ2NHJMMzMroGIPjl8FnCBpKXACsB7YGRH3A/OBJ4DZwO+BnVkOLOliSVWSqjZs2NDMYZuZtV+FTBzrSVoJtcrSsjoR8VJEjIuIcuCradmm9HlaRAyNiE8AAp4FNgI9JXXc0zFzjj0zIioioqJPnz7NeV5mZu1aIRPHYuDQ9CqofYALgHm5O0jqLak2hknAzWl5h7TLCklDgCHA/RERJGMh56R1xgO/LuA5mJlZPQVLHOk4xBXAAmA1MDciVkmaKun0dLcxwBpJzwIfBKal5Z2AxyQ9DcwE/jlnXOO/gC9LWksy5nFToc7BzMx2p+Qf8W1bRUVFVFVVFTsMM7NWRdKSiKioX17swXEzM2tl8kockgYXOhAzM2sd8m1x/EjSHyX9u6QeBY3IzMxKWl6JIyKOAy4kubx2iaSfS/pEQSMzM7OSlPcYR0Q8B3yN5KqmE4AZkp6RNK7xmmZm1pbkO8YxRNKNJJfVngR8JiIOT1/fWMD4zMysxHRsehcAvg/8DPhKRGytLYyIlyR9rSCRmZlZScq3q+pXEXFHbtKQ9CWAiLijIJGZmVlJyjdxfK6BsgnNGIeZmbUSjXZVSaoEPgv0l5Q7z1R34G+FDMzMzEpTU2McTwAvA72B/84p30KyuJKZmbUzjSaOiPgz8GfgmJYJx8zMSl1TXVW/i4hjJW0BcmdDFBARsf8eqpqZWRvVVIvj2PS5e8uEY2ZmpS7fGwCPltQ95313SSMLF5aZmZWqfC/H/THwRs77N9MyMzNrZ/JNHIqcFZ8i4h/kf9e5mZm1IfkmjhckTZTUKX18CXihkIGZmVlpyjdxXAqMAtYDNcBI4OJCBWVmZqUrr+6miHgVuKDAsZiZWSvQ1H0c10TEtyV9n13v4wAgIiYWLDIzMytJTbU4nk6fqwodiJmZtQ5NJY7zgd8APSPiey0Qj5mZlbimBseHSzoIuEjS+yUdkPtoiQDNzKy0NNXi+AmwEDgEWEIyR1WtSMvNzKwdabTFEREz0rXFb46IQyKif87DScPMrB1q6qqq/SPideCrDXVNRYQXczIza2ea6qr6OfBpkm6qwF1VZmbtXlPTqn86fe7fMuGYmVmpy3da9bMk9ch531PSmXnUGytpjaS1kq5tYPuHJS2UVC3pEUllOdu+LWmVpNWSZkhSWv5Iesxl6eMD+Z2qmZk1h3znqpocEZtr30TEJmByYxUkdQB+CJwCDAIqJQ2qt9sNwO0RMQSYClyf1h0FjAaGAEcARwEn5NS7MCKGpo9X8zwHMzNrBvkmjob2a2p8ZASwNiJeiIhtwBzgjHr7DAIeSl8/nLM9gC7APkBnoBPwSp6xmplZAeWbOKokfUfSR9LHd0gGzBvTF3gx531NWpZrOTAufX0W0F1Sr4j4PUkieTl9LIiI1Tn1bkm7qb5e24VlZmYtI9/E8UVgG3AnScvhbeDyZvj8q4ATJC0l6YpaD+yU9FHgcKCMJNmcJOm4tM6FETEYOC59/EtDB5Z0saQqSVUbNmxohlDNzAzyn1b9TeBaSfulr/OxHjg4531ZWpZ73JdIWxySugFnR8QmSV8AnoyIN9Jt9wHHAI9FxPq07hZJPyfpEru9gZhnAjMBKioqdpvZ18zM3pt8r6oaJelpYHX6/khJP2qi2mLgUEn9Je1Dsp7HvHrH7S2pNoZJwM3p67+QtEQ6SupE0hpZnb7vndbtRHKPycp8zsHMzJpHvl1VNwKfAjYCRMRy4PjGKkTEDuAKYAFJwpkbEaskTZV0errbGGCNpGeBDwLT0vK7gOeBFSTjIMsj4l6SgfIFkqqBZSQtmJ/meQ5mZtYM8uqqAoiIF+uNQ+/Mo858YH69sutyXt9FkiTq19sJXNJA+ZvA8HxjNjOz5pdv4ngxvbci0i6iL5F2W5mZWfuSb1fVpSRXUfUFXgKG0jxXVZmZWSuT71VVrwEXFjgWMzNrBfK9quoQSfdK2iDpVUm/luSZcc3M2qF8u6p+DswFDgQOAn4BzC5UUGZmVrryTRz7RsQdEbEjffwPyVxSZmbWzuR7VdV96bToc0gmIDwfmF+7KqBXAjQzaz/yTRznpc8Xp8+1N3RcgFcCNDNrV5pac/wo4MXaFQAljQfOBtYBU9zSMDNrf5oa4/h/JLPiIul4koWWbgM2k04gaGZm7UtTXVUdcloV5wMzI+Ju4G5JywobmpmZlaKmWhwdJNUml5N5d7U+yDDPlZmZtR1N/fjPBh6V9BqwFXgMIF1oaXNjFc3MrG1qNHFExDRJC0lu/Ls/ImoXRHofyaqAZmbWzjTZ3RQRTzZQ9mxhwjEzs1KX753jZmZmgBOHmZll5MRhZmaZOHGYmVkmThxmZpaJE4eZmWXixGFmZpk4cZiZWSZOHGZmlokTh5mZZeLEYWZmmThxmJlZJk4cZmaWiROHmZllUtDEIWmspDWS1kq6toHtH5a0UFK1pEckleVs+7akVZJWS5ohSWn5cEkr0mPWlZuZWcsoWOKQ1AH4IXAKMAiolDSo3m43ALdHxBBgKnB9WncUMBoYAhwBHAWckNb5MfAF4ND0MbZQ52BmZrsrZItjBLA2Il6IiG3AHOCMevsM4t11zB/O2R5AF2AfoDPQCXhF0oHA/hHxZLoa4e3AmQU8BzMzq6eQiaMv8GLO+5q0LNdyYFz6+iygu6ReEfF7kkTycvpYEBGr0/o1TRzTzMwKqNiD41cBJ0haStIVtR7YKemjwOFAGUliOEnScVkOLOliSVWSqjZs2NDccZuZtVuFTBzrgYNz3pelZXUi4qWIGBcR5cBX07JNJK2PJyPijYh4A7gPOCatX9bYMXOOPTMiKiKiok+fPs11TmZm7V7HAh57MXCopP4kP+4XAJ/N3UFSb+BvEfEPYBJwc7rpL8AXJF0PiKQ18t2IeFnS65KOBv4AfA74fgHPwWw39yxdz/QFa3hp01YO6tmVqz81gDPLW77H1HE4jmLFULDEERE7JF0BLAA6ADdHxCpJU4GqiJgHjAGulxTAIuDytPpdwEnACpKB8v+NiHvTbf8O3Ap0JWmJ3FeoczCr756l65n0yxVs3b4TgPWbtjLplysAWvTHwXE4jmLGoOTipLatoqIiqqqqih2GtQGjv/kQ6zdt3a28b8+uPH7tSY7DcRQ9juaMQdKSiKioX17IriqzZlUKXQAvNfAH2Vi543AcLR1HS8RQ7KuqzPJS2/xev2krwbvN73uWNnhtRMEc1LNrpnLH4ThaOo6WiMGJw1qF6QvW1PXZ1tq6fSfTF6xp0Tiu/tQAunbqsEtZ104duPpTAxyH4yiJOFoiBndVWatQCl0A8O7gYrG7zByH4yhmDB4ct1ahFAYdzdqbPQ2Ou6vKWoVS6AIws4S7qqxVKIUuADNLOHFYXkrhUtgzy/s6UZiVACcOa1Ip3A1rZqXDYxzWpFK5FNbMSoMThzWpVC6FNbPS4K6qElcKYwsH9eza4KWwLX1XrpmVBrc4SlipTLPhS2HNLJcTRwkrlbGFM8v7cv24wfTt2RWR3HR3/bjBHhg3a6fcVVXCSmlswZfCmlkttzhKWCnMtGlmVp8TRwnz2IKZlSJ3VZUwT7NhZqXIiaPEeWzBzEqNu6rMzCwTtzjMrFXZvn07NTU1vP3228UOpc3o0qULZWVldOrUKa/9nTjMrFWpqamhe/fu9OvXD0nFDqfViwg2btxITU0N/fv3z6uOu6rMrFV5++236dWrl5NGM5FEr169MrXgnDjMrNVx0mheWb9PJw4zsww2btzI0KFDGTp0KP/0T/9E3759695v27at0bpVVVVMnDixyc8YNWpUc4VbEB7jMDPLoFevXixbtgyAKVOm0K1bN6666qq67Tt27KBjx4Z/WisqKqioqGjyM5544onmCbZA3OIwszbtnqXrGf3Nh+h/7W8Z/c2HCjK79IQJE7j00ksZOXIk11xzDX/84x855phjKC8vZ9SoUaxZk0xM+sgjj/DpT38aSJLORRddxJgxYzjkkEOYMWNG3fG6detWt/+YMWM455xzGDhwIBdeeCERAcD8+fMZOHAgw4cPZ+LEiXXHbQlucZhZm9WSyx7X1NTwxBNP0KFDB15//XUee+wxOnbsyIMPPshXvvIV7r777t3qPPPMMzz88MNs2bKFAQMGcNlll+12SezSpUtZtWoVBx10EKNHj+bxxx+noqKCSy65hEWLFtG/f38qKyub9Vya4sRhZm1WY0sTNHfiOPfcc+nQIZlbbvPmzYwfP57nnnsOSWzfvr3BOqeddhqdO3emc+fOfOADH+CVV16hrKxsl31GjBhRVzZ06FDWrVtHt27dOOSQQ+oun62srGTmzJnNej6NKWhXlaSxktZIWivp2ga2f1jSQknVkh6RVJaWnyhpWc7jbUlnpttulfSnnG1DC3kOZtZ6teTSBPvtt1/d669//euceOKJrFy5knvvvXePl7p27ty57nWHDh3YsWPHe9qnpRUscUjqAPwQOAUYBFRKGlRvtxuA2yNiCDAVuB4gIh6OiKERMRQ4CXgLuD+n3tW12yNiWaHOwcxat2ItTbB582b69k1aNLfeemuzH3/AgAG88MILrFu3DoA777yz2T+jMYVscYwA1kbECxGxDZgDnFFvn0HAQ+nrhxvYDnAOcF9EvFWwSM2sTSrW0gTXXHMNkyZNory8vCAthK5du/KjH/2IsWPHMnz4cLp3706PHj2a/XP2RLUj9M1+YOkcYGxE/Fv6/l+AkRFxRc4+Pwf+EBHfkzQOuBvoHREbc/Z5CPhORPwmfX8rcAzwDrAQuDYi3mksloqKiqiqqmrW8zOz4li9ejWHH3543vvfs3R9m1ya4I033qBbt25EBJdffjmHHnooV1555Xs+XkPfq6QlEbHb9cPFHhy/CviBpAnAImA9UDeSJelAYDCwIKfOJOCvwD7ATOC/SLq5diHpYuBigA996EOFid7MSl5bXZrgpz/9Kbfddhvbtm2jvLycSy65pMU+u5CJYz1wcM77srSsTkS8BIwDkNQNODsiNuXsch7wq4jYnlPn5fTlO5JuIUk+u4mImSSJhYqKisI0q8zMiuTKK6/cqxbG3ijkGMdi4FBJ/SXtA1wAzMvdQVJvSbUxTAJurneMSmB2vToHps8CzgRWFiB2MzPbg4IljojYAVxB0s20GpgbEaskTZV0errbGGCNpGeBDwLTautL6kfSYnm03qFnSVoBrAB6A/+nUOdgZma7K+gYR0TMB+bXK7su5/VdwF17qLsO2K1jMiJOat4ozcwsC89VZWZmmThxmJllcOKJJ7JgwYJdyr773e9y2WWXNbj/mDFjqL0d4NRTT2XTpk277TNlyhRuuOGGRj/3nnvu4emnn657f9111/Hggw9mDb9ZOHGYmWVQWVnJnDlzdimbM2dOXhMNzp8/n549e76nz62fOKZOncrHP/7x93SsveXEYWZtW/VcuPEImNIzea6eu1eHO+ecc/jtb39bt2jTunXreOmll5g9ezYVFRV87GMfY/LkyQ3W7devH6+99hoA06ZN47DDDuPYY4+tm3YdkvszjjrqKI488kjOPvts3nrrLZ544gnmzZvH1VdfzdChQ3n++eeZMGECd92VDBEvXLiQ8vJyBg8ezEUXXcQ777xT93mTJ09m2LBhDB48mGeeeWavzr2WE4eZtV3Vc+HeibD5RSCS53sn7lXyOOCAAxgxYgT33XcfkLQ2zjvvPKZNm0ZVVRXV1dU8+uijVFdX7/EYS5YsYc6cOSxbtoz58+ezePHium3jxo1j8eLFLF++nMMPP5ybbrqJUaNGcfrppzN9+nSWLVvGRz7ykbr93377bSZMmMCdd97JihUr2LFjBz/+8Y/rtvfu3ZunnnqKyy67rMnusHw5cZhZ27VwKmyvNxPu9q1J+V7I7a6q7aaaO3cuw4YNo7y8nFWrVu3SrVTfY489xllnncW+++7L/vvvz+mnn163beXKlRx33HEMHjyYWbNmsWrVqkZjWbNmDf379+ewww4DYPz48SxatKhu+7hx4wAYPnx43aSIe8uJw8zars012crzdMYZZ7Bw4UKeeuop3nrrLQ444ABuuOEGFi5cSHV1Naeddtoep1JvyoQJE/jBD37AihUrmDx58ns+Tq3aadmbc0p2J449aInlJs2swHqUZSvPU7du3TjxxBO56KKLqKys5PXXX2e//fajR48evPLKK3XdWHty/PHHc88997B161a2bNnCvffeW7dty5YtHHjggWzfvp1Zs2bVlXfv3p0tW7bsdqwBAwawbt061q5dC8Add9zBCSecsFfn1xQnjgbULje5ftNWgneXm3TyMGtlTr4OOtVbe6NT16R8L1VWVrJ8+XIqKys58sgjKS8vZ+DAgXz2s59l9OjRjdYdNmwY559/PkceeSSnnHIKRx11VN22b3zjG4wcOZLRo0czcODAuvILLriA6dOnU15ezvPPP19X3qVLF2655RbOPfdcBg8ezPve9z4uvfTSvT6/xhRsWvVSknVa9dHffIj1DawQ1rdnVx6/1jeumxVT1mnVqZ6bjGlsrklaGidfB0POK1yArVRrmla9JLXkcpNmVmBDznOiaGbuqmpAsZabNDNrDZw4GlCs5SbNzFoDd1U1oHa1sLa43KRZWxARJEvyWHPIOtbtxLEHbXW5SbPWrkuXLmzcuJFevXo5eTSDiGDjxo106dIl7zpOHGbWqpSVlVFTU8OGDRuKHUqb0aVLF8rK8r+3xYnDzFqVTp060b9//2KH0a55cNzMzDJx4jAzs0ycOMzMLJN2MeWIpA3An4sdx17qDbxW7CBKhL+LXfn72JW/j3ft7Xfx4YjoU7+wXSSOtkBSVUNzxrRH/i525e9jV/4+3lWo78JdVWZmlokTh5mZZeLE0XrMLHYAJcTfxa78fezK38e7CvJdeIzDzMwycYvDzMwyceIoYZIOlvSwpKclrZL0pWLHVAokdZC0VNJvih1LsUnqKekuSc9IWi3pmGLHVCySrkz/TlZKmi0p/1n72gBJN0t6VdLKnLIDJD0g6bn0+f3N8VlOHKVtB/CfETEIOBq4XNKgIsdUCr4ErC52ECXie8D/RsRA4Eja6fciqS8wEaiIiCOADsAFxY2qxd0KjK1Xdi2wMCIOBRam7/eaE0cJi4iXI+Kp9PUWkh+Fdj3Xu6Qy4DTgZ8WOpdgk9QCOB24CiIhtEbGpuFEVVUegq6SOwL7AS0WOp0VFxCLgb/WKzwBuS1/fBpzZHJ/lxNFKSOoHlAN/KG4kRfdd4BrgH8UOpAT0BzYAt6Rddz+TtF+xgyqGiFgP3AD8BXgZ2BwR9xc3qpLwwYh4OX39V+CDzXFQJ45WQFI34G7gPyLi9WLHUyySPg28GhFLih1LiegIDAN+HBHlwJs0U1dEa5P23Z9BkkwPAvaT9M/Fjaq0RHIJbbNcRuvEUeIkdSJJGrMi4pfFjqfIRgOnS1oHzAFOkvQ/xQ2pqGqAmoiobYXeRZJI2qOPA3+KiA0RsR34JTCqyDGVglckHQiQPr/aHAd14ihhStbFvAlYHRHfKXY8xRYRkyKiLCL6kQx8PhQR7fZflRHxV+BFSQPSopOBp4sYUjH9BTha0r7p383JtNMLBeqZB4xPX48Hft0cB3XiKG2jgX8h+Zf1svRxarGDspLyRWCWpGpgKPB/ixxPUaStrruAp4AVJL9t7eoOckmzgd8DAyTVSPpX4JvAJyQ9R9Iq+2azfJbvHDczsyzc4jAzs0ycOMzMLBMnDjMzy8SJw8zMMnHiMDOzTJw4rFWRtDO9LHmVpOWS/lNSSf9/LKmPpD+k04IcV2/bI5LWpOfyeM49GQ0dp1/uzKctSdLpkhq9K13SBEk/aKmYrHg6FjsAs4y2RsRQAEkfAH4O7A9M3tsDS+oQETv39jgNOBlYERH/toftF0ZElaSLgenA6QWIYa9ExDySm8nM3OKw1isiXgUuBq5QooOk6ZIWS6qWdAmApPdJ+lG6ZsUDkuZLOifdtk7StyQ9BZwr6ZOSfi/pKUm/SOcJQ9JwSY9KWiJpQe00DrnSFsFD6WcvlPQhSUOBbwNnpC2lro2c0iLgo+m5TE/XlVgh6fwGPmtReuza97+TdKSkKem6DI9IekHSxJx9vpwec6Wk/8iJ+RlJt0p6VtIsSR9PWz/PSRqR7lfXmpD0mZwW1IOSmmXiPGs9nDisVYuIF0jWXvgA8K8ks6IeBRwFfEFSf2Ac0A8YRHInfv3FjjZGxDDgQeBrwMfT91XAl9P5wr4PnBMRw4GbgWkNhPN94LaIGALMAmZExDLgOuDOiBgaEVsbOZ3PkNz1PI7kLvAjSe72nd5AoroJmAAg6TCgS0QsT7cNBD4FjAAmS+okaTjweWAkydouX5BUnu7/UeC/03oDgc8CxwJXAV9pIM7fAUenEyvOIZmt2NoRd1VZW/JJYEhtawLoARxK8iP4i4j4B/BXSQ/Xq3dn+nw0SXJ5PJnuiH1Ip3AAjgAeSMs7kEzdXd8xJKm02GUAAAI3SURBVD/6AHeQtDTyMUvSVmAdyRQiXwZmp91mr0h6lCQRVufU+QXwdUlXAxeRLOJT67cR8Q7wjqRXSabSPhb4VUS8CSDpl8BxJN1Pf4qIFWn5KpKFf0LSCpKEW18ZcGeazPYB/pTneVob4cRhrZqkQ4CdJLN+CvhiRCyot09T83u9Wbsr8EBEVNarPxhYFRGFWpb1woioyvm8JitExFuSHiCZSvw8YHjO5ndyXu+k6b/z3P3/kfP+H3uo+33gOxExT9IYYEqTAVub4q4qa7Uk9QF+AvwgXWtgAXBZ2rWEpMOULGz0OHB2OtbxQWDMHg75JDBa0kfT+vul3UBrgD5K1/NOu34+1kD9J3h3udILgcfe46k9Bpyfjtn0IVnl748N7PczYAawOCL+nscxz1Qye+x+wFl7EV8PYH36enxjO1rb5BaHtTZdJS0DOpGsyX4HUDvl/M9IulaeUvLP9g0kS2XezbtTjr9IMoPq5voHjogNkiYAsyV1Tou/FhHPpt1fM5Qs19qRZCXCVfUO8UWS1fiuTj/78+/xHH9F0u21nGThnWsi4q9KVoHMjXeJpNeBW5o6YEQ8JelW3k1AP4uIpfWPmacpwC8k/R14iGTxJGtHPDuutQuSukXEG5J6kfx4jk7Xs2i1JB0EPAIMTMdvzFqEWxzWXvxGUk+SwdxvtIGk8TmSK7u+7KRhLc0tDjMzy8SD42ZmlokTh5mZZeLEYWZmmThxmJlZJk4cZmaWiROHmZll8v8Bb9+x6yh5pmUAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d2YjvOT4EJOX",
        "outputId": "87f8a3dd-2f40-4eab-baa1-6f7e346cbc9f"
      },
      "source": [
        "#Regular Fitting\n",
        "SVM_linear = svm.SVC(C=1.0,kernel ='linear')\n",
        "SVM_linear.fit(x_train_tfidf, y_train)\n",
        "\n",
        "SVM_poly = svm.SVC(C=1.0,kernel ='poly', degree=10)\n",
        "SVM_poly.fit(x_train_tfidf, y_train)\n",
        "\n",
        "SVM_rbf = svm.SVC(C=1.0,kernel ='rbf')\n",
        "SVM_rbf.fit(x_train_tfidf, y_train)\n",
        "\n",
        "SVM_sigmoid = svm.SVC(C=1.0,kernel ='sigmoid')\n",
        "SVM_sigmoid.fit(x_train_tfidf, y_train)\n",
        "\n",
        "#Data Augmentation Fitting\n",
        "SVM_linear_1 = svm.SVC(C=1.0,kernel ='linear')\n",
        "SVM_linear_1.fit(x_train_tfidf_1, y_train_1)\n",
        "\n",
        "SVM_linear_2 = svm.SVC(C=1.0,kernel ='linear')\n",
        "SVM_linear_2.fit(x_train_tfidf_2, y_train_2)\n",
        "\n",
        "SVM_linear_del = svm.SVC(C=1.0,kernel ='linear')\n",
        "SVM_linear_del.fit(x_train_tfidf_del, y_train)"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "SVC(kernel='linear')"
            ]
          },
          "metadata": {},
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9DV7LL9xIFkB"
      },
      "source": [
        "test_tokens = token_generator(test_text)\n",
        "svm_linear_predictions = list()\n",
        "svm_poly_predictions = list()\n",
        "svm_rbf_predictions = list()\n",
        "svm_sigmoid_predictions = list()\n",
        "\n",
        "svm_linear_predictions_1 = list()\n",
        "svm_linear_predictions_2 = list()\n",
        "svm_linear_predictions_del = list()\n",
        "\n",
        "for token in test_tokens:\n",
        "  test_sparse_vector = sparse_vector_create(train_vocab_dict.doc2bow(token),len(train_vocab_dict))\n",
        "  x_test_tfidf = tfidf.transform(test_sparse_vector.reshape(1,-1))\n",
        "  svm_linear_predictions.append(SVM_linear.predict(x_test_tfidf)[0])\n",
        "  svm_poly_predictions.append(SVM_poly.predict(x_test_tfidf)[0])\n",
        "  svm_rbf_predictions.append(SVM_rbf.predict(x_test_tfidf)[0])\n",
        "  svm_sigmoid_predictions.append(SVM_sigmoid.predict(x_test_tfidf)[0])\n",
        "\n",
        "  x_test_tfidf_del = tfidf_del.transform(test_sparse_vector_del.reshape(1,-1))\n",
        "  svm_linear_predictions_1.append(SVM_linear_1.predict(x_test_tfidf)[0])\n",
        "  svm_linear_predictions_2.append(SVM_linear_2.predict(x_test_tfidf)[0])\n",
        "  svm_linear_predictions_del.append(SVM_linear_del.predict(x_test_tfidf_del)[0])\n",
        "  "
      ],
      "execution_count": 55,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UZ3gwLdk3XQD"
      },
      "source": [
        "# Where to next?\n",
        "So far we have only examined sparse word vectors. That is, vector representations of words that contain many 0's and sparsely contain data. This may perform adequately for the models presented, but for many cases, working with large-dimension vectors can be a large problem due to high memory usage and computational cost.\n",
        "\n",
        "There has been work done to create dense vectors of words based on \"distributed\" representations (aka based on word distribution). These vectors are are usually only several tens to hundreds of dimensions long as opposed to many thousands. Earlier approaches for doing so include Word2vec and GloVe which create static vectors for word representations. Newer embedding approaches integrate context into embedding creation such as seen in ELMo or BERT's embedding process. \n",
        "\n",
        "Popular deep learning architectures for analyzing text include convolutional neural networks, recurrent neural networks and its cousins LSTM and GRU as well as the more recent transformer model. Deep learning approaches to NLP start with converting words into a vector in an embedding layer, often using one of the aforementioned techniques. We will not explore deep learning approaches to NLP in this tutorial, but since we have mentioned a number of popular NLP packages, we will also mention [HuggingFace](https://huggingface.co/) for their extensive NLP deep learning library. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kJxrGbD62U26"
      },
      "source": [
        "# Evaluation\n",
        "\n",
        "Evaluation of model performance is a very important part of the NLP pipeline. Common metrics (for classification) include sensitivity (recall), specificity, positive predictive value (precision), and the F1 score. These quantities can be derived from a \"confusion matrix\" which is simply a matrix that tabulates model classification results into the categories of true negatives, false positives, false negatives, and true positives.\n",
        "\n",
        "For sklearn confusion matrix, the format is - row i is actual class, row j is predicted class. For our case, 0 - normal, 1 - abnormal. thus $C_{0,0}$ is true negatives, $C_{0,1}$ is false positive, $C_{1,0}$ is false negative, $C_{1,1}$ is true positives where $C_{i,j}$ refers to element in row i and column j. In this example, a \"positive\" label by the model is the equivalent of flagging an example as \"abnormal\".\n",
        "\n",
        "Note we cannot naturally generate a ROC curve for Naive Bayes or SVM as ROC curves require some threshold (usually some \"probability\" cut off for prediction of one class or another) but Naive Bayes simply compares two(or more) quantities with each other to determine class. While using relative comparisons works reasonably well, the numerical value of these outputs is unstable and unsuitable for a fixed cutoff."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qO3zQ04dnziV"
      },
      "source": [
        "from sklearn import metrics\n",
        "\n",
        "class metric_calc:\n",
        "  def __init__(self, y,y_hat):\n",
        "  # y is true label, y_hat is predicted label\n",
        "    self.y_hat = y_hat\n",
        "    self.y = y\n",
        "    conf_matrix = metrics.confusion_matrix(y,y_hat)\n",
        "    \n",
        "    self.true_neg = conf_matrix[0][0]\n",
        "    self.false_pos = conf_matrix[0][1]\n",
        "    self.false_neg = conf_matrix[1][0]\n",
        "    self.true_pos = conf_matrix[1][1]\n",
        "\n",
        "  def conf_matrix_values(self):\n",
        "    print(f\"TN: {self.true_neg}, FP: {self.false_pos}, FN: {self.false_neg} TP: {self.true_pos}\")\n",
        "\n",
        "  def sens_spec(self):\n",
        "    sens = self.true_pos/(self.true_pos+self.false_neg)\n",
        "    spec = self.true_neg/(self.true_neg+self.false_pos)\n",
        "    print(f\"Sensitivity (aka recall) is {sens:.4f}\")\n",
        "    print(f\"Specificity is {spec:.4f}\")\n",
        "\n",
        "  def sens(self):\n",
        "    sens = self.true_pos/(self.true_pos+self.false_neg)\n",
        "    return sens\n",
        "\n",
        "  def spec(self):\n",
        "    spec = self.true_neg/(self.true_neg+self.false_pos)\n",
        "    return spec\n",
        "\n",
        "  def incorrect_index(self):\n",
        "    # return indices of examples that are incorrectly predicted\n",
        "    index = list()\n",
        "    for i,label in enumerate(self.y):\n",
        "      if label != self.y_hat[i]:\n",
        "        index.append(i)\n",
        "    return index"
      ],
      "execution_count": 56,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Regular Evaluation\n",
        "print(\"Naive Bayes\")\n",
        "nb_metric = metric_calc(y_test,nb_predictions)\n",
        "nb_metric.conf_matrix_values()\n",
        "nb_metric.sens_spec()\n",
        "\n",
        "print(\"SVM-Linear\")\n",
        "svm_linear_metric = metric_calc(y_test,svm_linear_predictions)\n",
        "svm_linear_metric.conf_matrix_values()\n",
        "svm_linear_metric.sens_spec()\n",
        "\n",
        "print(\"SVM-Poly\")\n",
        "svm_poly_metric = metric_calc(y_test,svm_poly_predictions)\n",
        "svm_poly_metric.conf_matrix_values()\n",
        "svm_poly_metric.sens_spec()\n",
        "\n",
        "print(\"SVM-rbf\")\n",
        "svm_rbf_metric = metric_calc(y_test,svm_rbf_predictions)\n",
        "svm_rbf_metric.conf_matrix_values()\n",
        "svm_rbf_metric.sens_spec()\n",
        "\n",
        "print(\"SVM-sigmoid\")\n",
        "svm_sigmoid_metric = metric_calc(y_test,svm_sigmoid_predictions)\n",
        "svm_sigmoid_metric.conf_matrix_values()\n",
        "svm_sigmoid_metric.sens_spec()\n",
        "\n",
        "#Data Augmentation Evaluation\n",
        "svm_linear_metric_1 = metric_calc(y_test,svm_linear_predictions_1)\n",
        "svm_linear_metric_1.conf_matrix_values()\n",
        "svm_linear_metric_1.sens_spec()\n",
        "\n",
        "svm_linear_metric_2 = metric_calc(y_test,svm_linear_predictions_2)\n",
        "svm_linear_metric_2.conf_matrix_values()\n",
        "svm_linear_metric_2.sens_spec()\n",
        "\n",
        "svm_linear_metric_del = metric_calc(y_test,svm_linear_predictions_del)\n",
        "svm_linear_metric_del.conf_matrix_values()\n",
        "svm_linear_metric_del.sens_spec()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5Z_3HW68OBqF",
        "outputId": "3a50fe95-8909-4d0b-8ebf-3c0db2590e97"
      },
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Naive Bayes\n",
            "TN: 211, FP: 12, FN: 46 TP: 325\n",
            "Sensitivity (aka recall) is 0.8760\n",
            "Specificity is 0.9462\n",
            "SVM-Linear\n",
            "TN: 209, FP: 14, FN: 12 TP: 359\n",
            "Sensitivity (aka recall) is 0.9677\n",
            "Specificity is 0.9372\n",
            "SVM-Poly\n",
            "TN: 164, FP: 59, FN: 1 TP: 370\n",
            "Sensitivity (aka recall) is 0.9973\n",
            "Specificity is 0.7354\n",
            "SVM-rbf\n",
            "TN: 206, FP: 17, FN: 11 TP: 360\n",
            "Sensitivity (aka recall) is 0.9704\n",
            "Specificity is 0.9238\n",
            "SVM-sigmoid\n",
            "TN: 209, FP: 14, FN: 14 TP: 357\n",
            "Sensitivity (aka recall) is 0.9623\n",
            "Specificity is 0.9372\n",
            "TN: 216, FP: 7, FN: 21 TP: 350\n",
            "Sensitivity (aka recall) is 0.9434\n",
            "Specificity is 0.9686\n",
            "TN: 215, FP: 8, FN: 16 TP: 355\n",
            "Sensitivity (aka recall) is 0.9569\n",
            "Specificity is 0.9641\n",
            "TN: 0, FP: 223, FN: 0 TP: 371\n",
            "Sensitivity (aka recall) is 1.0000\n",
            "Specificity is 0.0000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        " \n",
        "# set width of bar\n",
        "barWidth = 0.25\n",
        "fig = plt.subplots(figsize =(12, 8))\n",
        " \n",
        "# set height of bar\n",
        "sensitivity = [nb_metric.sens(), svm_linear_metric.sens(), svm_poly_metric.sens(), svm_rbf_metric.sens(), svm_sigmoid_metric.sens()]\n",
        "specificity = [nb_metric.spec(), svm_linear_metric.spec(), svm_poly_metric.spec(), svm_rbf_metric.spec(), svm_sigmoid_metric.spec()]\n",
        " \n",
        "# Set position of bar on X axis\n",
        "br_sensitivity = np.arange(len(sensitivity))\n",
        "br_specificity = [x + barWidth for x in br_sensitivity]\n",
        " \n",
        "# Make the plot\n",
        "plt.bar(br_sensitivity, sensitivity, width = barWidth,\n",
        "        edgecolor ='grey', label ='Sensitivity')\n",
        "plt.bar(br_specificity, specificity, width = barWidth,\n",
        "        edgecolor ='grey', label ='Specificity')\n",
        " \n",
        "# Adding Xticks\n",
        "plt.xlabel('Model Type', fontweight ='bold', fontsize = 15)\n",
        "plt.xticks([r + barWidth for r in range(len(specificity))],\n",
        "        ['Naive Bayes', 'SVM Linear', 'SVM Poly', 'SVM RBF', 'SVM Sigmoid'])\n",
        " \n",
        "plt.legend()\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 502
        },
        "id": "r8LJRlDxX95D",
        "outputId": "892cdc19-d22f-4b13-99d8-6ac2fa554c38"
      },
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 864x576 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsIAAAHlCAYAAAD/U5rKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de7hdZXkv7N9DQIJA0Zq0WwwYVGw5hSBRBKVmq7iFXaEeEWk5iPWiCijWftLqFkHth7uKRwRpVaSKgFo1VayWitYDIglGFBBFRIn6aaCAhhoBfb8/5kxYLLKSFTKTteS97+vKtcbhnWM8Y86ROX9zzHeMUa21AABAbzab6gIAAGAqCMIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXNp+qFc+aNavNnTt3qlYPAEAnlixZclNrbfb46VMWhOfOnZvFixdP1eoBAOhEVf1wTdN1jQAAoEuCMAAAXRKEAQDo0pT1EV6TO++8M8uWLcvKlSunupT7nZkzZ2bOnDnZYostproUAIBpYVoF4WXLlmXbbbfN3LlzU1VTXc79RmstN998c5YtW5addtppqssBAJgWplXXiJUrV+YhD3mIEDxiVZWHPOQhjrQDAIwxrYJwEiF4I/G8AgDc07QLwtPBG9/4xuy2226ZN29e5s+fn8suu2xky95vv/2SJDfccEPOO++81dMXL16cE044Ya2PPeuss3LuuecmSc4555z85Cc/GVldAAC9mVZ9hMd705tPz8rbfzmy5c3cetu86pWvWGubSy+9NJ/61KdyxRVXZMstt8xNN92UO+64Y2Q1fPWrX01ydxB+wQtekCRZsGBBFixYsNbHHnvssauHzznnnOy+++7ZfvvtR1YbAEBPpnUQXnn7L/P+X609HK6Po7PuO9n99Kc/zaxZs7LlllsmSWbNmpUkWbJkSV7xildkxYoVmTVrVs4555w89KEPzcKFC7PPPvvkkksuya233pr3vve92X///XPVVVfl6KOPzh133JHf/va3+djHPpadd94522yzTVasWJGTTjop11xzTebPn58jjzwye+21V9785jdn0aJFecQjHpGlS5fmQQ96UJJk5513zpe//OWceeaZ2WabbVbfle/www/PVlttlTe+8Y35x3/8x3ziE59Ikvz7v/973v3ud+fjH//4yJ47AID7G10jxnna056WG2+8MY9+9KPzkpe8JF/84hdz55135vjjj89HP/rRLFmyJC984Qvz6le/evVj7rrrrnz961/P2972tpxyyilJBt0YXvayl2Xp0qVZvHhx5syZc4/1nHbaadl///2zdOnSnHjiiaunb7bZZjnkkENWh9jLLrssD3/4w/OHf/iHq9s85znPyYIFC/KhD30oS5cuzUEHHZTvfOc7Wb58eZLk/e9/f174whdutOcIAOD+QBAeZ5tttsmSJUty9tlnZ/bs2Tn00EPznve8J9/+9rdzwAEHZP78+XnDG96QZcuWrX7Ms571rCTJ3nvvnRtuuCFJsu++++bv//7v86Y3vSk//OEPs9VWW026hkMPPTQXXHBBkuT888/PoYceutb2VZW/+Iu/yAc/+MHceuutufTSS3PggQeu55YDAPRlWneNmCozZszIwoULs3Dhwuyxxx4544wzsttuu+XSSy9dY/tV3ShmzJiRu+66K0nyghe8IPvss08+/elP56CDDsp73vOePPnJT57U+vfdd99cd911Wb58eT7xiU/kNa95zTofc/TRR+cZz3hGZs6cmec+97nZfHMvLQDA2jgiPM61116b733ve6vHly5dml122SXLly9fHYTvvPPOXHXVVWtdzvXXX59HPOIROeGEE3LIIYfkyiuvvMf8bbfdNr/85ZpPBKyqPPOZz8wrXvGK7LLLLnnIQx5yrzbjH7/99ttn++23zxve8IYcffTRk95eAIBerTMIV9X7qurnVfXtCeZXVb2jqq6rqiur6jGjL3PTWbFiRY488sjsuuuumTdvXq6++uqceuqp+ehHP5pXvepV2XPPPTN//vzVV3+YyIUXXpjdd9898+fPz7e//e0cccQR95g/b968zJgxI3vuuWfe+ta33uvxhx56aD74wQ9O2C3iqKOOyrHHHpv58+fnV7/6VZLk8MMPzw477JBddtnlPm49AEA/qrW29gZVf5JkRZJzW2u7r2H+QUmOT3JQkn2SvL21ts+6VrxgwYK2ePE9r+JwzTXX3CPETcXl036XHXfccdlrr71yzDHHrHH++OcXAKAHVbWktXavS5GtsyNpa+0/q2ruWpockkFIbkm+VlUPqqqHttZ+ep+rHbo/h9ZR23vvvbP11lvnLW95y1SXAgDwO2EUZ1Q9LMmNY8aXDadtcBBm8pYsWTLVJQAA/E7ZpCfLVdWLq2pxVS1edc1bAACYCqM4IvzjJDuMGZ8znHYvrbWzk5ydDPoIj2DdAJvEqM9ZuK/u7+c6AGxKowjCi5IcV1XnZ3Cy3G2j6B8MMJ2M+pbv99VkbhUPwOSsMwhX1YeTLEwyq6qWJTk5yRZJ0lo7K8lFGVwx4rok/53ERWwBAJj21tlHuLV2WGvtoa21LVprc1pr722tnTUMwWkDL22tPbK1tkdr7Xf+cMUb3/jG7Lbbbpk3b17mz5+fyy67bGTLPuigg3LrrbcmSd7xjndkl112yeGHH55FixbltNNOW+tj99tvvyTJDTfckPPOO29kNQEA9Gha34f3bW9+U267feXIlrfd1jPz8le+aq1tLr300nzqU5/KFVdckS233DI33XRT7rjjjpHVcNFFF60efve7352LL744c+bMSZIcfPDBa33sqpt4rArCL3jBC0ZWFwBAb6Z1EL7t9pU5uZ0+suWdcvu6TzD56U9/mlmzZmXLLbdMksyaNStJMnfu3Dzvec/LZz7zmWy11VY577zz8qhHPSrLly/Psccemx/96EdJkre97W15whOekBUrVuT444/P4sWLU1U5+eST8+xnPztz587N4sWL85rXvCbXX399DjzwwLzwhS/Mgx/84CxevDjvete78rOf/SzHHntsrr/++iTJmWeemf322y/bbLNNVqxYkZNOOinXXHNN5s+fnyOPPDIf//jH8453vCPz589PkjzxiU/MGWeckT333HNkzx0AwP3NJr182u+Cpz3tabnxxhvz6Ec/Oi95yUvyxS9+cfW87bbbLt/61rdy3HHH5eUvf3mS5GUve1lOPPHEXH755fnYxz6WF73oRUmS17/+9avbX3nllXnyk598j/WcddZZ2X777XPJJZfkxBNPvMe8E044IU960pPyzW9+M1dccUV22223e8w/7bTTsv/++2fp0qU58cQTc8wxx+Scc85Jknz3u9/NypUrhWAAgHUQhMfZZpttsmTJkpx99tmZPXt2Dj300NUh87DDDlv999JLL02SXHzxxTnuuOMyf/78HHzwwfnFL36RFStW5OKLL85LX/rS1ct98IMfPOkaPv/5z+ev/uqvkiQzZszIdtttt9b2z33uc/OpT30qd955Z973vvflqKOOWo8tBgDo07TuGjFVZsyYkYULF2bhwoXZY4898oEPfCBJUlWr26wa/u1vf5uvfe1rmTlz5pTUmiQPfOADc8ABB+STn/xkLrzwQneZAwCYBEeEx7n22mvzve99b/X40qVL8/CHPzxJcsEFF6z+u++++yYZdKV45zvfeY/2SXLAAQfkjDPOWD39lltumXQNT3nKU3LmmWcmSX7zm9/ktttuu8f8bbfdNr/85T0v7P+iF70oJ5xwQh772Meu19FnAIBeCcLjrFixIkceeWR23XXXzJs3L1dffXVe97rXJRmE2Xnz5uXtb3973vrWtyYZXAJt8eLFmTdvXnbdddecddZZSZLXvOY1ueWWW7L77rtnzz33zCWXXDLpGt7+9rfnkksuyR577JG99947V1999T3mz5s3LzNmzMiee+65uo699947v/d7v5ejj3YZZwCAyajWpuZOxwsWLGiLF9/zksPXXHNNdtlll9XjU3H5tImsutrDqqtITDc/+clPsnDhwnznO9/JZput+fvN+OcXmLxTTjlletxZbqvFOfnkk6e6DMZw+22Y/qpqSWvtXm/i07qP8H0Nrb0599xz8+pXvzqnn376hCGYyfGBBqwvt9+G313TOghPJzfccMNUlzChI444IkccccRUl3G/4AMNAPohCAMAbCTT4ZdGvzJObNoF4dbaPS5TxmhMVV9wAOjZdPil0a+ME5tWHUpnzpyZm2++WWgbsdZabr755im91jEAwHQzrY4Iz5kzJ8uWLcvy5cunupT7nZkzZ2bOnDlTXQYAwLQxrYLwFltskZ122mmqywAAoAPTqmsEAABsKoIwAABdmlZdI3oy6rvm3Vcbcrc9AIDfZYLwFLnt9pU5uZ0+1WXklNtdVxAA6JOuEQAAdEkQBgCgS7pGwDQ0o92VU045ZUpr0H8c4P5hOnymJNPzc0UQhmnoN7X5lPch138c4P5hOnymJNPzc0XXCAAAuiQIAwDQJUEYAIAu6SMMAPcDToiC9ScIA8D9gBOiYP3pGgEAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6tPlUFzAV3vTm07Py9l9OdRkAAEyhLoPwytt/mff/asGU1nD0VoundP0AAL3TNQIAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdGlSQbiqnl5V11bVdVV10hrm71hVl1TVN6rqyqo6aPSlAgDA6KwzCFfVjCRnJDkwya5JDquqXcc1e02SC1treyV5fpJ3j7pQAAAYpckcEX5ckutaa9e31u5Icn6SQ8a1aUl+bzi8XZKfjK5EAAAYvc0n0eZhSW4cM74syT7j2rwuyeeq6vgkWyd56kiqAwCAjWRUJ8sdluSc1tqcJAcl+eequteyq+rFVbW4qhYvX758RKsGAID1N5kg/OMkO4wZnzOcNtYxSS5MktbapUlmJpk1fkGttbNbawtaawtmz5593yoGAIARmEwQvjzJzlW1U1U9IIOT4RaNa/OjJE9JkqraJYMg7JAvAADT1jqDcGvtriTHJflskmsyuDrEVVV1alUdPGz210n+sqq+meTDSY5qrbWNVTQAAGyoyZwsl9baRUkuGjfttWOGr07yhNGWBgAAG487ywEA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXdp8qgsAYPJmtLtyyimnTHUZ2W7rmXn5K1811WUAbBBBGOB3yG9q85zcTp/qMnLK7a+Y6hIANpiuEQAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdmlQQrqqnV9W1VXVdVZ00QZvnVdXVVXVVVZ032jIBAGC0Nl9Xg6qakeSMJAckWZbk8qpa1Fq7ekybnZP8bZIntNZuqao/2FgFAwDAKEzmiPDjklzXWru+tXZHkvOTHDKuzV8mOaO1dkuStNZ+PtoyAQBgtCYThB+W5MYx48uG08Z6dJJHV9VXquprVfX0URUIAAAbwzq7RqzHcnZOsjDJnCT/WVV7tNZuHduoql6c5MVJsuOOO45o1QAAsP4mc0T4x0l2GDM+ZzhtrGVJFrXW7myt/SDJdzMIxvfQWju7tbagtbZg9uzZ97VmAADYYJMJwpcn2bmqdqqqByR5fpJF49p8IoOjwamqWRl0lbh+hHUCAMBIrTMIt9buSnJcks8muSbJha21q6rq1Ko6eNjss0lurqqrk1yS5G9aazdvrKIBAGBDTaqPcGvtoiQXjZv22jHDLckrhv8AAGDac2c5AAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANClSQXhqnp6VV1bVddV1UlraffsqmpVtWB0JQIAwOitMwhX1YwkZyQ5MMmuSQ6rql3X0G7bJC9LctmoiwQAgFGbzBHhxyW5rrV2fWvtjiTnJzlkDe1en+RNSVaOsD4AANgoJhOEH5bkxjHjy4bTVquqxyTZobX26RHWBgAAG80GnyxXVZslOT3JX0+i7YuranFVLV6+fPmGrhoAAO6zyQThHyfZYcz4nOG0VbZNsnuSL1TVDUken2TRmk6Ya62d3Vpb0FpbMHv27PteNQAAbKDJBOHLk+xcVTtV1QOSPD/JolUzW2u3tdZmtdbmttbmJvlakoNba4s3SsUAADAC6wzCrbW7khyX5LNJrklyYWvtqqo6taoO3tgFAgDAxrD5ZBq11i5KctG4aa+doO3CDS8LAAA2LneWAwCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdmlQQrqqnV9W1VXVdVZ20hvmvqKqrq+rKqvqPqnr46EsFAIDRWWcQrqoZSc5IcmCSXZMcVlW7jmv2jSQLWmvzknw0yf8ddaEAADBKkzki/Lgk17XWrm+t3ZHk/CSHjG3QWruktfbfw9GvJZkz2jIBAGC0JhOEH5bkxjHjy4bTJnJMks9sSFEAALCxbT7KhVXVnydZkORJE8x/cZIXJ8mOO+44ylUDAMB6mcwR4R8n2WHM+JzhtHuoqqcmeXWSg1trv17TglprZ7fWFrTWFsyePfu+1AsAACMxmSB8eZKdq2qnqnpAkucnWTS2QVXtleQ9GYTgn4++TAAAGK11BuHW2l1Jjkvy2STXJLmwtXZVVZ1aVQcPm/1Dkm2SfKSqllbVogkWBwAA08Kk+gi31i5KctG4aa8dM/zUEdcFAAAblTvLAQDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuTSoIV9XTq+raqrquqk5aw/wtq+qC4fzLqmruqAsFAIBRWmcQrqoZSc5IcmCSXZMcVlW7jmt2TJJbWmuPSvLWJG8adaEAADBKkzki/Lgk17XWrm+t3ZHk/CSHjGtzSJIPDIc/muQpVVWjKxMAAEZrMkH4YUluHDO+bDhtjW1aa3cluS3JQ0ZRIAAAbAzVWlt7g6rnJHl6a+1Fw/G/SLJPa+24MW2+PWyzbDj+/WGbm8Yt68VJXjwc/aMk145qQzo1K8lN62xFr+wfTMS+wUTsG0zkd33feHhrbfb4iZtP4oE/TrLDmPE5w2lrarOsqjZPsl2Sm8cvqLV2dpKzJ1sxa1dVi1trC6a6DqYn+wcTsW8wEfsGE7m/7huT6RpxeZKdq2qnqnpAkucnWTSuzaIkRw6Hn5Pk821dh5oBAGAKrfOIcGvtrqo6Lslnk8xI8r7W2lVVdWqSxa21RUnem+Sfq+q6JP+VQVgGAIBpazJdI9JauyjJReOmvXbM8Mokzx1taUyCbiasjf2Didg3mIh9g4ncL/eNdZ4sBwAA90dusQwAQJcE4RGoqlZVbxkz/sqqet06HnPwmm5XfR/WfVRVLa+qpVV1VVV9tKoeuKHLZeOoqlcPX6crh6/ZPlV1clX9v+Paza+qa4bDN1TVl8bNXzq8bOH45c+dYPqpVfXUUW8PG9cm2l9+NZx/dVWdVVUTfi5U1euq6pWj2j42zBTsH+dW1RbDeQur6rbhvCur6uKq+oPhvLGfS0ur6tyN9yz0ZRO85ptV1Tuq6ttV9a2quryqdhrOu6iqHrSRt++rE0w/Z3g535EThEfj10meVVWzJvuA1tqi1tppI1r/Ba21+a213ZLckeTQES2XEaqqfZP8aZLHtNbmJXlqBjei+XDu/Zo9fzh9lW2raofhcnZZ33W31l7bWrv4PhU+CcPLJjJCm3B/+X5rbX6SeUl2TfJnIyifjWwK9o89Mrh86vPGzPvS8LNnXgZXmHrpmHmrPpfmt9aOWM/NYw020Wt+aJLtk8xrre2R5JlJbk2S1tpBrbVbR7EtE2mt7bcxl78mgvBo3JVBJ/ITx8+oqmdU1WVV9Y3hN+Y/HE4/qqreVVXbVdUPVx2Fqaqtq+rGqtqiqh5ZVf9WVUuq6ktV9cdrK2IYRrZOcstE6x5+2/teVc0ettmsqq6rqtnDfx8bfgO8vKqeMGzzpDHf7L9RVduO8snryEOT3NRa+3WStNZuaq39pLX23SS3VNU+Y9o+L/d8E7swd7/RHTZu3jqN/TY9PDpwSlVdMfzG/8fD6VtX1fuq6uvD1/mQ4fS5w/3viuG//YbTFw6nL0py9Xo/G6zLJt1fhncF/WqSRw1f888Pjzr9R1XtOLbt8L3pijHjO48dZ5PY1PvHb5J8Pfe+s2yqqpJsm+FnDxvNpnjNH5rkp6213w7Xsay1tipT3LDqgF9V/Z+quraqvlxVH67hL0VV9YWqemtVLa6qa6rqsVX1L8Pc8YZVK6mqVwyPOn+7ql4+ZvqK4d8aZqRrq+riJH9wH5+zdRKER+eMJIdX1Xbjpn85yeNba3slOT/J/zN2ZmvttiRLkzxpOOlPk3y2tXZnBuH6+Nba3klemeTdE6z70KpamsGNTX4/yb9OtO7hzv3BJIcP2zw1yTdba8uTvD3JW1trj03y7CT/NGzzyiQvHR4V2D/Jryb5nHBPn0uyQ1V9t6reXVVPGjPvwxledrCqHp/kv1pr3xsz/2NJnjUcfkbufo3vq5taa49JcmYGr2+SvDqDa4A/Lsn/TPIPVbV1kp8nOWDY/tAk7xiznMckeVlr7dEbWA/3tkn3lxp0qXpKkm8leWeSDwyPOn0o93zN01r7fpLbqmr+cNLRSd6/ntvHhtnU+8fMJPsk+bcxk/cffvb8KIPPkveNmXfomAMoR6/ntrFmm+I1vzDJM4av21uqaq/xDapqVUbYM8mBScbfZOOO4Y03zkryyQx+Kdg9yVFV9ZCq2juD94x9kjw+yV+uYT3PzOAOxLsmOSLJRjtSLAiPSGvtF0nOTXLCuFlzkny2qr6V5G+S7LaGh1+Qu7+pPT/JBVW1TQYv/EeGbzTvyeCb2ppcMAyp/yODD7G/Wce635fBjpUkL8zdH2BPTfKu4foWJfm9YR1fSXJ6VZ2Q5EHDI0esp9baiiR7Z3Cb8eUZvM5HDWdfkOQ5NfhlYPxPWsngTo23VNXzk1yT5L83sJx/Gf5dkmTucPhpSU4avhPNqWYAAAmESURBVP5fSDIzyY5Jtkjyj8P96CMZvDGt8vXW2g82sBbWYBPuL48cvuZfSfLp1tpnkuyb5Lzh/H9O8sQ1PO6fkhxdVTMyeP86bw1t2EimYP/4WQZHCq8cM29V14gdMvgc+b9j5o3tGuFL0ghsite8tbYsgwD6t0l+m+Q/quop45o9IcknW2srW2u/zL1D9aqbrn0ryVWttZ8Oj2Jfn8FdiJ+Y5OOttduH2/QvGRxkG+tPkny4tfab1tpPknx+oudlQ+nXN1pvS3JF7nlk5J1JTm+tLaqqhUlet4bHLUry91X1+xns5J/PoIvDrcOAOymttVZV/5rk+CSnTbTu1tqNVfWzqnpyksfl7qPDm2VwBHnluEWfVlWfTnJQkq9U1f9qrX1nsnVxt+HPi19I8oVhsDwyyTnD1+QHGfwy8OwMgsh4F2Twy8NRIyjl18O/v8nd7wOV5NmttWvHNqzBiZ8/y+Db/2ZJxu4ft4+gFiawifaX76/P+8wYH0tycgbvV0taazffh2WwATbl/jH8SfwrVXXw8EZa4y3KYJ9gI9oUr/kwtH4myWeq6mcZnDfwH+tR5qrPl9+OGV41Pu1ypyPCI9Ra+68MflY4Zszk7TLospDcfRvq8Y9bkcGJBm9P8qnhN6BfJPlBVT03Wd1fZs9JlPHEJN+fxLr/KYMuEh8Z/sdKBj+7HL+qwaqfPavqka21b7XW3jSsc619lVmzqvqjqtp5zKT5SX44ZvzDSd6a5Prht/LxPp7BEZfPbqQSP5vk+GF/v4z5qWq73N1n7C8yuMMkG9kU7y9fzd13CD08yZfGNxh+Yf5sBt1rHPHbxDb1/tFauynJSRkcKVyTsZ89bASb4jWvqsdU1fbD4c0yOIn2h+OafSWD7hMzh78a/+l6bsqXkvxZVT1w2P3umbn3e8x/ZtC9ZkZVPTSD7nobhSA8em9JMvbqEa/LoHvDkiQ3reVxFyT58+HfVQ5PckxVfTPJVUkOmeCxq/piXZlkrySvn8S6FyXZJvf8ADshyYIanCBzdZJjh9NfPuzQfmWSOzP4psj62ybJB2pwGaIrM+hi8Lox8z+SQfeVNZ7E0Fr7ZWvtTa21O9axnj+qqmVj/k32ro+vz6AbxJVVdVXu3o/eneTI4X74x3EUeFPZVPvLmhyfQbeHKzP48vOyCdp9KIOjPJ+7D+tgw0zF/vGJJA+sqlU/Y+8//Oz5Zgb7yV+v70awXjbFa/4HSf61BpdWuzKDiwG8a9xyLs8gQ1yZQR74VpLbJrsRrbUrkpyTwcmXlyX5p9baN8Y1+3iS72VwIva5SS6d7PLXlzvLdaqqFmRwYtz4fjkAk1KDM8W3a639n6muBdh0qmqb1tqKGpxk+59JXjwMuL9zpl1fDTa+GtzI469yd99ggPVSVR9P8sgkT57qWoBN7uyq2jWDk6o/8LsaghNHhAEA6JQ+wgAAdEkQBgCgS4IwAABdEoSB7lTVOVXVxvx77Lj5+42b/08TLes+rHvVMs/ZkLrX0uaocbVP9O8LG7IdAPcHgjDA4FbjYx2zxlYA3K8IwgDJYVW1VTK4PmaS501xPfdZa+2c1lqt+pfki2Nm7zRm3sIpKhFg2hCEgd79KIPbSD97OH5oBndw+tFED6iqZ1TVF6vqF1W1sqq+VVWvrKoZ49r9eVVdV1W/qqovVNUua1nmC6rq0qpaMWz/9ao6dATbt6Z1vXrYPeI3VbXjmOk7j+k6ceJw2g2rulJU1XOGd7VaWVXfqKonjVvullX12qq6Ztjm1qr6dFXtvTG2A2BDCcJA784Z/l3VPWJVt4j337tpUlV/lcHtRf8kybZJtkyye5J/SHLemHZPyeDWoI/M4KLzT0ry+QmWeWoGtyt+fJKth+0fm+T84d3bRu29GdwufbMkR42ZvurLwF0Zsy1D8zK4BfwuGWzz/CT/VlWPHG7D5hncbvWUDG7FvWUGXzAOSvKVqnrCRtgOgA0iCAO9Oz/J7UkWVtWfJtk3yS+TfGR8w6raNsmbhqM/TrJnkj/M3QH3eVW1cDh8SpJK8pskf5rkwUk+t4Zl7pTk74ajZyT5/WHbDw+nnVpVD77vm3dvrbX/L8nHh6NHVVUNh1cF4c+11n427mEPTvLXGYTbE4bTZo6p/bAk/3M4fGSSrZLslOSaDELxW0a5DQCjIAgDvVsVeiuDI7jJ3eF4vP0yOAqcJP/YWruytfbzJKeOafO0YReJxw3Hv9ha+3Rr7dYkr1vDMg9IsqpLxUuT/FeSWzIIlskgUD5+fTdqEs4c/t0pgy8BOyZZMJz2z2tof2Nr7W2ttV+01t6Z5Mbh9FVHeg8c0/YDSX6V5AcZHEFOksdV1QNHVj3ACAjCAIOuAsngqOfY8fFmjRm+cczwsjHDs4ftthiO/3jMvJ+sYZmzJ1Hf70+izXpprX0hg6O1yaBbyLOGw79I8sk1PGTZuPFV2/Ww4d91bUfl7ucXYFoQhIHutda+nOS7w9GrWmuXTdD0pjHDcyYYvmn4787h+MPGzNt+Hcv8s7FXfBhe9WGz1tqH1rkR981Zw7/PTnLEcPijrbVfraHtwyYYXxWIV23Hb5P8/gTb8eMATCOCMMDA6zM4EvrGtbS5NMmK4fBfVtUeVTU7yWvGtPlca+03SVaF6SdV1UFV9aCsuWvEv2cQHpPkDVU1r6oeUFVzq+qlw/kby7lJ/juD7hd7jZm2JjtW1fFVtW1VHZdkh+H0rwz//tvw72ZJzqqqhw2vIjGvqt6Y5K0boX6ADSIIAyRprX2wtfZnrbUPr6XNL5L87XB0TpIrk/w8yVOH0z7aWrtkOPy6JC2D/r+fzqDf7/9ewzKvz90n4O2e5JtJfp1B/9p3JXnUfd+qtRv2Wx67vT9M8p8TNL8pgzD7iyTvHE5bmeTvh8PnjXns8zLoSrEyg+35uyQPGlnhACMiCAOsh9bau5I8M8mXMjg6/OskVyd5Ve4+wS2ttf/IoLvBD4ZtvpzkKRMs8++S/HmSrw6X+ask1yX5YJKXbKRNWeXMMcMfbK1NdPvmqzLoR3x1kjuSLE1yYGvt+0nSWrszyf9KcnIGfY9/neS2JN9K8rYkp2+U6gE2QE38ngfA/d3wknH/mkH3jD9qrV03bv4NSR6ewdUvFm7yAgE2IkeEATpUVc+squ8l+cRw0kfGh2CA+7vNp7oAAKbEdhn0P16RwY0+jp3acgA2PV0jAADokq4RAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC69P8DZ4OCWiRr7f0AAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# set width of bar\n",
        "barWidth = 0.25\n",
        "fig = plt.subplots(figsize =(12, 8))\n",
        " \n",
        "# set height of bar\n",
        "sensitivity = [svm_linear_metric_1.sens(), svm_linear_metric_2.sens(), svm_linear_metric_del.sens()]\n",
        "specificity = [svm_linear_metric_1.spec(), svm_linear_metric_2.spec(), svm_linear_metric_del.spec()]\n",
        " \n",
        "# Set position of bar on X axis\n",
        "br_sensitivity = np.arange(len(sensitivity))\n",
        "br_specificity = [x + barWidth for x in br_sensitivity]\n",
        " \n",
        "# Make the plot\n",
        "plt.bar(br_sensitivity, sensitivity, width = barWidth,\n",
        "        edgecolor ='grey', label ='Sensitivity')\n",
        "plt.bar(br_specificity, specificity, width = barWidth,\n",
        "        edgecolor ='grey', label ='Specificity')\n",
        " \n",
        "# Adding Xticks\n",
        "plt.xlabel('Data Augmentation Method', fontweight ='bold', fontsize = 15)\n",
        "plt.xticks([r + barWidth for r in range(len(specificity))],\n",
        "        ['Method #1', 'Method #2', 'Deletion'])\n",
        " \n",
        "plt.legend()\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 502
        },
        "id": "mC19_ywDfj1e",
        "outputId": "2283813c-4261-4af3-f219-ea697cfd581a"
      },
      "execution_count": 60,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 864x576 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsIAAAHlCAYAAAD/U5rKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df7xcVX0v/M/XgEAF0ZJUhQjRCi2gIVwiFpTHXC0oPBVafxTRKiDKpRZRqL2lxaeCSh9srb8q/sCKiJaiV6tSxfqUilavAUkwooAoRZSIrYELSLAIyHr+mDnJ5HByckImhLDe79drXjN77bXXXrPPTPKZPWvWrtZaAACgNw/b1B0AAIBNQRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6tMWm2vHs2bPbvHnzNtXuAQDoxNKlS29qrc2ZXL7JgvC8efOyZMmSTbV7AAA6UVU/nKrc0AgAALokCAMA0CVBGACALm2yMcIAAD26++67s3z58tx5552buisPOVtvvXXmzp2bLbfcckb1BWEAgAfQ8uXLs91222XevHmpqk3dnYeM1lpuvvnmLF++PE94whNmtI2hEQAAD6A777wzO+ywgxA8ZlWVHXbYYb3OtAvCAAAPMCF441jf4yoIAwB05vTTT8+ee+6Z+fPnZ8GCBbn00kvH1vb++++fJLn++utz3nnnrSpfsmRJTjjhhGm3ff/7359zzz03SXLOOefkxhtvHFu/pmKMMADAJvTWt709d95x+9ja2/oR2+VPX3/SWtcvXrw4n/vc53L55Zdnq622yk033ZS77rprbPv/+te/nmR1EH7JS16SJFm4cGEWLlw47bbHHXfcqsfnnHNOnvzkJ2fHHXccW98mE4QBADahO++4PR/+r+kD4vo4OtNfufcnP/lJZs+ena222ipJMnv27CTJ0qVLc9JJJ2XlypWZPXt2zjnnnDzucY/LokWL8rSnPS0XX3xxbr311nzoQx/KAQcckCuvvDJHH3107rrrrtx777351Kc+lV133TXbbrttVq5cmZNPPjlXX311FixYkCOPPDJ777133va2t+WCCy7IE5/4xCxbtiyPetSjkiS77rprvva1r+V973tftt1221VXIH7pS1+abbbZJqeffno++MEP5jOf+UyS5F/+5V/y3ve+N5/+9Kc36FgZGgEA0JGDDjooN9xwQ3bbbbe8+tWvzle+8pXcfffdec1rXpNPfvKTWbp0aV7xilfklFNOWbXNPffck2984xt55zvfmdNOOy3JYBjDa1/72ixbtixLlizJ3Llz19jPGWeckQMOOCDLli3LiSeeuKr8YQ97WA477LBVIfbSSy/NLrvsksc85jGr6rzwhS/MwoUL8/d///dZtmxZDjnkkHz3u9/NihUrkiQf/vCH84pXvGKDj4UgDADQkW233TZLly7NWWedlTlz5uTwww/PBz7wgXznO9/JgQcemAULFuQtb3lLli9fvmqb5z//+UmSffbZJ9dff32SZL/99stf/uVf5q1vfWt++MMfZptttplxHw4//PB8/OMfT5Kcf/75Ofzww6etX1V52ctelo997GO59dZbs3jx4hx88MHr+czvy9AIAIDOzJo1K4sWLcqiRYvylKc8JWeeeWb23HPPLF68eMr6E8MoZs2alXvuuSdJ8pKXvCRPe9rT8vnPfz6HHHJIPvCBD+RZz3rWjPa/33775dprr82KFSvymc98Jm94wxvWuc3RRx+d5z3vedl6663zohe9KFtsseEx1hlhAICOXHPNNfn+97+/annZsmXZfffds2LFilVB+O67786VV145bTvXXXddnvjEJ+aEE07IYYcdliuuuGKN9dttt11uv33qHwFWVX7v934vJ510UnbffffssMMO96kzefsdd9wxO+64Y97ylrfk6KOPnvHznY4zwgAAHVm5cmVe85rX5NZbb80WW2yRJz3pSTnrrLNy7LHH5oQTTshtt92We+65J6973euy5557rrWdT3ziE/noRz+aLbfcMo997GPz53/+52usnz9/fmbNmpW99torRx11VPbee+811h9++OF56lOfmnPOOWfK9o866qgcd9xx2WabbbJ48eJss802eelLX5oVK1Zk99133+DjkCTVWpu+QtXZSX4nyU9ba0+eYn0leVeSQ5L8PMlRrbXL17XjhQsXtiVLpv9VIwDAQ83VV1+9RpB7oKdP25wdf/zx2XvvvXPMMcestc7k45skVbW0tXafqTlmckb4nCTvSXLuWtYfnGTX4e1pSd43vAcAYB0eqqF13PbZZ5884hGPyN/8zd+Mrc11BuHW2r9V1bxpqhyW5Nw2OLV8SVU9qqoe11r7yZj6CABA55YuXTr2NsfxY7mdktwwsrx8WAYAAA9aD+iP5arq2CTHJsnOO+/8QO4aAHiQGvcY2Qe7gw46KD/5j//I4x772E3dle6NIwj/OMnjR5bnDsvuo7V2VpKzksGP5cawbwBgMzfuSww/2O3ftkq7995N3Q0ynqERFyR5eQ38VpLbjA8GAODBbp1BuKr+IcniJL9RVcur6piqOq6qjhtWuTDJdUmuTfLBJK/eaL0FAGCDnX766dlzzz0zf/78LFiwIJdeeunY2j7kkENy6623Jkne/e53Z/fdd89LX/rSXHDBBTnjjDOm3Xb//fdPklx//fU577zzxtantZnJrBFHrGN9S/JHY+sRAEBH3vm2t+a2O+4cW3vbP2LrvO71f7rW9YsXL87nPve5XH755dlqq61y00035a677hrb/i+88MJVj9/73vfmoosuyty5c5Mkhx566LTbfv3rX0+yOgi/5CUvGVu/puLKcgAAm9Btd9yZN7a3j6290+6Yfl7in/zkJ5k9e3a22mqrJMns2bOTJPPmzcvv//7v5wtf+EK22WabnHfeeXnSk56UFStW5LjjjsuPfvSjJMk73/nOPP3pT191hbolS5akqvLGN74xL3jBCzJv3rwsWbIkb3jDG3Ldddfl4IMPzite8Yo8+tGPzpIlS/Ke97wn//mf/5njjjsu1113XZLkfe97X/bff/9su+22WblyZU4++eRcffXVWbBgQY488sh8+tOfzrvf/e4sWLAgSfKMZzwjZ555Zvbaa68NOlbjGCMMAMBm4qCDDsoNN9yQ3XbbLa9+9avzla98ZdW67bffPt/+9rdz/PHH53Wve12S5LWvfW1OPPHEXHbZZfnUpz6VV77ylUmSN7/5zavqX3HFFXnWs561xn7e//73Z8cdd8zFF1+cE088cY11J5xwQp75zGfmW9/6Vi6//PL7XMr5jDPOyAEHHJBly5blxBNPzDHHHLPqUszf+973cuedd25wCE4EYQCArmy77bZZunRpzjrrrMyZMyeHH374qpB5xBFHrLpfvHhxkuSiiy7K8ccfnwULFuTQQw/Nz372s6xcuTIXXXRR/uiPVo+OffSjHz3jPnzpS1/KH/7hHyZJZs2ale23337a+i960Yvyuc99LnfffXfOPvvsHHXUUevxjNfO0AgAgM7MmjUrixYtyqJFi/KUpzwlH/nIR5IkVbWqzsTje++9N5dcckm23nrrTdLXJPmVX/mVHHjggfnsZz+bT3ziE2O7ypwzwgAAHbnmmmvy/e9/f9XysmXLsssuuyRJPv7xj6+632+//ZIMhlL87d/+7Rr1k+TAAw/MmWeeuar8lltumXEfnv3sZ+d973tfkuSXv/xlbrvttjXWb7fddrn99jUvsvLKV74yJ5xwQp761Keu19nn6QjCAAAdWblyZY488sjssccemT9/fq666qqceuqpSQZhdv78+XnXu96Vd7zjHUkGU6AtWbIk8+fPzx577JH3v//9SZI3vOENueWWW/LkJz85e+21Vy6++OIZ9+Fd73pXLr744jzlKU/JPvvsk6uuumqN9fPnz8+sWbOy1157rerHPvvsk0c+8pE5+uijx3AUBmow+9kDb+HChW3JkiWbZN8AwIPHaaed1tWV5T546OOy57zHZscdd0zywE+ftjYTsz1MzCLxYHPjjTdm0aJF+e53v5uHPWzt53Kvvvrq7L777muUVdXS1tp9XmTGCAMAbEL3J7T25txzz80pp5ySt7/97dOG4PUlCHdg3J80Nwf399MwAPTq+uuv39RdWKuXv/zlefnLXz72dgXhDox7ou7NwbomE4fNlQ+2AOMjCANsRnywhc1fS8um+o3WQ936HlezRgAAPIB+eOvdueOOO4ThMWut5eabb16v+Y6dEQY2W29929tz5x23r7siwIPI3156S3aedVt++ctfbuquPORsvfXWmTt37ozrC8LAZuvOO27vasqlJDl6G9NOwubuZ7+4N5dc8o085znP2dRd6Z6hEQAAdEkQBgCgS4IwAABdEoQBAOhSlz+W80tzAAC6DMK9/dLcr8wBAO7L0AgAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF2aURCuqudW1TVVdW1VnTzF+p2r6uKq+mZVXVFVh4y/qwAAMD7rDMJVNSvJmUkOTrJHkiOqao9J1d6Q5BOttb2TvDjJe8fdUQAAGKeZnBHeN8m1rbXrWmt3JTk/yWGT6rQkjxw+3j7JjePrIgAAjN9MgvBOSW4YWV4+LBt1apI/qKrlSS5M8pqpGqqqY6tqSVUtWbFixf3oLgAAjMe4fix3RJJzWmtzkxyS5KNVdZ+2W2tntdYWttYWzpkzZ0y7BgCA9TeTIPzjJI8fWZ47LBt1TJJPJElrbXGSrZPMHkcHAQBgY5hJEL4sya5V9YSqengGP4a7YFKdHyV5dpJU1e4ZBGFjHwAAeNBaZxBurd2T5PgkX0xydQazQ1xZVW+qqkOH1f44yauq6ltJ/iHJUa21trE6DQAAG2qLmVRqrV2YwY/gRsv+YuTxVUmePt6uAQDAxuPKcgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgSzMKwlX13Kq6pqquraqT11Ln96vqqqq6sqrOG283AQBgvLZYV4WqmpXkzCQHJlme5LKquqC1dtVInV2T/FmSp7fWbqmqX9tYHQYAgHGYyRnhfZNc21q7rrV2V5Lzkxw2qc6rkpzZWrslSVprPx1vNwEAYLxmEoR3SnLDyPLyYdmo3ZLsVlX/u6ouqarnTtVQVR1bVUuqasmKFSvuX48BAGAMxvVjuS2S7JpkUZIjknywqh41uVJr7azW2sLW2sI5c+aMadcAALD+ZhKEf5zk8SPLc4dlo5YnuaC1dndr7QdJvpdBMAYAgAelmQThy5LsWlVPqKqHJ3lxkgsm1flMBmeDU1WzMxgqcd0Y+wkAAGO1ziDcWrsnyfFJvpjk6iSfaK1dWVVvqqpDh9W+mOTmqroqycVJ/qS1dvPG6jQAAGyodU6fliSttQuTXDip7C9GHrckJw1vAADwoOfKcgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdGlGQbiqnltV11TVtVV18jT1XlBVraoWjq+LAAAwfusMwlU1K8mZSQ5OskeSI6pqjynqbZfktUkuHXcnAQBg3GZyRnjfJNe21q5rrd2V5Pwkh01R781J3prkzjH2DwAANoqZBOGdktwwsrx8WLZKVf23JI9vrX1+jH0DAICNZoN/LFdVD0vy9iR/PIO6x1bVkqpasmLFig3dNQAA3G8zCcI/TvL4keW5w7IJ2yV5cpIvV9X1SX4ryQVT/WCutXZWa21ha23hnDlz7n+vAQBgA80kCF+WZNeqekJVPTzJi5NcMLGytXZba212a21ea21ekkuSHNpaW7JRegwAAGOwziDcWrsnyfFJvpjk6iSfaK1dWVVvqqpDN3YHAQBgY9hiJpVaaxcmuXBS2V+spe6iDe8WAABsXK4sBwBAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6NKMgXFXPraprquraqjp5ivUnVdVVVXVFVf1rVe0y/q4CAMD4rDMIV9WsJGcmOTjJHkmOqKo9JlX7ZpKFrbX5ST6Z5K/G3VEAABinmZwR3jfJta2161prdyU5P8lhoxVaaxe31n4+XLwkydzxdhMAAMZrJkF4pyQ3jCwvH5atzTFJvrAhnQIAgI1ti3E2VlV/kGRhkmeuZf2xSY5Nkp133nmcuwYAgPUykzPCP07y+JHlucOyNVTVbyc5JcmhrbVfTNVQa+2s1trC1trCOXPm3J/+AgDAWMwkCF+WZNeqekJVPTzJi5NcMFqhqvZO8oEMQvBPx99NAAAYr3UG4dbaPUmOT/LFJFcn+URr7cqqelNVHTqs9tdJtk3yv6pqWVVdsJbmAADgQWFGY4RbaxcmuXBS2V+MPP7tMfcLAAA2KleWAwCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIAwDQJUEYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECXBGEAALokCAMA0CVBGACALgnCAAB0SRAGAKBLgjAAAF0ShAEA6JIgDABAlwRhAAC6JAgDANAlQRgAgC4JwgAAdEkQBgCgS4IwAABdmlEQrqrnVtU1VXVtVZ08xfqtqurjw/WXVtW8cXcUAADGaZ1BuKpmJTkzycFJ9khyRFXtManaMUluaa09Kck7krx13B0FAIBxmskZ4X2TXNtau661dleS85McNqnOYUk+Mnz8ySTPrqoaXzcBAGC8ZhKEd0pyw8jy8mHZlHVaa/ckuS3JDuPoIAAAbAzVWpu+QtULkzy3tfbK4fLLkjyttXb8SJ3vDOssHy7/+7DOTZPaOjbJscPF30hyzbieCA9as5PctM5awIOd9zI8dPT4ft6ltTZncuEWM9jwx0keP7I8d1g2VZ3lVbVFku2T3Dy5odbaWUnOmmmP2fxV1ZLW2sJN3Q9gw3gvw0OH9/NqMxkacVmSXavqCVX18CQvTnLBpDoXJDly+PiFSb7U1nWqGQAANqF1nhFurd1TVccn+WKSWUnObq1dWVVvSrKktXZBkg8l+WhVXZvk/2QQlgEA4EFrnWOEYUNU1bHDITHAZsx7GR46vJ9XE4QBAOiSSywDANAlQbhDVdWq6mMjy1tU1Yqq+tw6tltQVYeMLJ9aVa/fgH5Mu31VPaKqLho+/tpwRpKJdf9cVbeuq8/wULe5v5+H/VhcVVdW1RVVdfj97QM8VFXVL6tq2fB98q2q+uOqmjbDVdW84fS266rzkpHlhVX17nH1e3MgCPfpjiRPrqpthssH5r5T4k1lQZJD1llrfPZLsriqHp3kjuHFWib8dZKXPYB9gQerzf39/PMkL2+t7ZnkuUneWVWPegD7BZuD/2qtLRi+Tw5McnCSN46h3XlJVgXh1tqS1toJY2h3syEI9+vCJP/38PERSf5hYsXwzM3ZVfWNqvpmVR02nDrvTUkOH34qnThrs0dVfbmqrquqE0baOKmqvjO8vW6k/JSq+l5VfS2Di6rcR1X9elUtS/KxDN6gS5PsNdzvryVJa+1fk9w+roMBm7nN9v3cWvtea+37SdJauzHJT5PcZ9J7YKC19tMMLk52fA3Mqqq/rqrLht+q/I/J20xT54wkBwzfjydW1aKJb5Oq6ler6jPD+pdU1fxh+anDf1Pu82/F5kgQ7tf5SV5cVVsnmZ/k0pF1p2QwF/S+Sf57Bmdft0zyF0k+PvxU+vFh3d9M8pwk+yZ5Y1VtWVX7JDk6ydOS/FaSV1XV3sPyF2f1mainTtWx1tq/t9YWZPAf5r5JPpLkmOF+fzq+QwAPGQ+J93NV7Zvk4Un+fcMOBzy0tdauy2BK219LckyS21prT83gffiqqnrCpE3WVufkJF8dvh/fMWmb05J8s7U2P8mfJzl3ZN19/q0Y7zN84MzkynI8BLXWrqiqeRmcPbpw0uqDkhw6Mt5v6yQ7r6Wpz7fWfpHkF1X10ySPSfKMJJ9urd2RJFX1j0kOyOCD16dbaz8flk++MMtkv9Zau3n4KfRD6/P8oCcPhfdzVT0uyUeTHNlau3cdbQGrHZRkflW9cLi8fZJdk3xvBnXumqbdZyR5QZK01r5UVTtU1SOH66b6t2L5WJ7NA0wQ7tsFSd6WZFGSHUbKK8kLWmvXjFauqqdN0cYvRh7/MmN4TVXV+zN4A84dfqW6a5LPVdVHpvjECgxstu/n4X+un09ySmvtkg3dJzzUVdUTM3iP/jSD9/hrWmtfnFRn3ujiWuosup9dGPu/FZuKoRF9OzvJaa21b08q/2KS11RVJUlV7T0svz3JdjNo96tJfreqfqWqHpHk94Zl/zYs36aqtkvyvKk2bq0dl8FXMm9O8rsZfPKc6msbYLXN8v08HK/86STnttY+OfOnC32qqjlJ3p/kPW1wMYgvJvnDieEJVbXb8L06am11pvt34KtJXjqsvyjJTa21n437+Wxqm22CZ8O11pYnmWqalDcneWeSK2owPcsPkvxOkouTnDw8q/P/TtPu5VV1TpJvDIv+rrX2zSSpqo8n+VYGn2Ivm6Z7z8xgPNIBSb4yeWVVfTWDMUrbVtXyDMYcfnFyPejFZvx+/v0k/1eSHarqqGHZUa21ZdO0B73ZZvhe3TLJPRkMI3r7cN3fZTD7w+XDD7wrMvjQOWptda5I8suq+laSc5J8c2SbU5OcXVVXZDC7y5HjflIPBq4sBwBAlwyNAACgS4IwAABdEoQBAOiSIAwAQJcEYQAAuiQIw0NcVZ1TVW14u7eq/quqllfVv1TVK4fzuN6fdh81vOb8qRswKfu69rHnSN9bVa11mq+HuqqaN3K8F2xAOxNtTJ5eKSPH+ZwN6uz969c5k/7WT520fv9J6//ufuxjrcewqhaNtH3UBj6ddfXj+uF+vrwx9wOsm3mEoS+VwSV2dxrefjvJUVX1O621W9ezrUcleePI8pfH0sM1HT5p+UVJ/mwj7GdzMC+rj/f1Se7vPLsTbXwkyWc2rEsb1Suy5tzEx4yhzXkZzzEEHiKcEYa+/PckWyXZJ8k/D/rykYUAAAnpSURBVMuenuTDm6xH03vRpOVfr6p9NklPOtBaq+HtqE3dlyRHVNU2SVJV22Zw4Q2AsRKEoTOttbtaa5cnOSyDq4wlg0vl7p0kVfUbVfWPVfWDqlpZVb+oqmur6q8mLts5/Or4ByPNvnHka+VFVbXF8KvuK6vqlqq6u6p+UlXnVdUTZtLPqpqfwdUDk+RjI6sOn1TvqNF9j5R/eVh2/aT6Jw6HhtxRVZ+uqv1Gtj91pN6qYQJV9fph/2+tqrdX1ayqevHwuNw2bGfOpP3sVFVnVdUNVXVXVd1YVX9XVY8dqTNvZD9vqqo3VtWPJ7c57NfFI81/eGS7eVU1p6r+oaq+V1U/G+7vR1X1gaqaPWxjUVWNXkHpyMlDAdY2NGJ4jL8xPGY/r6olVXX0pDqjQxvmV9XFw7rfr6o/mPz3XYcfJdk+yQuGy4cn2XZYPqWqek5V/evw2N1ZVd+uqj+qWnVp6WmP4aTmHl5VZ1TVf1bVTVV1dk26ZG1VPb2qLhy+vn8xPPZvmQjvI/UOGvblzuFxe/p6HgtgY2qtubm5PYRvGVw2sw1viyat+58j604elj13pGzy7ePDOkdNU2dRBsMv1rb++iRbz6Dfp49ss0+SK4ePfzCp3mhfFo2Uf3lifyNlr5iiPzeOPD51pO5E2c1TbHNBknsnlZ0/su1Ok9qd/PxnD+vNGym/dYq65w/rnTrN8ZyXwQeGta2/dNjGomnqHDXpOZ8z8lzeOs12b13L62zyc7k3yR7r8To9bXj/peG6r0/8fUbq/N3ItsdM08f3zPAYjh6f/5yizhkj+3teBpe5naqtrybZclhv9yS/mLR+ZZLbh4+/vKn/fXBz6/3mjDD07Xsjj3ce3n83yXOSPDbJw5PskOSDw3UvqqodWmvnJBk9s3taW/21+peT3J3kxUl2ySAUPyLJK4d1d0lyyAz6NjEs4settaVJ/mm4PK+q9p3Z01utqh6W1eNDf5bBkJDHJLlmHZs+KoP+7pzktmHZ8zIIa49Ocumw7PnDfSTJm5I8LslNSX4rg+EoB2QQinZJ8vop9rN1koOHffr2aJuttVMzGNYy4eiR4319kp8m+d0MAvhWGZxNPXVYd9+q+m+ttS+31mqkjY+MtHHOVE+8qn59pK/fTvLEDP7uE/17/bDOZF9OMjvJsRNNJXn+VPtYi/OT3JFkUVX9TpL9MgiP/2uKPm6b5O3DxX/M4Lhvm+RvhmWvrqrdZ3AMR81Ksm8Gz/U/hmUvHO6vkrx7WGdlkmcm+dWs/tbiGUleOnx8SgbvoWRwLLZP8q5h/4AHAUEY+lZTlP0kg7NjX8kgfNyc5FUj9XddV6OttV8m+ZUkn8wgDN6RZPRX/rtN26nBMI2J/fzTpPvkvj+im4m5WR32P9la+3pr7adJ3rKO7b7eWvtCa+2GDD4kJMldGZwNvTXJl4ZlW2YQYpNBoE0GYfCSDALwVzMIqcmagWzCZ1tr/zzs0xemaHM6t2RwTL+QwdnY27I6CCfrON7T+O2s/n/ib1prPxiGxomQ+bAkz55iuz9rrd2cNYe0PH499jsReivJucOyiXA82f5JHjl8/PwMXr8rk/zxsKwyeD2vjw+11i4bPtd/G5ZN9H+3DM4gJ4PX0b+11m5J8oaR7Q8a6VuSXNda+2Br7WdJ3pzB6wd4EBCEoW+jofaHw/u/zmBmht/I6uA2aut1NVpVL0hydpKnZuqzX+tqYzToXl1VT84gHK0clr1wYuznNGZNWn7cyOMfjzxevo52fjjy+M7h/YrW2sTj0VAzcbzWGC88hV+douz7U+xntM3pnJjkr5LMT7LNFOvX+Tdbi9kjj28YeTx6zKZ6rhPPZX2fx6gPDe8fPWl5snUd62Tq4z2dqf4WE2d21+eYTLzmVr3ehq+b/7Oe/QE2EkEYOlWD+YOPGyn64vB+YkjCd5LsPPw6/TVTNNGmaf6Fw/s7kzwtg6kan7Ie3RudLeJdGXwV/62sDtU7ZzDkIBmcbZ0wGvjmTWrzxpHHo6F4XWcq75lh2aibhvfLRr5+X3XL1GdoR9uc6thOd7wnjtd/ZDAu9WEZDN/YUDeNPJ67lsejdZIkrbV7hvfT9XlarbWvZfXQnStba5eupero/l83xbF+WGvt9IlmZ7j76f4W63NMfjK832liRVVtnfUP5sBGIghDZ6pqy+HQg89m9TjfT7fWJuZUnTjzdXeSO6rqN5L80RRN3TLy+DerasuR5Yk2WgZnch+VNb+qn65/CzMYi7ouE2eNR8/EPWfYxlFZM5gkg7NyE7MOvKCqFg5nZThlJv1aTxNT0y2oqv9ZVdtX1bbDmRvOzeoxpOtj9HjvUVWjZ7wnjvcvMzjeOyX503W086TJMxxM4aIMfuiWJH88nKFilyQnDcvuHdbZWN6cwev09GnqfD2D55wkfzKczWGrqtqxqo5McvlI3emO4Ux9L6u/JXhhVR1QVY/KYFz4hP9veP+/h/dPrKpXVdUjk/w/Wf33AjYxQRj6cnEGX+VfnsHsEEnytQxmU5jw+eH93hmMD/5uBmNV1zAc7zhxxu7wJHcNp6LaYqSNbZJclcEZspleDW10WMTvTjq7t2VWn22bGB5xaVaH4ddV1e0ZzIv8X5P6e28GP3BLBl+3X5bBj8x+c7TaDPu4Lm/MYOaBZDDrwq0ZhLWLk7ws9+9iRtdmdZD7kyT3VNXE85443jtlcCxuyJpnvUdNXKTi6Ul+PvybPWmqiq21f0/yjuHi/AymzLt++DhJ3jGss1G01j7WWvvd1to/TFPn9gyORzJ4/l/L4JuIH2cwE8Xo6266YzjTPrUkr83gQ8d2GYwhviXJy4dVFmf12OjTs3rozFkZjN0+KcnP12efwMYjCEN/fpFBSLgog2mnntXWvKrca5Ocl0F4uymDsadru7TxkRkEq/+aVP7hDM4A35jBD5z+MckRM+zfxNf8t2T1j8aSrPrK/VPDxZ2SPL21dlcGcyJ/Y9iP5Un+YLicSdufncGPqG4c1v2nJP9jpMotk7e5P1prP0qyMIPZNm7I4Oz6TzMISadk9Rnj9Wnz5xmE6Ctz3x9b/WWS92Tw97otg/G0r11LUydkMKvD7WtZP3m/r89gxo+lGRyzOzP4IPWq4bpNrrX2gQx+oPivGTz/X2QQ2v8xI2ff13EM12d/n03yrAyGE92Wwd/335OckeS3W2t3D+t9N8mhI/tblsEMJCvu776B8aoNGMIFsFkZXsxip+F0bBNTb30oq69atldr7YpN1T8AHlj35+s5gM3Vk5J8taruyODs72OyetjHB4RggL4YGgH05IYMfnz1swxC8J0Z/NjqmCR/uAn7BcAmYGgEAABdckYYAIAuCcIAAHRJEAYAoEuCMAAAXRKEAQDokiAMAECX/n+T4kUAcoRrNwAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LWrHXW7Wwj88"
      },
      "source": [
        "# Manual examination of model\n",
        "\n",
        "Sometimes it is helpful (or simply entertaining) to try inserting your own free text into a NLP model and seeing how it behaves"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eC6LqmCcr11r",
        "outputId": "c69eda9b-2e48-41d1-9923-7fa4ac3cb029"
      },
      "source": [
        "#@title Try entering some free text for classification via the classifiers we trained\n",
        "user_text = \"everything looks fine\" #@param {type:\"string\"}\n",
        "user_token = preprocess(nlp,user_text)\n",
        "user_sparse_vector = sparse_vector_create(train_vocab_dict.doc2bow(user_token),len(train_vocab_dict))\n",
        "user_nb_prediction = nb_classifier.predict(user_sparse_vector.reshape(1,-1))[0]\n",
        "user_svm_prediction = SVM_linear.predict(user_sparse_vector.reshape(1,-1))[0]\n",
        "\n",
        "print(\"Convention used - 0 is normal, 1 is abnormal\")\n",
        "print(f\"Naive Bayes: {user_nb_prediction}\")\n",
        "print(f\"SVM: {user_svm_prediction}\")"
      ],
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Convention used - 0 is normal, 1 is abnormal\n",
            "Naive Bayes: 1\n",
            "SVM: 1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AVnGEqzcO1ZE"
      },
      "source": [
        "## Error Analysis\n",
        "\n",
        "Manual examination of model errors for systematic errors is also an important post evaluation step."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "F17is6e3L1l5",
        "outputId": "2cf68a3b-d6bf-42fb-f13f-5bd64343e3b2"
      },
      "source": [
        "troubleshoot_df = test_df.iloc[svm_linear_metric.incorrect_index()][[\"full-text\",\"label\"]]\n",
        "troubleshoot_df[\"SVM\"] = [svm_linear_predictions[i] for i in svm_linear_metric.incorrect_index()]\n",
        "troubleshoot_df"
      ],
      "execution_count": 62,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  full-text  \\\n",
              "1140  The lungs are clear. No pleural effusion is seen. The heart and mediastinum are normal. Arthritic changes of the spine are present. No active disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  \n",
              "2869  Lungs are clear. No focal consolidation, effusion, or pneumothorax. Interval resolution of left effusion. Central venous dialysis catheter unchanged in position. Heart and mediastinal contours are normal. Osseous structures intact. No acute cardiopulmonary disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                               \n",
              "2164  Lungs are clear bilaterally. Cardiac and mediastinal silhouettes are normal. Pulmonary vasculature is normal. No pneumothorax or pleural effusion. Limbus vertebra noted within the partial visualized lumbar vertebral body. No acute cardiopulmonary disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         \n",
              "2836  No there is an dextroscoliosis of the thoracic spine. The cardiomediastinal silhouette and pulmonary vasculature are within normal limits. There is no pneumothorax or pleural effusion. There are no focal areas of consolidation. No acute cardiopulmonary abnormality.                                                                                                                                                                                                                                                                                                                                                                                                                                                                               \n",
              "2854  The heart is normal in size and contour. The lungs are clear, without evidence of infiltrate. There is no pneumothorax or effusion. Multiple punctate round XXXX XXXX over the abdomen on the lateral view. These may reside within, or outside of the patient. No acute cardiopulmonary disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                       \n",
              "3307  No focal lung opacity, pleural effusion or pneumothorax. Cardiomediastinal silhouette is unremarkable. No focal lung opacity, pleural effusion of pneumothorax.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         \n",
              "3125  PA and lateral views of the chest were obtained. The cardiomediastinal silhouette is within limits. Postoperative changes from spinal rods are demonstrated. There is elevation of the left hemidiaphragm. Multiple colonic loops are demonstrated in the left upper quadrant. The lungs are clear bilaterally. Left humeral head is positioned anterior and inferior to the glenoid, concerning for anterior shoulder subluxation. 1. No active cardiopulmonary disease. 2. Left humeral head is positioned anterior and inferior to the glenoid, concerning for anterior shoulder subluxation. This is XXXX related to the muscular dystrophy and decreased shoulder muscles support. 3. XXXX postoperative changes from the spinal XXXX placement.   \n",
              "3203  There may be a subtle airspace opacity in the right base near the midclavicular line. There is no pleural effusion or pneumothorax. The heart and mediastinum are normal. The skeletal structures are normal. Possible area of pneumonitis right lower lobe.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            \n",
              "3298  Frontal and lateral views of the chest show an unchanged cardiomediastinal silhouette. Normal mediastinal contour, pulmonary XXXX and vasculature, central airways and aeration of the lungs. No focal airspace consolidation or pleural effusion. There is subphrenic intraperitoneal extraluminal XXXX free XXXX. 1. No acute or active cardiac, pulmonary or pleural disease. 2. Pneumoperitoneum, attributed to the patient's recent abdominal surgery.                                                                                                                                                                                                                                                                                             \n",
              "1112  Heart size within normal limits, stable mediastinal and hilar contours. No focal alveolar consolidation, no definite pleural effusion seen. No typical findings of pulmonary edema. XXXX convexity also present on the previous exam. No acute findings                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 \n",
              "1172   Heart size upper limits normal. Sternotomy. Right-sided aortic XXXX. Lungs are clear.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  \n",
              "2322  The mediastinal silhouette is widened with overlying sternotomy XXXX. The heart size is normal. The lungs are clear without evidence of effusion, infiltrate or pneumothorax. Visualized bony structures are intact with no acute abnormalities. 1. Wide mediastinal XXXX, consistent with history of aortic dissection. 2. Otherwise normal chest x-XXXX.                                                                                                                                                                                                                                                                                                                                                                                              \n",
              "2743                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
              "3089   Heart size normal. Mediastinal silhouette and vascularity are within normal limits. Lungs are clear, hyperinflated. There is no pleural effusion or pneumothorax.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
              "3218  The XXXX examination consists of frontal and lateral radiographs of the chest. Atherosclerotic calcifications of the aortic XXXX are again seen. The cardiomediastinal contours are within normal limits. Pulmonary vascularity is within normal limits. No focal consolidation, pleural effusion, or pneumothorax identified. The visualized osseous structures and upper abdomen are unremarkable. No evidence of acute cardiopulmonary process.                                                                                                                                                                                                                                                                                                      \n",
              "1258  Lungs are clear. No pleural effusions or pneumothoraces. Heart and mediastinum of normal size and contour. Degenerative changes in the thoracic spine. Clear lungs.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     \n",
              "3636                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
              "2321  The heart size and pulmonary vascularity appear within normal limits. Lungs are free of focal airspace disease. No pleural effusion or pneumothorax is seen. VP shunt tubing is identified. The bony structures, as visualized, appear unremarkable. No evidence of active disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                     \n",
              "775   The lungs are hypoventilated. There is no focal airspace opacity. The cardiomediastinal silhouette is normal in size. There is no pneumothorax or large pleural effusion. No acute cardiopulmonary abnormality.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         \n",
              "151   2 images. Vague density within the lower lung XXXX is secondary to soft tissue overlap. Heart size and pulmonary vascular engorgement appear within limits of normal. Mediastinal contour is unremarkable. No focal consolidation, pleural effusion, or pneumothorax identified. No convincing acute bony findings. No acute cardiopulmonary abnormality identified.                                                                                                                                                                                                                                                                                                                                                                                    \n",
              "2088  The lungs are clear. There is no pleural effusion or pneumothorax. The heart and mediastinum are normal. The skeletal structures short thready changes of the spine. No acute pulmonary disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        \n",
              "1131  Artifact in the region of the central upper abdomen. No focal areas of consolidation. No pleural effusions. No evidence of pneumothorax. Heart size within normal limits. Osseous structures intact. Limited exam secondary to artifact within the upper abdomen (this does not represent free intra-abdominal XXXX). Recommend repeat chest x-XXXX.                                                                                                                                                                                                                                                                                                                                                                                                    \n",
              "616   The lungs are clear. The heart and pulmonary XXXX appear normal. Pleural spaces are clear. The mediastinal contours are normal. Bony overlap in the lung apices could obscure a small pulmonary nodule. No acute cardiopulmonary disease                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                \n",
              "1890  Normal heart size and mediastinal contours. The lungs are hyperinflated but clear. No pneumothorax or pleural effusion. No acute bony abnormalities. No acute cardiopulmonary process. .                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                \n",
              "2779   Rib films. No fractures or dislocations. Chest. Heart size normal. Lungs are clear. No effusion or pneumothorax. Minimal degenerative disease thoracic spine                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
              "3824                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
              "\n",
              "         label  SVM  \n",
              "1140  normal    1    \n",
              "2869  abnormal  0    \n",
              "2164  normal    1    \n",
              "2836  abnormal  0    \n",
              "2854  normal    1    \n",
              "3307  normal    1    \n",
              "3125  normal    1    \n",
              "3203  normal    1    \n",
              "3298  abnormal  0    \n",
              "1112  normal    1    \n",
              "1172  normal    1    \n",
              "2322  abnormal  0    \n",
              "2743  normal    1    \n",
              "3089  abnormal  0    \n",
              "3218  abnormal  0    \n",
              "1258  normal    1    \n",
              "3636  normal    1    \n",
              "2321  abnormal  0    \n",
              "775   abnormal  0    \n",
              "151   abnormal  0    \n",
              "2088  normal    1    \n",
              "1131  abnormal  0    \n",
              "616   abnormal  0    \n",
              "1890  abnormal  0    \n",
              "2779  normal    1    \n",
              "3824  normal    1    "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a421223c-ad71-4870-b4f3-a029fdeb9991\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>full-text</th>\n",
              "      <th>label</th>\n",
              "      <th>SVM</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1140</th>\n",
              "      <td>The lungs are clear. No pleural effusion is seen. The heart and mediastinum are normal. Arthritic changes of the spine are present. No active disease.</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2869</th>\n",
              "      <td>Lungs are clear. No focal consolidation, effusion, or pneumothorax. Interval resolution of left effusion. Central venous dialysis catheter unchanged in position. Heart and mediastinal contours are normal. Osseous structures intact. No acute cardiopulmonary disease.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2164</th>\n",
              "      <td>Lungs are clear bilaterally. Cardiac and mediastinal silhouettes are normal. Pulmonary vasculature is normal. No pneumothorax or pleural effusion. Limbus vertebra noted within the partial visualized lumbar vertebral body. No acute cardiopulmonary disease.</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2836</th>\n",
              "      <td>No there is an dextroscoliosis of the thoracic spine. The cardiomediastinal silhouette and pulmonary vasculature are within normal limits. There is no pneumothorax or pleural effusion. There are no focal areas of consolidation. No acute cardiopulmonary abnormality.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2854</th>\n",
              "      <td>The heart is normal in size and contour. The lungs are clear, without evidence of infiltrate. There is no pneumothorax or effusion. Multiple punctate round XXXX XXXX over the abdomen on the lateral view. These may reside within, or outside of the patient. No acute cardiopulmonary disease.</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3307</th>\n",
              "      <td>No focal lung opacity, pleural effusion or pneumothorax. Cardiomediastinal silhouette is unremarkable. No focal lung opacity, pleural effusion of pneumothorax.</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3125</th>\n",
              "      <td>PA and lateral views of the chest were obtained. The cardiomediastinal silhouette is within limits. Postoperative changes from spinal rods are demonstrated. There is elevation of the left hemidiaphragm. Multiple colonic loops are demonstrated in the left upper quadrant. The lungs are clear bilaterally. Left humeral head is positioned anterior and inferior to the glenoid, concerning for anterior shoulder subluxation. 1. No active cardiopulmonary disease. 2. Left humeral head is positioned anterior and inferior to the glenoid, concerning for anterior shoulder subluxation. This is XXXX related to the muscular dystrophy and decreased shoulder muscles support. 3. XXXX postoperative changes from the spinal XXXX placement.</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3203</th>\n",
              "      <td>There may be a subtle airspace opacity in the right base near the midclavicular line. There is no pleural effusion or pneumothorax. The heart and mediastinum are normal. The skeletal structures are normal. Possible area of pneumonitis right lower lobe.</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3298</th>\n",
              "      <td>Frontal and lateral views of the chest show an unchanged cardiomediastinal silhouette. Normal mediastinal contour, pulmonary XXXX and vasculature, central airways and aeration of the lungs. No focal airspace consolidation or pleural effusion. There is subphrenic intraperitoneal extraluminal XXXX free XXXX. 1. No acute or active cardiac, pulmonary or pleural disease. 2. Pneumoperitoneum, attributed to the patient's recent abdominal surgery.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1112</th>\n",
              "      <td>Heart size within normal limits, stable mediastinal and hilar contours. No focal alveolar consolidation, no definite pleural effusion seen. No typical findings of pulmonary edema. XXXX convexity also present on the previous exam. No acute findings</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1172</th>\n",
              "      <td>Heart size upper limits normal. Sternotomy. Right-sided aortic XXXX. Lungs are clear.</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2322</th>\n",
              "      <td>The mediastinal silhouette is widened with overlying sternotomy XXXX. The heart size is normal. The lungs are clear without evidence of effusion, infiltrate or pneumothorax. Visualized bony structures are intact with no acute abnormalities. 1. Wide mediastinal XXXX, consistent with history of aortic dissection. 2. Otherwise normal chest x-XXXX.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2743</th>\n",
              "      <td></td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3089</th>\n",
              "      <td>Heart size normal. Mediastinal silhouette and vascularity are within normal limits. Lungs are clear, hyperinflated. There is no pleural effusion or pneumothorax.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3218</th>\n",
              "      <td>The XXXX examination consists of frontal and lateral radiographs of the chest. Atherosclerotic calcifications of the aortic XXXX are again seen. The cardiomediastinal contours are within normal limits. Pulmonary vascularity is within normal limits. No focal consolidation, pleural effusion, or pneumothorax identified. The visualized osseous structures and upper abdomen are unremarkable. No evidence of acute cardiopulmonary process.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1258</th>\n",
              "      <td>Lungs are clear. No pleural effusions or pneumothoraces. Heart and mediastinum of normal size and contour. Degenerative changes in the thoracic spine. Clear lungs.</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3636</th>\n",
              "      <td></td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2321</th>\n",
              "      <td>The heart size and pulmonary vascularity appear within normal limits. Lungs are free of focal airspace disease. No pleural effusion or pneumothorax is seen. VP shunt tubing is identified. The bony structures, as visualized, appear unremarkable. No evidence of active disease.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>775</th>\n",
              "      <td>The lungs are hypoventilated. There is no focal airspace opacity. The cardiomediastinal silhouette is normal in size. There is no pneumothorax or large pleural effusion. No acute cardiopulmonary abnormality.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>151</th>\n",
              "      <td>2 images. Vague density within the lower lung XXXX is secondary to soft tissue overlap. Heart size and pulmonary vascular engorgement appear within limits of normal. Mediastinal contour is unremarkable. No focal consolidation, pleural effusion, or pneumothorax identified. No convincing acute bony findings. No acute cardiopulmonary abnormality identified.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2088</th>\n",
              "      <td>The lungs are clear. There is no pleural effusion or pneumothorax. The heart and mediastinum are normal. The skeletal structures short thready changes of the spine. No acute pulmonary disease.</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1131</th>\n",
              "      <td>Artifact in the region of the central upper abdomen. No focal areas of consolidation. No pleural effusions. No evidence of pneumothorax. Heart size within normal limits. Osseous structures intact. Limited exam secondary to artifact within the upper abdomen (this does not represent free intra-abdominal XXXX). Recommend repeat chest x-XXXX.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>616</th>\n",
              "      <td>The lungs are clear. The heart and pulmonary XXXX appear normal. Pleural spaces are clear. The mediastinal contours are normal. Bony overlap in the lung apices could obscure a small pulmonary nodule. No acute cardiopulmonary disease</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1890</th>\n",
              "      <td>Normal heart size and mediastinal contours. The lungs are hyperinflated but clear. No pneumothorax or pleural effusion. No acute bony abnormalities. No acute cardiopulmonary process. .</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2779</th>\n",
              "      <td>Rib films. No fractures or dislocations. Chest. Heart size normal. Lungs are clear. No effusion or pneumothorax. Minimal degenerative disease thoracic spine</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3824</th>\n",
              "      <td></td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a421223c-ad71-4870-b4f3-a029fdeb9991')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-a421223c-ad71-4870-b4f3-a029fdeb9991 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-a421223c-ad71-4870-b4f3-a029fdeb9991');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 62
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "vrA-F4wQOjJ_",
        "outputId": "6ab0163f-9a09-4ad0-c53e-58b526a71a8b"
      },
      "source": [
        "troubleshoot_df = test_df.iloc[nb_metric.incorrect_index()][[\"full-text\",\"label\"]]\n",
        "troubleshoot_df[\"nb\"] = [nb_predictions[i] for i in nb_metric.incorrect_index()]\n",
        "troubleshoot_df"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  full-text  \\\n",
              "1140  The lungs are clear. No pleural effusion is seen. The heart and mediastinum are normal. Arthritic changes of the spine are present. No active disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  \n",
              "944   The heart is normal in size. The mediastinum is unremarkable. The lungs are hypoinflated but clear. No acute disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   \n",
              "2469  The heart is normal in size with normal appearance of the cardiomediastinal silhouette. The lungs are clear without focal airspace opacity, pleural effusion, or pneumothorax. The osseous structures are intact, with mild degenerative changes in the thoracic spine. No acute cardiopulmonary finding.                                                                                                                                                                                                                                                                                                                                                                                                                                               \n",
              "2836  No there is an dextroscoliosis of the thoracic spine. The cardiomediastinal silhouette and pulmonary vasculature are within normal limits. There is no pneumothorax or pleural effusion. There are no focal areas of consolidation. No acute cardiopulmonary abnormality.                                                                                                                                                                                                                                                                                                                                                                                                                                                                               \n",
              "2854  The heart is normal in size and contour. The lungs are clear, without evidence of infiltrate. There is no pneumothorax or effusion. Multiple punctate round XXXX XXXX over the abdomen on the lateral view. These may reside within, or outside of the patient. No acute cardiopulmonary disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                       \n",
              "3307  No focal lung opacity, pleural effusion or pneumothorax. Cardiomediastinal silhouette is unremarkable. No focal lung opacity, pleural effusion of pneumothorax.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         \n",
              "78    The lungs are clear. Heart size is normal. No pneumothorax. There are endplate changes in the spine. Clear lungs. No acute cardiopulmonary abnormality. .                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               \n",
              "1594  Lungs are hyperexpanded bilaterally, with no focal consolidation, pleural effusion, or pneumothoraces. Cardiomediastinal silhouette is within normal limits. XXXX are unremarkable. No acute cardiopulmonary abnormality.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               \n",
              "3125  PA and lateral views of the chest were obtained. The cardiomediastinal silhouette is within limits. Postoperative changes from spinal rods are demonstrated. There is elevation of the left hemidiaphragm. Multiple colonic loops are demonstrated in the left upper quadrant. The lungs are clear bilaterally. Left humeral head is positioned anterior and inferior to the glenoid, concerning for anterior shoulder subluxation. 1. No active cardiopulmonary disease. 2. Left humeral head is positioned anterior and inferior to the glenoid, concerning for anterior shoulder subluxation. This is XXXX related to the muscular dystrophy and decreased shoulder muscles support. 3. XXXX postoperative changes from the spinal XXXX placement.   \n",
              "664   The heart size is normal. The mediastinal contour is within normal limits. The lungs are free of any focal infiltrates. There are no nodules or masses. No visible pneumothorax. No visible pleural fluid. The XXXX are grossly normal. There is no visible free intraperitoneal air under the diaphragm. Surgical clips are seen the right upper quadrant. 1. No acute radiographic cardiopulmonary process.                                                                                                                                                                                                                                                                                                                                           \n",
              "3203  There may be a subtle airspace opacity in the right base near the midclavicular line. There is no pleural effusion or pneumothorax. The heart and mediastinum are normal. The skeletal structures are normal. Possible area of pneumonitis right lower lobe.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            \n",
              "3298  Frontal and lateral views of the chest show an unchanged cardiomediastinal silhouette. Normal mediastinal contour, pulmonary XXXX and vasculature, central airways and aeration of the lungs. No focal airspace consolidation or pleural effusion. There is subphrenic intraperitoneal extraluminal XXXX free XXXX. 1. No acute or active cardiac, pulmonary or pleural disease. 2. Pneumoperitoneum, attributed to the patient's recent abdominal surgery.                                                                                                                                                                                                                                                                                             \n",
              "1112  Heart size within normal limits, stable mediastinal and hilar contours. No focal alveolar consolidation, no definite pleural effusion seen. No typical findings of pulmonary edema. XXXX convexity also present on the previous exam. No acute findings                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 \n",
              "1045  The Heart size is normal. Cardiomediastinal silhouette is normal in contour. The lungs are clear bilaterally. Lateral views obscured by patient body habitus. There is no evidence of apical disease. XXXX are unchanged from previous exam and appear normal. Thoracic spine shows osteophyte formations at several levels. no radiographic evidence of tuberculosis or sarcoidosis.                                                                                                                                                                                                                                                                                                                                                                   \n",
              "2968  The heart, pulmonary XXXX and mediastinum are within normal limits. There is no pleural effusion or pneumothorax. There is no focal air space opacity to suggest a pneumonia. There are degenerative changes of the spine. No acute cardiopulmonary disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            \n",
              "1172   Heart size upper limits normal. Sternotomy. Right-sided aortic XXXX. Lungs are clear.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  \n",
              "363   The heart size and pulmonary vascularity appear within normal limits. Lungs are free of focal airspace disease. No pleural effusion or pneumothorax is seen. XXXX XXXX foreign body is noted in the soft tissues of the left chest wall. No evidence of active disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 \n",
              "2322  The mediastinal silhouette is widened with overlying sternotomy XXXX. The heart size is normal. The lungs are clear without evidence of effusion, infiltrate or pneumothorax. Visualized bony structures are intact with no acute abnormalities. 1. Wide mediastinal XXXX, consistent with history of aortic dissection. 2. Otherwise normal chest x-XXXX.                                                                                                                                                                                                                                                                                                                                                                                              \n",
              "2288  The lungs are clear without focal consolidation, effusion, or pneumothorax. Normal heart size. Unchanged multiple XXXX foreign bodies overlying the left clavicle and midline in the posterior soft tissues. The bony thorax is grossly intact. Negative for acute cardiopulmonary abnormality.                                                                                                                                                                                                                                                                                                                                                                                                                                                         \n",
              "2743                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
              "3089   Heart size normal. Mediastinal silhouette and vascularity are within normal limits. Lungs are clear, hyperinflated. There is no pleural effusion or pneumothorax.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
              "318   The XXXX examination consists of frontal and lateral radiographs of the chest. There are diminished lung volumes. The cardiomediastinal contours are within normal limits. Pulmonary vascularity is within normal limits. No focal consolidation, pleural effusion, or pneumothorax identified. XXXX scoliosis is unchanged. Visualized upper abdomen is grossly unremarkable. No evidence of acute cardiopulmonary process or significant interval change.                                                                                                                                                                                                                                                                                             \n",
              "3486  The XXXX and soft tissue appear normal. The cardiac silhouette and mediastinum size are normal. The aortic XXXX is on the left. The trachea is well seen and appears normal. The lungs are clear. Unremarkable chest study. .                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
              "242   There's been interval enlargement in the cardiac silhouette. These XXXX't contours are within normal limits. The lungs are normally inflated and clear. Osseous structures are within normal limits for patient age. 1. No acute radiographic cardiopulmonary process. 2. Interval development of cardiomegaly.                                                                                                                                                                                                                                                                                                                                                                                                                                         \n",
              "155   The trachea is midline. The cardiomediastinal silhouette is normal and unchanged compared to prior examination. The lungs are clear, without evidence of acute infiltrate or effusion. There is no pneumothorax. The visualized bony structures reveal no acute abnormalities. There is hiatal hernia. No acute cardiopulmonary abnormalities.                                                                                                                                                                                                                                                                                                                                                                                                          \n",
              "580   The heart, pulmonary XXXX and mediastinum are within normal limits. There is no pleural effusion or pneumothorax. There is no focal air space opacity to suggest a pneumonia. There are minimal degenerative changes of the spine. No acute cardiopulmonary disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    \n",
              "3218  The XXXX examination consists of frontal and lateral radiographs of the chest. Atherosclerotic calcifications of the aortic XXXX are again seen. The cardiomediastinal contours are within normal limits. Pulmonary vascularity is within normal limits. No focal consolidation, pleural effusion, or pneumothorax identified. The visualized osseous structures and upper abdomen are unremarkable. No evidence of acute cardiopulmonary process.                                                                                                                                                                                                                                                                                                      \n",
              "3636                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
              "2321  The heart size and pulmonary vascularity appear within normal limits. Lungs are free of focal airspace disease. No pleural effusion or pneumothorax is seen. VP shunt tubing is identified. The bony structures, as visualized, appear unremarkable. No evidence of active disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                     \n",
              "775   The lungs are hypoventilated. There is no focal airspace opacity. The cardiomediastinal silhouette is normal in size. There is no pneumothorax or large pleural effusion. No acute cardiopulmonary abnormality.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         \n",
              "2076  No The cardiomediastinal silhouette and pulmonary vasculature are within normal limits. There is no pneumothorax or pleural effusion. There are no focal areas of consolidation. There are T-spine osteophytes. No acute cardiopulmonary abnormality.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   \n",
              "151   2 images. Vague density within the lower lung XXXX is secondary to soft tissue overlap. Heart size and pulmonary vascular engorgement appear within limits of normal. Mediastinal contour is unremarkable. No focal consolidation, pleural effusion, or pneumothorax identified. No convincing acute bony findings. No acute cardiopulmonary abnormality identified.                                                                                                                                                                                                                                                                                                                                                                                    \n",
              "1802   Well-expanded and clear lungs. Apical oligemia suggestive of some emphysema. Mediastinal contour within normal limits. No acute cardiopulmonary abnormality identified.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                \n",
              "3320  Frontal and lateral views of the chest with overlying external cardiac monitor leads show an unchanged cardiomediastinal silhouette. No XXXX focal airspace consolidation or pleural effusion. No acute or active cardiac, pulmonary or pleural disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                \n",
              "638   Trachea is midline. The cardiomediastinal silhouette is normal. The lungs are clear, without evidence of acute infiltrate or effusion. There is no pneumothorax. The visualized bony structures show no acute abnormalities. Lateral view reveals mild degenerative changes of the thoracic spine. No acute cardiopulmonary abnormalities.                                                                                                                                                                                                                                                                                                                                                                                                              \n",
              "3257  Sternotomy sutures and bypass grafts have been placed in the interval. Both lungs remain clear and expanded with no infiltrates. Pulmonary XXXX are normal. Post operative chest with no acute disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 \n",
              "122   Cardiac and mediastinal contours are within normal limits. Prior granulomatous disease. The lungs are otherwise clear. Bony structures are intact. No acute findings.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   \n",
              "2329  Heart size and mediastinal contours appear normal limits. No focal pulmonary opacity, pleural effusion or pneumothorax. No acute bony abnormality. Degenerative changes of the spine. No acute cardiopulmonary abnormality.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             \n",
              "2426  Heart size and pulmonary vascularity appear within normal limits. The lungs are free of focal airspace disease. No pleural effusion or pneumothorax is seen. Vascular calcification is noted. 1. No evidence of active disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         \n",
              "1320  There are low lung volumes. The lungs are clear. No focal airspace consolidation. No pleural effusion or pneumothorax. Heart size and mediastinal contour appear within normal limits. No acute abnormality demonstrated.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               \n",
              "968   Spinal stimulator in XXXX. Lungs are clear without focal airspace disease. No pleural effusions or pneumothoraces. Heart and mediastinum of normal size and contour. Degenerative changes in the thoracic spine. Clear lungs.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
              "533   Frontal and lateral views of the chest show an unchanged cardiomediastinal silhouette. Normal mediastinal contour, pulmonary XXXX and vasculature, central airways and aeration of the lungs. No pleural effusion. There are XXXX spine marginal osteophytes. Stable chest. No acute disease process identified.                                                                                                                                                                                                                                                                                                                                                                                                                                        \n",
              "2078  Cardiac and mediastinal contours are within normal limits. The lungs are clear. Mild prominence left hilar contour. Bony structures are intact. No acute preoperative findings.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         \n",
              "3441  Lungs are clear. Heart size is normal. No pneumothorax. There are surgical clips seen within the upper abdomen. Clear lungs. No acute cardiopulmonary abnormality. .                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    \n",
              "2835  The lungs are clear bilaterally. Specifically, no evidence of focal consolidation, pneumothorax, or pleural effusion.. Cardio mediastinal silhouette is unremarkable. Visualized osseous structures of the thorax demonstrates stable mild multilevel degenerative disc disease of the thoracolumbar spine as well as chronic left-sided rib fractures without acute abnormality. No acute cardiopulmonary abnormality.                                                                                                                                                                                                                                                                                                                                 \n",
              "1881  The heart and lungs have XXXX XXXX in the interval. Both lungs are clear and expanded. Heart and mediastinum unremarkable. No change in the mediastinal surgical clips. No active disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              \n",
              "1448  The heart is normal in size and contour. There is no mediastinal widening. The lungs are hyperexpanded. No focal airspace disease. No large pleural effusion or pneumothorax. Exaggerated kyphosis. No acute cardiopulmonary abnormalities.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             \n",
              "280   Heart size and pulmonary vascularity appear within normal limits. The lungs are free of focal airspace disease. No pleural effusion or pneumothorax is seen. Degenerative changes are present in the spine. 1. No evidence of active disease.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
              "1131  Artifact in the region of the central upper abdomen. No focal areas of consolidation. No pleural effusions. No evidence of pneumothorax. Heart size within normal limits. Osseous structures intact. Limited exam secondary to artifact within the upper abdomen (this does not represent free intra-abdominal XXXX). Recommend repeat chest x-XXXX.                                                                                                                                                                                                                                                                                                                                                                                                    \n",
              "2254  The lungs are clear. The cardiomediastinal silhouette is within normal limits. There is ectasia of the thoracic aorta. No pleural effusion is identified. Normal chest film.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            \n",
              "1851   Heart size is normal lungs are clear. No evidence of tuberculosis. Minimal scoliosis.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  \n",
              "1899  Heart size within normal limits. Trachea is midline. The lung volumes are is somewhat low. Both lungs are otherwise clear bilaterally. No pleural effusion. No pulmonary nodules visualized. No pulmonary nodules. Negative chest.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
              "3040  There are no focal areas of consolidation. No suspicious bony opacities. Heart size within normal limits. No pleural effusions. No evidence of pneumothorax. Mild degenerative changes of the thoracic spine. No acute cardiopulmonary abnormality.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     \n",
              "616   The lungs are clear. The heart and pulmonary XXXX appear normal. Pleural spaces are clear. The mediastinal contours are normal. Bony overlap in the lung apices could obscure a small pulmonary nodule. No acute cardiopulmonary disease                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                \n",
              "1890  Normal heart size and mediastinal contours. The lungs are hyperinflated but clear. No pneumothorax or pleural effusion. No acute bony abnormalities. No acute cardiopulmonary process. .                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                \n",
              "2779   Rib films. No fractures or dislocations. Chest. Heart size normal. Lungs are clear. No effusion or pneumothorax. Minimal degenerative disease thoracic spine                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
              "3824                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
              "1305  The cardiomediastinal silhouette is within normal limits for size. Pulmonary vasculature is within normal limits. No focal consolidations, effusions, or pneumothoraces. Mild degeneration of the thoracic spine without acute bony abnormality. No acute cardiopulmonary abnormality.                                                                                                                                                                                                                                                                                                                                                                                                                                                                  \n",
              "\n",
              "         label  nb  \n",
              "1140  normal    1   \n",
              "944   abnormal  0   \n",
              "2469  abnormal  0   \n",
              "2836  abnormal  0   \n",
              "2854  normal    1   \n",
              "3307  normal    1   \n",
              "78    abnormal  0   \n",
              "1594  abnormal  0   \n",
              "3125  normal    1   \n",
              "664   abnormal  0   \n",
              "3203  normal    1   \n",
              "3298  abnormal  0   \n",
              "1112  normal    1   \n",
              "1045  abnormal  0   \n",
              "2968  abnormal  0   \n",
              "1172  normal    1   \n",
              "363   abnormal  0   \n",
              "2322  abnormal  0   \n",
              "2288  abnormal  0   \n",
              "2743  normal    1   \n",
              "3089  abnormal  0   \n",
              "318   abnormal  0   \n",
              "3486  normal    1   \n",
              "242   abnormal  0   \n",
              "155   abnormal  0   \n",
              "580   abnormal  0   \n",
              "3218  abnormal  0   \n",
              "3636  normal    1   \n",
              "2321  abnormal  0   \n",
              "775   abnormal  0   \n",
              "2076  abnormal  0   \n",
              "151   abnormal  0   \n",
              "1802  abnormal  0   \n",
              "3320  abnormal  0   \n",
              "638   abnormal  0   \n",
              "3257  abnormal  0   \n",
              "122   abnormal  0   \n",
              "2329  abnormal  0   \n",
              "2426  abnormal  0   \n",
              "1320  abnormal  0   \n",
              "968   abnormal  0   \n",
              "533   abnormal  0   \n",
              "2078  abnormal  0   \n",
              "3441  abnormal  0   \n",
              "2835  abnormal  0   \n",
              "1881  abnormal  0   \n",
              "1448  abnormal  0   \n",
              "280   abnormal  0   \n",
              "1131  abnormal  0   \n",
              "2254  abnormal  0   \n",
              "1851  abnormal  0   \n",
              "1899  abnormal  0   \n",
              "3040  abnormal  0   \n",
              "616   abnormal  0   \n",
              "1890  abnormal  0   \n",
              "2779  normal    1   \n",
              "3824  normal    1   \n",
              "1305  abnormal  0   "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-6a4390d1-de51-46db-a86a-a7e647128c02\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>full-text</th>\n",
              "      <th>label</th>\n",
              "      <th>nb</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1140</th>\n",
              "      <td>The lungs are clear. No pleural effusion is seen. The heart and mediastinum are normal. Arthritic changes of the spine are present. No active disease.</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>944</th>\n",
              "      <td>The heart is normal in size. The mediastinum is unremarkable. The lungs are hypoinflated but clear. No acute disease.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2469</th>\n",
              "      <td>The heart is normal in size with normal appearance of the cardiomediastinal silhouette. The lungs are clear without focal airspace opacity, pleural effusion, or pneumothorax. The osseous structures are intact, with mild degenerative changes in the thoracic spine. No acute cardiopulmonary finding.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2836</th>\n",
              "      <td>No there is an dextroscoliosis of the thoracic spine. The cardiomediastinal silhouette and pulmonary vasculature are within normal limits. There is no pneumothorax or pleural effusion. There are no focal areas of consolidation. No acute cardiopulmonary abnormality.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2854</th>\n",
              "      <td>The heart is normal in size and contour. The lungs are clear, without evidence of infiltrate. There is no pneumothorax or effusion. Multiple punctate round XXXX XXXX over the abdomen on the lateral view. These may reside within, or outside of the patient. No acute cardiopulmonary disease.</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3307</th>\n",
              "      <td>No focal lung opacity, pleural effusion or pneumothorax. Cardiomediastinal silhouette is unremarkable. No focal lung opacity, pleural effusion of pneumothorax.</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>78</th>\n",
              "      <td>The lungs are clear. Heart size is normal. No pneumothorax. There are endplate changes in the spine. Clear lungs. No acute cardiopulmonary abnormality. .</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1594</th>\n",
              "      <td>Lungs are hyperexpanded bilaterally, with no focal consolidation, pleural effusion, or pneumothoraces. Cardiomediastinal silhouette is within normal limits. XXXX are unremarkable. No acute cardiopulmonary abnormality.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3125</th>\n",
              "      <td>PA and lateral views of the chest were obtained. The cardiomediastinal silhouette is within limits. Postoperative changes from spinal rods are demonstrated. There is elevation of the left hemidiaphragm. Multiple colonic loops are demonstrated in the left upper quadrant. The lungs are clear bilaterally. Left humeral head is positioned anterior and inferior to the glenoid, concerning for anterior shoulder subluxation. 1. No active cardiopulmonary disease. 2. Left humeral head is positioned anterior and inferior to the glenoid, concerning for anterior shoulder subluxation. This is XXXX related to the muscular dystrophy and decreased shoulder muscles support. 3. XXXX postoperative changes from the spinal XXXX placement.</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>664</th>\n",
              "      <td>The heart size is normal. The mediastinal contour is within normal limits. The lungs are free of any focal infiltrates. There are no nodules or masses. No visible pneumothorax. No visible pleural fluid. The XXXX are grossly normal. There is no visible free intraperitoneal air under the diaphragm. Surgical clips are seen the right upper quadrant. 1. No acute radiographic cardiopulmonary process.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3203</th>\n",
              "      <td>There may be a subtle airspace opacity in the right base near the midclavicular line. There is no pleural effusion or pneumothorax. The heart and mediastinum are normal. The skeletal structures are normal. Possible area of pneumonitis right lower lobe.</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3298</th>\n",
              "      <td>Frontal and lateral views of the chest show an unchanged cardiomediastinal silhouette. Normal mediastinal contour, pulmonary XXXX and vasculature, central airways and aeration of the lungs. No focal airspace consolidation or pleural effusion. There is subphrenic intraperitoneal extraluminal XXXX free XXXX. 1. No acute or active cardiac, pulmonary or pleural disease. 2. Pneumoperitoneum, attributed to the patient's recent abdominal surgery.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1112</th>\n",
              "      <td>Heart size within normal limits, stable mediastinal and hilar contours. No focal alveolar consolidation, no definite pleural effusion seen. No typical findings of pulmonary edema. XXXX convexity also present on the previous exam. No acute findings</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1045</th>\n",
              "      <td>The Heart size is normal. Cardiomediastinal silhouette is normal in contour. The lungs are clear bilaterally. Lateral views obscured by patient body habitus. There is no evidence of apical disease. XXXX are unchanged from previous exam and appear normal. Thoracic spine shows osteophyte formations at several levels. no radiographic evidence of tuberculosis or sarcoidosis.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2968</th>\n",
              "      <td>The heart, pulmonary XXXX and mediastinum are within normal limits. There is no pleural effusion or pneumothorax. There is no focal air space opacity to suggest a pneumonia. There are degenerative changes of the spine. No acute cardiopulmonary disease.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1172</th>\n",
              "      <td>Heart size upper limits normal. Sternotomy. Right-sided aortic XXXX. Lungs are clear.</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>363</th>\n",
              "      <td>The heart size and pulmonary vascularity appear within normal limits. Lungs are free of focal airspace disease. No pleural effusion or pneumothorax is seen. XXXX XXXX foreign body is noted in the soft tissues of the left chest wall. No evidence of active disease.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2322</th>\n",
              "      <td>The mediastinal silhouette is widened with overlying sternotomy XXXX. The heart size is normal. The lungs are clear without evidence of effusion, infiltrate or pneumothorax. Visualized bony structures are intact with no acute abnormalities. 1. Wide mediastinal XXXX, consistent with history of aortic dissection. 2. Otherwise normal chest x-XXXX.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2288</th>\n",
              "      <td>The lungs are clear without focal consolidation, effusion, or pneumothorax. Normal heart size. Unchanged multiple XXXX foreign bodies overlying the left clavicle and midline in the posterior soft tissues. The bony thorax is grossly intact. Negative for acute cardiopulmonary abnormality.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2743</th>\n",
              "      <td></td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3089</th>\n",
              "      <td>Heart size normal. Mediastinal silhouette and vascularity are within normal limits. Lungs are clear, hyperinflated. There is no pleural effusion or pneumothorax.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>318</th>\n",
              "      <td>The XXXX examination consists of frontal and lateral radiographs of the chest. There are diminished lung volumes. The cardiomediastinal contours are within normal limits. Pulmonary vascularity is within normal limits. No focal consolidation, pleural effusion, or pneumothorax identified. XXXX scoliosis is unchanged. Visualized upper abdomen is grossly unremarkable. No evidence of acute cardiopulmonary process or significant interval change.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3486</th>\n",
              "      <td>The XXXX and soft tissue appear normal. The cardiac silhouette and mediastinum size are normal. The aortic XXXX is on the left. The trachea is well seen and appears normal. The lungs are clear. Unremarkable chest study. .</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>242</th>\n",
              "      <td>There's been interval enlargement in the cardiac silhouette. These XXXX't contours are within normal limits. The lungs are normally inflated and clear. Osseous structures are within normal limits for patient age. 1. No acute radiographic cardiopulmonary process. 2. Interval development of cardiomegaly.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>155</th>\n",
              "      <td>The trachea is midline. The cardiomediastinal silhouette is normal and unchanged compared to prior examination. The lungs are clear, without evidence of acute infiltrate or effusion. There is no pneumothorax. The visualized bony structures reveal no acute abnormalities. There is hiatal hernia. No acute cardiopulmonary abnormalities.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>580</th>\n",
              "      <td>The heart, pulmonary XXXX and mediastinum are within normal limits. There is no pleural effusion or pneumothorax. There is no focal air space opacity to suggest a pneumonia. There are minimal degenerative changes of the spine. No acute cardiopulmonary disease.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3218</th>\n",
              "      <td>The XXXX examination consists of frontal and lateral radiographs of the chest. Atherosclerotic calcifications of the aortic XXXX are again seen. The cardiomediastinal contours are within normal limits. Pulmonary vascularity is within normal limits. No focal consolidation, pleural effusion, or pneumothorax identified. The visualized osseous structures and upper abdomen are unremarkable. No evidence of acute cardiopulmonary process.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3636</th>\n",
              "      <td></td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2321</th>\n",
              "      <td>The heart size and pulmonary vascularity appear within normal limits. Lungs are free of focal airspace disease. No pleural effusion or pneumothorax is seen. VP shunt tubing is identified. The bony structures, as visualized, appear unremarkable. No evidence of active disease.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>775</th>\n",
              "      <td>The lungs are hypoventilated. There is no focal airspace opacity. The cardiomediastinal silhouette is normal in size. There is no pneumothorax or large pleural effusion. No acute cardiopulmonary abnormality.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2076</th>\n",
              "      <td>No The cardiomediastinal silhouette and pulmonary vasculature are within normal limits. There is no pneumothorax or pleural effusion. There are no focal areas of consolidation. There are T-spine osteophytes. No acute cardiopulmonary abnormality.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>151</th>\n",
              "      <td>2 images. Vague density within the lower lung XXXX is secondary to soft tissue overlap. Heart size and pulmonary vascular engorgement appear within limits of normal. Mediastinal contour is unremarkable. No focal consolidation, pleural effusion, or pneumothorax identified. No convincing acute bony findings. No acute cardiopulmonary abnormality identified.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1802</th>\n",
              "      <td>Well-expanded and clear lungs. Apical oligemia suggestive of some emphysema. Mediastinal contour within normal limits. No acute cardiopulmonary abnormality identified.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3320</th>\n",
              "      <td>Frontal and lateral views of the chest with overlying external cardiac monitor leads show an unchanged cardiomediastinal silhouette. No XXXX focal airspace consolidation or pleural effusion. No acute or active cardiac, pulmonary or pleural disease.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>638</th>\n",
              "      <td>Trachea is midline. The cardiomediastinal silhouette is normal. The lungs are clear, without evidence of acute infiltrate or effusion. There is no pneumothorax. The visualized bony structures show no acute abnormalities. Lateral view reveals mild degenerative changes of the thoracic spine. No acute cardiopulmonary abnormalities.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3257</th>\n",
              "      <td>Sternotomy sutures and bypass grafts have been placed in the interval. Both lungs remain clear and expanded with no infiltrates. Pulmonary XXXX are normal. Post operative chest with no acute disease.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>122</th>\n",
              "      <td>Cardiac and mediastinal contours are within normal limits. Prior granulomatous disease. The lungs are otherwise clear. Bony structures are intact. No acute findings.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2329</th>\n",
              "      <td>Heart size and mediastinal contours appear normal limits. No focal pulmonary opacity, pleural effusion or pneumothorax. No acute bony abnormality. Degenerative changes of the spine. No acute cardiopulmonary abnormality.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2426</th>\n",
              "      <td>Heart size and pulmonary vascularity appear within normal limits. The lungs are free of focal airspace disease. No pleural effusion or pneumothorax is seen. Vascular calcification is noted. 1. No evidence of active disease.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1320</th>\n",
              "      <td>There are low lung volumes. The lungs are clear. No focal airspace consolidation. No pleural effusion or pneumothorax. Heart size and mediastinal contour appear within normal limits. No acute abnormality demonstrated.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>968</th>\n",
              "      <td>Spinal stimulator in XXXX. Lungs are clear without focal airspace disease. No pleural effusions or pneumothoraces. Heart and mediastinum of normal size and contour. Degenerative changes in the thoracic spine. Clear lungs.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>533</th>\n",
              "      <td>Frontal and lateral views of the chest show an unchanged cardiomediastinal silhouette. Normal mediastinal contour, pulmonary XXXX and vasculature, central airways and aeration of the lungs. No pleural effusion. There are XXXX spine marginal osteophytes. Stable chest. No acute disease process identified.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2078</th>\n",
              "      <td>Cardiac and mediastinal contours are within normal limits. The lungs are clear. Mild prominence left hilar contour. Bony structures are intact. No acute preoperative findings.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3441</th>\n",
              "      <td>Lungs are clear. Heart size is normal. No pneumothorax. There are surgical clips seen within the upper abdomen. Clear lungs. No acute cardiopulmonary abnormality. .</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2835</th>\n",
              "      <td>The lungs are clear bilaterally. Specifically, no evidence of focal consolidation, pneumothorax, or pleural effusion.. Cardio mediastinal silhouette is unremarkable. Visualized osseous structures of the thorax demonstrates stable mild multilevel degenerative disc disease of the thoracolumbar spine as well as chronic left-sided rib fractures without acute abnormality. No acute cardiopulmonary abnormality.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1881</th>\n",
              "      <td>The heart and lungs have XXXX XXXX in the interval. Both lungs are clear and expanded. Heart and mediastinum unremarkable. No change in the mediastinal surgical clips. No active disease.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1448</th>\n",
              "      <td>The heart is normal in size and contour. There is no mediastinal widening. The lungs are hyperexpanded. No focal airspace disease. No large pleural effusion or pneumothorax. Exaggerated kyphosis. No acute cardiopulmonary abnormalities.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>280</th>\n",
              "      <td>Heart size and pulmonary vascularity appear within normal limits. The lungs are free of focal airspace disease. No pleural effusion or pneumothorax is seen. Degenerative changes are present in the spine. 1. No evidence of active disease.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1131</th>\n",
              "      <td>Artifact in the region of the central upper abdomen. No focal areas of consolidation. No pleural effusions. No evidence of pneumothorax. Heart size within normal limits. Osseous structures intact. Limited exam secondary to artifact within the upper abdomen (this does not represent free intra-abdominal XXXX). Recommend repeat chest x-XXXX.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2254</th>\n",
              "      <td>The lungs are clear. The cardiomediastinal silhouette is within normal limits. There is ectasia of the thoracic aorta. No pleural effusion is identified. Normal chest film.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1851</th>\n",
              "      <td>Heart size is normal lungs are clear. No evidence of tuberculosis. Minimal scoliosis.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1899</th>\n",
              "      <td>Heart size within normal limits. Trachea is midline. The lung volumes are is somewhat low. Both lungs are otherwise clear bilaterally. No pleural effusion. No pulmonary nodules visualized. No pulmonary nodules. Negative chest.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3040</th>\n",
              "      <td>There are no focal areas of consolidation. No suspicious bony opacities. Heart size within normal limits. No pleural effusions. No evidence of pneumothorax. Mild degenerative changes of the thoracic spine. No acute cardiopulmonary abnormality.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>616</th>\n",
              "      <td>The lungs are clear. The heart and pulmonary XXXX appear normal. Pleural spaces are clear. The mediastinal contours are normal. Bony overlap in the lung apices could obscure a small pulmonary nodule. No acute cardiopulmonary disease</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1890</th>\n",
              "      <td>Normal heart size and mediastinal contours. The lungs are hyperinflated but clear. No pneumothorax or pleural effusion. No acute bony abnormalities. No acute cardiopulmonary process. .</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2779</th>\n",
              "      <td>Rib films. No fractures or dislocations. Chest. Heart size normal. Lungs are clear. No effusion or pneumothorax. Minimal degenerative disease thoracic spine</td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3824</th>\n",
              "      <td></td>\n",
              "      <td>normal</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1305</th>\n",
              "      <td>The cardiomediastinal silhouette is within normal limits for size. Pulmonary vasculature is within normal limits. No focal consolidations, effusions, or pneumothoraces. Mild degeneration of the thoracic spine without acute bony abnormality. No acute cardiopulmonary abnormality.</td>\n",
              "      <td>abnormal</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6a4390d1-de51-46db-a86a-a7e647128c02')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-6a4390d1-de51-46db-a86a-a7e647128c02 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-6a4390d1-de51-46db-a86a-a7e647128c02');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Am8WBe6_dl5q"
      },
      "source": [
        "# References\n",
        "\n",
        "[1] Demner-Fuhshman D, Kohli MD, Rosenman MB, Shooshan SE, Rodriguez L, Antani S, Thoma GR, McDonald CJ. \"Preparing a collection of radiology examinations for distribution and retrieval\". J. Am. Med. Inform. Assoc. 2016. 23(2). 304-310. https://doi.org/10.1093/jamia/ocv080\n",
        "\n",
        "[2] Wiggins WF, Kitamura F, Santos I, Prevedello L. \"Natural Language Processing of Radiology Text Reports: Interactive Text Classification\". Radiol. Artif. Intell. 2021. 3(4). https://doi.org/10.1148/ryai.2021210035\n",
        "\n",
        "#Further Reading\n",
        "Jurafsky D, Martin JH. \"Speech and Language Processing. An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition\". Draft. 3ed. September 21, 2021.\n",
        "\n",
        "Pons E, Braun LM M, Hunink MG Myriam, Kors JA. \"Natural Language Processing in Radiology: A Systematic Review\". Radiology. 2016. 279(2). 329-343.\n",
        "\n",
        "Sorin V, Barash Y, Konen E, Klang E. \"Deep Learning for Natural Language Processing in Radiology - Fundamentals and a Systematic Review\". J. Am. Coll. Radiol. 2020. 17(5). 639-648.\n",
        "\n",
        "Mozayan A, Fabbri AR, Maneevese M, Tocino I, Chheang Sophie. \"Practical Guide to Natural Language Processing for Radiology\". Radiographics. 2021. 41(5). 1446-1453."
      ]
    }
  ]
}