1_pretrained_vectors.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "1-pretrained-vectors.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dpressel/dlss-tutorial/blob/master/1_pretrained_vectors.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GkZM-nI_5Qug",
        "colab_type": "text"
      },
      "source": [
        "# Part I: Pre-trained embeddings\n",
        "\n",
        "We are going to build some PyTorch models that are commonly used for text classification.  We also need to build out some infrastructure to run these models.\n",
        "\n",
        "Once we have the models and the boilerplate stuff out of the way, we can see the impact of pre-trained embeddings for classification tasks. Pre-training methods like word2vec are context-limited language models whose goal is to predict a word given a fixed context, or a fixed context given a word. Pre-trained embeddings are particularly useful for smaller datasets.\n",
        "\n",
        "Most of this code is inspired or derived from [Baseline](https://github.com/dpressel/baseline/) (Pressel et al, 2018), an open source project for building and evaluating NLP models across a variety of NLP tasks.  For this tutorial, we will only concern ourselves with Text Classification using a few useful models.\n",
        "\n",
        "## Word Embeddings in NLP\n",
        "\n",
        "We start our models with what are called \"one-hot\" vectors.  This is notionally a sparse vector with length `|V|` where V is our vocabulary, and where only the word representated at this temporal location is a 1.  The rest are zeros.\n",
        "\n",
        "![onehot](https://www.tensorflow.org/images/feature_columns/categorical_column_with_vocabulary.jpg)\n",
        "\n",
        "These vectors are not truly represented as a vector, but as an array of indices (in PyTorch, they are `torch.LongTensor`s), one for each word's index in the vocab.  This representation is not particularly helpful in DNNs since we want continuous representations for each word.\n",
        "\n",
        "![indices](https://www.tensorflow.org/images/feature_columns/categorical_column_with_identity.jpg)\n",
        "\n",
        "The general idea of an embedding is that we want to project from a large one-hot vector to a compact, distributed representation with smaller dimensionality.  We can look at this as a matrix multiply between a one-hot vector `|V|` and a weight matrix to a lower dimension of size `|D|`.  Since only a single vector value in the one-hot vector is on at a time, this matrix multiply is simplified to an address lookup in that matrix.\n",
        "\n",
        "![LUT](https://cdn-images-1.medium.com/max/800/1*fZj1Hk1mhS5pIMv3ZrpLYw.png)\n",
        "\n",
        "\n",
        "In PyTorch, this is called an `nn.Embedding`.  In fact, in Torch7, this was called a `nn.LookupTable` which may have actually been a better name, but which seems to have fallen out of favor in DNN toolkits.  In this tutorial we are going to refer to multiple types of embeddings, and in this case, we are referring to word vectors, which are typically lookup table embeddings.\n",
        "\n",
        "Embeddings make up lowest layer of a typical DNN for text and we will feed their output to some pooling mechanism yielding a fixed length representation, followed by some number of fully connected layers.\n",
        "\n",
        "### Pre-training with Word2Vec\n",
        "\n",
        "There has been a large amount of research that has gone into building distributed representations for words through pre-training. Some widely used algorithms in NLP include Word2Vec, GloVe and fastText.  For instance, word2vec is actually 2 different algorithms with 2 different objectives.  They can be thought of a fixed context window non-causal LMs, but they are shallow models and extremely fast to train\n",
        "\n",
        "![word2vec](https://deeplearning4j.org/img/word2vec_diagrams.png)\n",
        "\n",
        "\n",
        "* **CBOW objective** given all words in a fixed context window except the middle word, predict the middle word\n",
        "* **SkipGram objective**: given a word in a fixed context window, predict all other words in that window\n",
        "\n",
        "Once we have trained these models, the learned distributed representation matrix can be plugged right in as our embedding weights and this often improves the model significantly.\n",
        "\n",
        "Before we begin, we will download some data that can be used for our experiments"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rIAhO5RRf4jL",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "64239e21-a3cd-45dc-998a-bb80b2bdb0ae"
      },
      "source": [
        "!wget https://www.dropbox.com/s/7jyi4pi894bh2qh/sst2.tar.gz?dl=1\n",
        "!tar -xzf 'sst2.tar.gz?dl=1'\n",
        "\n",
        "!wget https://www.dropbox.com/s/08km2ean8bkt7p3/trec.tar.gz?dl=1\n",
        "!tar -xzf 'trec.tar.gz?dl=1'\n",
        "\n",
        "!wget https://www.dropbox.com/s/699kgut7hdb5tg9/GoogleNews-vectors-negative300.bin.gz?dl=1\n",
        "!mv 'GoogleNews-vectors-negative300.bin.gz?dl=1' GoogleNews-vectors-negative300.bin.gz\n",
        "!gunzip GoogleNews-vectors-negative300.bin.gz"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2019-06-30 01:12:58--  https://www.dropbox.com/s/7jyi4pi894bh2qh/sst2.tar.gz?dl=1\n",
            "Resolving www.dropbox.com (www.dropbox.com)... 162.125.81.1, 2620:100:6031:1::a27d:5101\n",
            "Connecting to www.dropbox.com (www.dropbox.com)|162.125.81.1|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: /s/dl/7jyi4pi894bh2qh/sst2.tar.gz [following]\n",
            "--2019-06-30 01:12:58--  https://www.dropbox.com/s/dl/7jyi4pi894bh2qh/sst2.tar.gz\n",
            "Reusing existing connection to www.dropbox.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://uc04e0a125eff9ec13572a9a6a9f.dl.dropboxusercontent.com/cd/0/get/AjzPSePvGb0BHoTcpOMyaVFbMZnr4Sv3S4e7fGsv-it1K8QAMhv9okwYdZyizFJPI1YLLmzxNzCo6_aqlZZdzy91IzKq8CoytjxPrD1RMFg2vw/file?dl=1# [following]\n",
            "--2019-06-30 01:12:59--  https://uc04e0a125eff9ec13572a9a6a9f.dl.dropboxusercontent.com/cd/0/get/AjzPSePvGb0BHoTcpOMyaVFbMZnr4Sv3S4e7fGsv-it1K8QAMhv9okwYdZyizFJPI1YLLmzxNzCo6_aqlZZdzy91IzKq8CoytjxPrD1RMFg2vw/file?dl=1\n",
            "Resolving uc04e0a125eff9ec13572a9a6a9f.dl.dropboxusercontent.com (uc04e0a125eff9ec13572a9a6a9f.dl.dropboxusercontent.com)... 162.125.81.6, 2620:100:6031:6::a27d:5106\n",
            "Connecting to uc04e0a125eff9ec13572a9a6a9f.dl.dropboxusercontent.com (uc04e0a125eff9ec13572a9a6a9f.dl.dropboxusercontent.com)|162.125.81.6|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 1759259 (1.7M) [application/binary]\n",
            "Saving to: ‘sst2.tar.gz?dl=1’\n",
            "\n",
            "sst2.tar.gz?dl=1    100%[===================>]   1.68M  3.94MB/s    in 0.4s    \n",
            "\n",
            "2019-06-30 01:13:00 (3.94 MB/s) - ‘sst2.tar.gz?dl=1’ saved [1759259/1759259]\n",
            "\n",
            "--2019-06-30 01:13:04--  https://www.dropbox.com/s/08km2ean8bkt7p3/trec.tar.gz?dl=1\n",
            "Resolving www.dropbox.com (www.dropbox.com)... 162.125.81.1, 2620:100:6031:1::a27d:5101\n",
            "Connecting to www.dropbox.com (www.dropbox.com)|162.125.81.1|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: /s/dl/08km2ean8bkt7p3/trec.tar.gz [following]\n",
            "--2019-06-30 01:13:04--  https://www.dropbox.com/s/dl/08km2ean8bkt7p3/trec.tar.gz\n",
            "Reusing existing connection to www.dropbox.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://uc8eb0b5de3f5bf250a5403c857d.dl.dropboxusercontent.com/cd/0/get/AjxktPxIg9dMjFh6y7djMj1wZ2zZfxHGq7BdLeSom_HCGa5lE95AMt4DOMcQTJtqIf72KG3bWidAK0oxCdRDXGtZ4PnqYLPRIhln_ne9KgJ50g/file?dl=1# [following]\n",
            "--2019-06-30 01:13:04--  https://uc8eb0b5de3f5bf250a5403c857d.dl.dropboxusercontent.com/cd/0/get/AjxktPxIg9dMjFh6y7djMj1wZ2zZfxHGq7BdLeSom_HCGa5lE95AMt4DOMcQTJtqIf72KG3bWidAK0oxCdRDXGtZ4PnqYLPRIhln_ne9KgJ50g/file?dl=1\n",
            "Resolving uc8eb0b5de3f5bf250a5403c857d.dl.dropboxusercontent.com (uc8eb0b5de3f5bf250a5403c857d.dl.dropboxusercontent.com)... 162.125.81.6, 2620:100:6031:6::a27d:5106\n",
            "Connecting to uc8eb0b5de3f5bf250a5403c857d.dl.dropboxusercontent.com (uc8eb0b5de3f5bf250a5403c857d.dl.dropboxusercontent.com)|162.125.81.6|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 117253 (115K) [application/binary]\n",
            "Saving to: ‘trec.tar.gz?dl=1’\n",
            "\n",
            "trec.tar.gz?dl=1    100%[===================>] 114.50K  --.-KB/s    in 0.01s   \n",
            "\n",
            "2019-06-30 01:13:05 (9.31 MB/s) - ‘trec.tar.gz?dl=1’ saved [117253/117253]\n",
            "\n",
            "--2019-06-30 01:13:07--  https://www.dropbox.com/s/699kgut7hdb5tg9/GoogleNews-vectors-negative300.bin.gz?dl=1\n",
            "Resolving www.dropbox.com (www.dropbox.com)... 162.125.81.1, 2620:100:6031:1::a27d:5101\n",
            "Connecting to www.dropbox.com (www.dropbox.com)|162.125.81.1|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: /s/dl/699kgut7hdb5tg9/GoogleNews-vectors-negative300.bin.gz [following]\n",
            "--2019-06-30 01:13:08--  https://www.dropbox.com/s/dl/699kgut7hdb5tg9/GoogleNews-vectors-negative300.bin.gz\n",
            "Reusing existing connection to www.dropbox.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://ucc94c6cb27db59d15946e726c0e.dl.dropboxusercontent.com/cd/0/get/AjxtsvhYKA888x7WZQOTs9rdfEmxm3sk44V8o6XNM13xter70pPBaST3EOBeTVxBoSq76DyBy_ZdiawefmaObDzO0NuQa6g3qWhzEuQ-6iHVEA/file?dl=1# [following]\n",
            "--2019-06-30 01:13:08--  https://ucc94c6cb27db59d15946e726c0e.dl.dropboxusercontent.com/cd/0/get/AjxtsvhYKA888x7WZQOTs9rdfEmxm3sk44V8o6XNM13xter70pPBaST3EOBeTVxBoSq76DyBy_ZdiawefmaObDzO0NuQa6g3qWhzEuQ-6iHVEA/file?dl=1\n",
            "Resolving ucc94c6cb27db59d15946e726c0e.dl.dropboxusercontent.com (ucc94c6cb27db59d15946e726c0e.dl.dropboxusercontent.com)... 162.125.81.6, 2620:100:6031:6::a27d:5106\n",
            "Connecting to ucc94c6cb27db59d15946e726c0e.dl.dropboxusercontent.com (ucc94c6cb27db59d15946e726c0e.dl.dropboxusercontent.com)|162.125.81.6|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 1743563840 (1.6G) [application/binary]\n",
            "Saving to: ‘GoogleNews-vectors-negative300.bin.gz?dl=1’\n",
            "\n",
            "GoogleNews-vectors- 100%[===================>]   1.62G  44.3MB/s    in 43s     \n",
            "\n",
            "2019-06-30 01:13:52 (38.5 MB/s) - ‘GoogleNews-vectors-negative300.bin.gz?dl=1’ saved [1743563840/1743563840]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OiXst1zUqOa4",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u_ZR8A0x6XQh",
        "colab_type": "text"
      },
      "source": [
        "## First, lets do some fun stuff\n",
        "\n",
        "We will start by building out some models that we will reuse later in the tutorial.  First, we will build a convolutional neural network (CNN) that can classify text. Basically CNNs learn a kernel that can be used to filter images or text.  An example of 2D filtering*:\n",
        "\n",
        "![2D filtering](https://cdn-images-1.medium.com/max/1600/0*9J3MK1gd2zrFDzDN.gif)\n",
        "\n",
        "In the case of text filtering, we have a one-dimensional filter operation like this*:\n",
        "\n",
        "![1D filtering](http://cs231n.github.io/assets/cnn/stride.jpeg)\n",
        "\n",
        "This type of model has been used often in text, including by [Collobert et al 2011](https://ronan.collobert.com/pub/matos/2011_nlp_jmlr.pdf), but we will implement a multiple parallel filter variation of this introduced by [Kim 2014](https://www.aclweb.org/anthology/D14-1181).\n",
        "\n",
        "### Convolutional Neural Network for Text Classification\n",
        "\n",
        "We are using PyTorch, so every layer we have is going to inherit `nn.Module`.\n",
        "\n",
        "#### Convolutions (actually cross correlations)\n",
        "\n",
        "The first characteristic of this model is that we will have multiple convolutional filter lengths, and some number of filters associated with each length.  For each filter of length `K` convolved with a signal of length `T`, the output signal will be `T - K + 1`.  To handle the ends of the signal where the filter is hanging off (e.g. centered at 0), we will add some zero-padding.  So if we have a filter of length `K=3`, we want to zero-pad the temporal signal by a single pad value on both ends of the signal.\n",
        "\n",
        "We are going to support multiple parallel filters, so we can add a `torch.nn.Conv1d` for each filter length, followed by a `torch.nn.ReLU` activation layer.  Since we have more than one of these, we will create a `nn.ModuleList` to track them.  When we call `forward()`, the data will be oriented as $$B \\times C \\times T$$ where `B` is the batch size,  `C` is the number of hidden units and `T` is the temporal length of the vector.\n",
        "\n",
        "#### Pooling\n",
        "\n",
        "Both of the papers mentioned above do max-over-time pooling over the features.  For each feature map in the vector, we simply select the maximum value for that feature map and concatenate all of these together.  Our $$B \\times C \\times T$$ vector is then going to be reduced along the time dimension to $$B \\times C$$\n",
        "\n",
        "*Images courtesy of http://cs231n.github.io/convolutional-networks/"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PTT1BSfB9kKO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "from typing import List, Tuple\n",
        "import os\n",
        "import io\n",
        "import re\n",
        "import codecs\n",
        "from collections import Counter\n",
        "from torch.utils.data import DataLoader, TensorDataset"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rD0S18xj9qi9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class ParallelConv(nn.Module):\n",
        "\n",
        "    def __init__(self, input_dims, filters, dropout=0.5):\n",
        "        super().__init__()\n",
        "        convs = []        \n",
        "        self.output_dims = sum([t[1] for t in filters])\n",
        "        for (filter_length, output_dims) in filters:\n",
        "            pad = filter_length//2\n",
        "            conv = nn.Sequential(\n",
        "                nn.Conv1d(input_dims, output_dims, filter_length, padding=pad),\n",
        "                nn.ReLU()\n",
        "            )\n",
        "            convs.append(conv)\n",
        "        # Add the module so its managed correctly\n",
        "        self.convs = nn.ModuleList(convs)\n",
        "        self.conv_drop = nn.Dropout(dropout)\n",
        "\n",
        "\n",
        "    def forward(self, input_bct):\n",
        "        mots = []\n",
        "        for conv in self.convs:\n",
        "            # In Conv1d, data BxCxT, max over time\n",
        "            conv_out = conv(input_bct)\n",
        "            mot, _ = conv_out.max(2)\n",
        "            mots.append(mot)\n",
        "        mots = torch.cat(mots, 1)\n",
        "        return self.conv_drop(mots)\n",
        "\n",
        "class ConvClassifier(nn.Module):\n",
        "\n",
        "    def __init__(self, embeddings, num_classes, embed_dims,\n",
        "                 filters=[(2, 100), (3, 100), (4, 100)],\n",
        "                 dropout=0.5, hidden_units=[]):\n",
        "        super().__init__()\n",
        "        self.embeddings = embeddings\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        self.convs = ParallelConv(embed_dims, filters, dropout)\n",
        "        \n",
        "        input_units = self.convs.output_dims\n",
        "        output_units = self.convs.output_dims\n",
        "        sequence = []\n",
        "        for h in hidden_units:\n",
        "            sequence.append(self.dropout(nn.Linear(input_units, h)))\n",
        "            input_units = h\n",
        "            output_units = h\n",
        "            \n",
        "        sequence.append(nn.Linear(output_units, num_classes))\n",
        "        self.outputs = nn.Sequential(*sequence)\n",
        "\n",
        "    def forward(self, inputs):\n",
        "        one_hots, lengths = inputs\n",
        "        embed = self.dropout(self.embeddings(one_hots))\n",
        "        embed = embed.transpose(1, 2).contiguous()\n",
        "        hidden = self.convs(embed)\n",
        "        linear = self.outputs(hidden)\n",
        "        return F.log_softmax(linear, dim=-1)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lBN5z1huBPqr",
        "colab_type": "text"
      },
      "source": [
        "### LSTM Model\n",
        "\n",
        "Our second model that we will explore uses Long Short-Term Memory (LSTM) units, which are a form of Recurrent Neural Networks.  These models tend to perform extremely well on NLP tasks.  Text classification is a simple case, where we give our inputs and take the final LSTM output as a form of pooling.  That looks like the **Many-to-One** image in this taxonomy from [Andrej Karpathy's 2015 blog post on using RNNs for character-level language modeling](https://karpathy.github.io/2015/05/21/rnn-effectiveness/)\n",
        "\n",
        "![Many-to-one LSTM](https://karpathy.github.io/assets/rnn/diags.jpeg)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aeQUDe_BCOYV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class LSTMClassifier(nn.Module):\n",
        "\n",
        "    def __init__(self, embeddings, num_classes, embed_dims, rnn_units, rnn_layers=1, dropout=0.5, hidden_units=[]):\n",
        "        super().__init__()\n",
        "        self.embeddings = embeddings\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        self.rnn = torch.nn.LSTM(embed_dims,\n",
        "                                 rnn_units,\n",
        "                                 rnn_layers,\n",
        "                                 dropout=dropout,\n",
        "                                 bidirectional=False,\n",
        "                                 batch_first=False)\n",
        "        nn.init.orthogonal_(self.rnn.weight_hh_l0)\n",
        "        nn.init.orthogonal_(self.rnn.weight_ih_l0)\n",
        "        sequence = []\n",
        "        input_units = rnn_units\n",
        "        output_units = rnn_units\n",
        "        for h in hidden_units:\n",
        "            sequence.append(nn.Linear(input_units, h))\n",
        "            input_units = h\n",
        "            output_units = h\n",
        "            \n",
        "        sequence.append(nn.Linear(output_units, num_classes))\n",
        "        self.outputs = nn.Sequential(*sequence)\n",
        "        \n",
        "        \n",
        "    def forward(self, inputs):\n",
        "        one_hots, lengths = inputs\n",
        "        embed = self.dropout(self.embeddings(one_hots))\n",
        "        embed = embed.transpose(0, 1)\n",
        "        packed = torch.nn.utils.rnn.pack_padded_sequence(embed, lengths.tolist())\n",
        "        _, hidden = self.rnn(packed)\n",
        "        hidden = hidden[0].view(hidden[0].shape[1:])\n",
        "        linear = self.outputs(hidden)\n",
        "        return F.log_softmax(linear, dim=-1)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JK47ZmfPCgyz",
        "colab_type": "text"
      },
      "source": [
        "## Training our model\n",
        "\n",
        "To set our model up for training (and evaluation), we need a loss function, some metrics, and an optimizer, along with some training data.\n",
        "\n",
        "### Defining Metrics\n",
        "\n",
        "For classification problems, most things we would like to know can be defined in terms of a confusion matrix.\n",
        "\n",
        "![Confusion Matrix](https://scikit-learn.org/stable/_images/sphx_glr_plot_confusion_matrix_001.png)\n",
        "\n",
        "The class below implements a confusion matrix and provides metrics associated using it.  This implementation is taken from verbatim from Baseline (https://github.com/dpressel/baseline/blob/master/python/baseline/confusion.py)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kDOWjUTGQaAH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "class ConfusionMatrix:\n",
        "    \"\"\"Confusion matrix with metrics\n",
        "\n",
        "    This class accumulates classification output, and tracks it in a confusion matrix.\n",
        "    Metrics are available that use the confusion matrix\n",
        "    \"\"\"\n",
        "    def __init__(self, labels):\n",
        "        \"\"\"Constructor with input labels\n",
        "\n",
        "        :param labels: Either a dictionary (`k=int,v=str`) or an array of labels\n",
        "        \"\"\"\n",
        "        if type(labels) is dict:\n",
        "            self.labels = []\n",
        "            for i in range(len(labels)):\n",
        "                self.labels.append(labels[i])\n",
        "        else:\n",
        "            self.labels = labels\n",
        "        nc = len(self.labels)\n",
        "        self._cm = np.zeros((nc, nc), dtype=np.int)\n",
        "\n",
        "    def add(self, truth, guess):\n",
        "        \"\"\"Add a single value to the confusion matrix based off `truth` and `guess`\n",
        "\n",
        "        :param truth: The real `y` value (or ground truth label)\n",
        "        :param guess: The guess for `y` value (or assertion)\n",
        "        \"\"\"\n",
        "\n",
        "        self._cm[truth, guess] += 1\n",
        "\n",
        "    def __str__(self):\n",
        "        values = []\n",
        "        width = max(8, max(len(x) for x in self.labels) + 1)\n",
        "        for i, label in enumerate([''] + self.labels):\n",
        "            values += [\"{:>{width}}\".format(label, width=width+1)]\n",
        "        values += ['\\n']\n",
        "        for i, label in enumerate(self.labels):\n",
        "            values += [\"{:>{width}}\".format(label, width=width+1)]\n",
        "            for j in range(len(self.labels)):\n",
        "                values += [\"{:{width}d}\".format(self._cm[i, j], width=width + 1)]\n",
        "            values += ['\\n']\n",
        "        values += ['\\n']\n",
        "        return ''.join(values)\n",
        "\n",
        "    def save(self, outfile):\n",
        "        ordered_fieldnames = OrderedDict([(\"labels\", None)] + [(l, None) for l in self.labels])\n",
        "        with open(outfile, 'w') as f:\n",
        "            dw = csv.DictWriter(f, delimiter=',', fieldnames=ordered_fieldnames)\n",
        "            dw.writeheader()\n",
        "            for index, row in enumerate(self._cm):\n",
        "                row_dict = {l: row[i] for i, l in enumerate(self.labels)}\n",
        "                row_dict.update({\"labels\": self.labels[index]})\n",
        "                dw.writerow(row_dict)\n",
        "\n",
        "    def reset(self):\n",
        "        \"\"\"Reset the matrix\n",
        "        \"\"\"\n",
        "        self._cm *= 0\n",
        "\n",
        "    def get_correct(self):\n",
        "        \"\"\"Get the diagonals of the confusion matrix\n",
        "\n",
        "        :return: (``int``) Number of correct classifications\n",
        "        \"\"\"\n",
        "        return self._cm.diagonal().sum()\n",
        "\n",
        "    def get_total(self):\n",
        "        \"\"\"Get total classifications\n",
        "\n",
        "        :return: (``int``) total classifications\n",
        "        \"\"\"\n",
        "        return self._cm.sum()\n",
        "\n",
        "    def get_acc(self):\n",
        "        \"\"\"Get the accuracy\n",
        "\n",
        "        :return: (``float``) accuracy\n",
        "        \"\"\"\n",
        "        return float(self.get_correct())/self.get_total()\n",
        "\n",
        "    def get_recall(self):\n",
        "        \"\"\"Get the recall\n",
        "\n",
        "        :return: (``float``) recall\n",
        "        \"\"\"\n",
        "        total = np.sum(self._cm, axis=1)\n",
        "        total = (total == 0) + total\n",
        "        return np.diag(self._cm) / total.astype(float)\n",
        "\n",
        "    def get_support(self):\n",
        "        return np.sum(self._cm, axis=1)\n",
        "\n",
        "    def get_precision(self):\n",
        "        \"\"\"Get the precision\n",
        "        :return: (``float``) precision\n",
        "        \"\"\"\n",
        "\n",
        "        total = np.sum(self._cm, axis=0)\n",
        "        total = (total == 0) + total\n",
        "        return np.diag(self._cm) / total.astype(float)\n",
        "\n",
        "    def get_mean_precision(self):\n",
        "        \"\"\"Get the mean precision across labels\n",
        "\n",
        "        :return: (``float``) mean precision\n",
        "        \"\"\"\n",
        "        return np.mean(self.get_precision())\n",
        "\n",
        "    def get_weighted_precision(self):\n",
        "        return np.sum(self.get_precision() * self.get_support())/float(self.get_total())\n",
        "\n",
        "    def get_mean_recall(self):\n",
        "        \"\"\"Get the mean recall across labels\n",
        "\n",
        "        :return: (``float``) mean recall\n",
        "        \"\"\"\n",
        "        return np.mean(self.get_recall())\n",
        "\n",
        "    def get_weighted_recall(self):\n",
        "        return np.sum(self.get_recall() * self.get_support())/float(self.get_total())\n",
        "\n",
        "    def get_weighted_f(self, beta=1):\n",
        "        return np.sum(self.get_class_f(beta) * self.get_support())/float(self.get_total())\n",
        "\n",
        "    def get_macro_f(self, beta=1):\n",
        "        \"\"\"Get the macro F_b, with adjustable beta (defaulting to F1)\n",
        "\n",
        "        :param beta: (``float``) defaults to 1 (F1)\n",
        "        :return: (``float``) macro F_b\n",
        "        \"\"\"\n",
        "        if beta < 0:\n",
        "            raise Exception('Beta must be greater than 0')\n",
        "        return np.mean(self.get_class_f(beta))\n",
        "\n",
        "    def get_class_f(self, beta=1):\n",
        "        p = self.get_precision()\n",
        "        r = self.get_recall()\n",
        "\n",
        "        b = beta*beta\n",
        "        d = (b * p + r)\n",
        "        d = (d == 0) + d\n",
        "\n",
        "        return (b + 1) * p * r / d\n",
        "\n",
        "    def get_f(self, beta=1):\n",
        "        \"\"\"Get 2 class F_b, with adjustable beta (defaulting to F1)\n",
        "\n",
        "        :param beta: (``float``) defaults to 1 (F1)\n",
        "        :return: (``float``) 2-class F_b\n",
        "        \"\"\"\n",
        "        p = self.get_precision()[1]\n",
        "        r = self.get_recall()[1]\n",
        "        if beta < 0:\n",
        "            raise Exception('Beta must be greater than 0')\n",
        "        d = (beta*beta * p + r)\n",
        "        if d == 0:\n",
        "            return 0\n",
        "        return (beta*beta + 1) * p * r / d\n",
        "\n",
        "    def get_all_metrics(self):\n",
        "        \"\"\"Make a map of metrics suitable for reporting, keyed by metric name\n",
        "\n",
        "        :return: (``dict``) Map of metrics keyed by metric names\n",
        "        \"\"\"\n",
        "        metrics = {'acc': self.get_acc()}\n",
        "        # If 2 class, assume second class is positive AKA 1\n",
        "        if len(self.labels) == 2:\n",
        "            metrics['precision'] = self.get_precision()[1]\n",
        "            metrics['recall'] = self.get_recall()[1]\n",
        "            metrics['f1'] = self.get_f(1)\n",
        "        else:\n",
        "            metrics['mean_precision'] = self.get_mean_precision()\n",
        "            metrics['mean_recall'] = self.get_mean_recall()\n",
        "            metrics['macro_f1'] = self.get_macro_f(1)\n",
        "            metrics['weighted_precision'] = self.get_weighted_precision()\n",
        "            metrics['weighted_recall'] = self.get_weighted_recall()\n",
        "            metrics['weighted_f1'] = self.get_weighted_f(1)\n",
        "        return metrics\n",
        "\n",
        "    def add_batch(self, truth, guess):\n",
        "        \"\"\"Add a batch of data to the confusion matrix\n",
        "\n",
        "        :param truth: The truth tensor\n",
        "        :param guess: The guess tensor\n",
        "        :return:\n",
        "        \"\"\"\n",
        "        for truth_i, guess_i in zip(truth, guess):\n",
        "            self.add(truth_i, guess_i)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IOfC-eQR92j9",
        "colab_type": "text"
      },
      "source": [
        "Our `Trainer` is simple, but it gets the job done.  We will use PyTorch's `DataLoader` to feed our batches to the trainer.  The `run()` method cycles a single epoch. \n",
        "For every batch, we will do a stochastic gradient minibatch update, and we return the loss and the predictions and ground truth back to the `run()` method for tabulation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FaQ2BraAR-qI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "class Trainer:\n",
        "    def __init__(self, optimizer: torch.optim.Optimizer):\n",
        "        self.optimizer = optimizer\n",
        "\n",
        "    def run(self, model, labels, train, loss, batch_size): \n",
        "        model.train()       \n",
        "        train_loader = DataLoader(train, batch_size=batch_size, shuffle=True)\n",
        "\n",
        "        cm = ConfusionMatrix(labels)\n",
        "\n",
        "        for batch in train_loader:\n",
        "            loss_value, y_pred, y_actual = self.update(model, loss, batch)\n",
        "            _, best = y_pred.max(1)\n",
        "            yt = y_actual.cpu().int().numpy()\n",
        "            yp = best.cpu().int().numpy()\n",
        "            cm.add_batch(yt, yp)\n",
        "\n",
        "        print(cm.get_all_metrics())\n",
        "        return cm\n",
        "    \n",
        "    def update(self, model, loss, batch):\n",
        "        self.optimizer.zero_grad()\n",
        "        x, lengths, y = batch\n",
        "        lengths, perm_idx = lengths.sort(0, descending=True)\n",
        "        x_sorted = x[perm_idx]\n",
        "        y_sorted = y[perm_idx]\n",
        "        y_sorted = y_sorted.to('cuda:0')\n",
        "        inputs = (x_sorted.to('cuda:0'), lengths)\n",
        "        y_pred = model(inputs)\n",
        "        loss_value = loss(y_pred, y_sorted)\n",
        "        loss_value.backward()\n",
        "        self.optimizer.step()\n",
        "        return loss_value.item(), y_pred, y_sorted"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kTV2epU5SZlg",
        "colab_type": "text"
      },
      "source": [
        "After training a epoch, we would like to test the validation performance.  Our evaluator class is similar to our `Trainer`, but it doesnt update our model -- it just gives us a way to evaluate the model on data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "baYbdBXqSoa6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "class Evaluator:\n",
        "    def __init__(self):\n",
        "        pass\n",
        "\n",
        "    def run(self, model, labels, dataset, batch_size=1):\n",
        "        model.eval()\n",
        "        valid_loader = DataLoader(dataset, batch_size=batch_size)\n",
        "        cm = ConfusionMatrix(labels)\n",
        "        for batch in valid_loader:\n",
        "            y_pred, y_actual = self.inference(model, batch)\n",
        "            _, best = y_pred.max(1)\n",
        "            yt = y_actual.cpu().int().numpy()\n",
        "            yp = best.cpu().int().numpy()\n",
        "            cm.add_batch(yt, yp)\n",
        "        return cm\n",
        "\n",
        "    def inference(self, model, batch):\n",
        "        with torch.no_grad():\n",
        "            x, lengths, y = batch\n",
        "            lengths, perm_idx = lengths.sort(0, descending=True)\n",
        "            x_sorted = x[perm_idx]\n",
        "            y_sorted = y[perm_idx]\n",
        "            y_sorted = y_sorted.to('cuda:0')\n",
        "            inputs = (x_sorted.to('cuda:0'), lengths)\n",
        "            y_pred = model(inputs)\n",
        "            return y_pred, y_sorted\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ySvFi6OVRnf5",
        "colab_type": "text"
      },
      "source": [
        "We can encapsulate training multiple epochs and testing in a single function.   The best model is defined in terms of some metric -- here accuracy, and we only save the checkpoints when we improve on the model.  This is called early stopping, and is particularly helpful on smaller datasets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mOhAZmODS3N8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def fit(model, labels, optimizer, loss, epochs, batch_size, train, valid, test):\n",
        "\n",
        "    trainer = Trainer(optimizer)\n",
        "    evaluator = Evaluator()\n",
        "    best_acc = 0.0\n",
        "    \n",
        "    for epoch in range(epochs):\n",
        "        print('EPOCH {}'.format(epoch + 1))\n",
        "        print('=================================')\n",
        "        print('Training Results')\n",
        "        cm = trainer.run(model, labels, train, loss, batch_size)\n",
        "        print('Validation Results')\n",
        "        cm = evaluator.run(model, labels, valid)\n",
        "        print(cm.get_all_metrics())\n",
        "        if cm.get_acc() > best_acc:\n",
        "            print('New best model {:.2f}'.format(cm.get_acc()))\n",
        "            best_acc = cm.get_acc()\n",
        "            torch.save(model.state_dict(), './checkpoint.pth')\n",
        "    if test:\n",
        "        model.load_state_dict(torch.load('./checkpoint.pth'))\n",
        "        cm = evaluator.run(model, labels, test)\n",
        "        print('Final result')\n",
        "        print(cm.get_all_metrics())\n",
        "    return cm.get_acc()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mp1tjK47TCe_",
        "colab_type": "text"
      },
      "source": [
        "### A Reader for our Data\n",
        "\n",
        "We need a reader to load our data from files and put it into a `Dataset`.\n",
        "\n",
        "The reader needs to perform a few steps\n",
        "\n",
        "* **read in sentences and labels**: it should convert the sentences into tokens and record a vocabulary of the labels\n",
        "* **vectorize tokens**: it should convert tokens into tensors that comprise rows in our `TensorDataset`\n",
        "* **tabulate the vocabulary**: if no vectorizer is provided, we need to build a vocab of attested words.  If a vectorizer is provided upfront, we dont need this step"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Uzf5KiQ1UFRU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "def whitespace_tokenizer(words: str) -> List[str]:\n",
        "    return words.split() \n",
        "\n",
        "def sst2_tokenizer(words: str) -> List[str]:\n",
        "    REPLACE = { \"'s\": \" 's \",\n",
        "                \"'ve\": \" 've \",\n",
        "                \"n't\": \" n't \",\n",
        "                \"'re\": \" 're \",\n",
        "                \"'d\": \" 'd \",\n",
        "                \"'ll\": \" 'll \",\n",
        "                \",\": \" , \",\n",
        "                \"!\": \" ! \",\n",
        "                }\n",
        "    words = words.lower()\n",
        "    words = re.sub(r\"[^A-Za-z0-9(),!?\\'\\`]\", \" \", words)\n",
        "    for k, v in REPLACE.items():\n",
        "            words = words.replace(k, v)\n",
        "    return [w.strip() for w in words.split()]\n",
        "\n",
        "\n",
        "class Reader:\n",
        "\n",
        "    def __init__(self, files, lowercase=True, min_freq=0,\n",
        "                 tokenizer=sst2_tokenizer, vectorizer=None):\n",
        "        self.lowercase = lowercase\n",
        "        self.tokenizer = tokenizer\n",
        "        build_vocab = vectorizer is None\n",
        "        self.vectorizer = vectorizer if vectorizer else self._vectorizer\n",
        "        x = Counter()\n",
        "        y = Counter()\n",
        "        for file_name in files:\n",
        "            if file_name is None:\n",
        "                continue\n",
        "            with codecs.open(file_name, encoding='utf-8', mode='r') as f:\n",
        "                for line in f:\n",
        "                    words = line.split()\n",
        "                    y.update(words[0])\n",
        "\n",
        "                    if build_vocab:\n",
        "                        words = self.tokenizer(' '.join(words[1:]))\n",
        "                        words = words if not self.lowercase else [w.lower() for w in words]\n",
        "                        x.update(words)\n",
        "        self.labels = list(y.keys())\n",
        "\n",
        "        if build_vocab:\n",
        "            x = dict(filter(lambda cnt: cnt[1] >= min_freq, x.items()))\n",
        "            alpha = list(x.keys())\n",
        "            alpha.sort()\n",
        "            self.vocab = {w: i+1 for i, w in enumerate(alpha)}\n",
        "            self.vocab['[PAD]'] = 0\n",
        "\n",
        "        self.labels.sort()\n",
        "\n",
        "    def _vectorizer(self, words: List[str]) -> List[int]:\n",
        "        return [self.vocab.get(w, 0) for w in words]\n",
        "\n",
        "    def load(self, filename: str) -> TensorDataset:\n",
        "        label2index = {l: i for i, l in enumerate(self.labels)}\n",
        "        xs = []\n",
        "        lengths = []\n",
        "        ys = []\n",
        "        with codecs.open(filename, encoding='utf-8', mode='r') as f:\n",
        "            for line in f:\n",
        "                words = line.split()\n",
        "                ys.append(label2index[words[0]])\n",
        "                words = self.tokenizer(' '.join(words[1:]))\n",
        "                words = words if not self.lowercase else [w.lower() for w in words]\n",
        "                vec = self.vectorizer(words)\n",
        "                lengths.append(len(vec))\n",
        "                xs.append(torch.tensor(vec, dtype=torch.long))\n",
        "        x_tensor = torch.nn.utils.rnn.pad_sequence(xs, batch_first=True)\n",
        "        lengths_tensor = torch.tensor(lengths, dtype=torch.long)\n",
        "        y_tensor = torch.tensor(ys, dtype=torch.long)\n",
        "        return TensorDataset(x_tensor, lengths_tensor, y_tensor)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3fMHlMQCUslk",
        "colab_type": "text"
      },
      "source": [
        "### Pre-trained Embeddings\n",
        "\n",
        "We would like to investigate how pre-training embeddings helps our models improve.  To do this, we need a mechanism to load in pre-trained embeddings and convert them into PyTorch's `nn.Embedding` object.  Specifically, we wish to support `word2vec`, `GloVe` and `fastText` embeddings. Rest-assured, these are simple file formats, and you do not need any 3rd party dependencies to read them in!  We will do it by hand.\n",
        "\n",
        "For binary files, the first line contains 2 numbers delimited by a space.  The first number is the vocab size and the second is the embedding dimension.  We then read each line, splitting it by a space and reading the first portion as the vocabulary (token) and the second portion as a binary vector.\n",
        "\n",
        "For text files, the first line may contain 2 numbers as in the binary file, but for `GloVe` files, this is omitted.  We can check if the first line contains the dimensions, and if it doesnt, we can just read in the first vector to figure out its dimension (again its space delimited, but the vector is also space delimited, so we split along the first space to find the token).\n",
        "\n",
        "Notice that in this code, we have already created an alphabet that we will pass in for each key, so if that word is present in the embedding file, we will use its value, otherwise, we will initialize the vector randomly."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w2nKdPwIVbFn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def init_embeddings(vocab_size, embed_dim, unif):\n",
        "    return np.random.uniform(-unif, unif, (vocab_size, embed_dim))\n",
        "    \n",
        "\n",
        "class EmbeddingsReader:\n",
        "\n",
        "    @staticmethod\n",
        "    def from_text(filename, vocab, unif=0.25):\n",
        "        \n",
        "        with io.open(filename, \"r\", encoding=\"utf-8\") as f:\n",
        "            for i, line in enumerate(f):\n",
        "                line = line.rstrip(\"\\n \")\n",
        "                values = line.split(\" \")\n",
        "\n",
        "                if i == 0:\n",
        "                    # fastText style\n",
        "                    if len(values) == 2:\n",
        "                        weight = init_embeddings(len(vocab), values[1], unif)\n",
        "                        continue\n",
        "                    # glove style\n",
        "                    else:\n",
        "                        weight = init_embeddings(len(vocab), len(values[1:]), unif)\n",
        "                word = values[0]\n",
        "                if word in vocab:\n",
        "                    vec = np.asarray(values[1:], dtype=np.float32)\n",
        "                    weight[vocab[word]] = vec\n",
        "        if '[PAD]' in vocab:\n",
        "            weight[vocab['[PAD]']] = 0.0\n",
        "        \n",
        "        embeddings = nn.Embedding(weight.shape[0], weight.shape[1])\n",
        "        embeddings.weight = nn.Parameter(torch.from_numpy(weight).float())\n",
        "        return embeddings, weight.shape[1]\n",
        "    \n",
        "    @staticmethod\n",
        "    def from_binary(filename, vocab, unif=0.25):\n",
        "        def read_word(f):\n",
        "\n",
        "            s = bytearray()\n",
        "            ch = f.read(1)\n",
        "\n",
        "            while ch != b' ':\n",
        "                s.extend(ch)\n",
        "                ch = f.read(1)\n",
        "            s = s.decode('utf-8')\n",
        "            # Only strip out normal space and \\n not other spaces which are words.\n",
        "            return s.strip(' \\n')\n",
        "\n",
        "        vocab_size = len(vocab)\n",
        "        with io.open(filename, \"rb\") as f:\n",
        "            header = f.readline()\n",
        "            file_vocab_size, embed_dim = map(int, header.split())\n",
        "            weight = init_embeddings(len(vocab), embed_dim, unif)\n",
        "            if '[PAD]' in vocab:\n",
        "                weight[vocab['[PAD]']] = 0.0\n",
        "            width = 4 * embed_dim\n",
        "            for i in range(file_vocab_size):\n",
        "                word = read_word(f)\n",
        "                raw = f.read(width)\n",
        "                if word in vocab:\n",
        "                    vec = np.fromstring(raw, dtype=np.float32)\n",
        "                    weight[vocab[word]] = vec\n",
        "        embeddings = nn.Embedding(weight.shape[0], weight.shape[1])\n",
        "        embeddings.weight = nn.Parameter(torch.from_numpy(weight).float())\n",
        "        return embeddings, embed_dim\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SOeo4W0zSvws",
        "colab_type": "text"
      },
      "source": [
        "### Now to run some stuff!\n",
        "\n",
        "We did a lot of work to set things up, but its pretty boilerplate and we will reuse a lot of it.  So far, we made 2 classifiers we can run along with code to train and evaluate our models, and a reader to load our data.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ahoBQQ-LddMr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "BASE = 'sst2'\n",
        "TRAIN = os.path.join(BASE, 'stsa.binary.phrases.train')\n",
        "VALID = os.path.join(BASE, 'stsa.binary.dev')\n",
        "TEST = os.path.join(BASE, 'stsa.binary.test')\n",
        "PRETRAINED_EMBEDDINGS_FILE = 'GoogleNews-vectors-negative300.bin'\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kbWM-ycVeL5b",
        "colab_type": "text"
      },
      "source": [
        "Lets read in our datasets:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tCwkXTO7eQEx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "r = Reader((TRAIN, VALID, TEST,))\n",
        "train = r.load(TRAIN)\n",
        "valid = r.load(VALID)\n",
        "test = r.load(TEST)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tJXsq3-xiATu",
        "colab_type": "text"
      },
      "source": [
        "## Model trained with randomly initialized embeddings\n",
        "\n",
        "First, we are going to train a model without any pretrained embeddings for 10 epochs.  During training, we will see the training and validation performance, and after the final epoch, we will see the results from the best model trained on these epochs. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "okeGF78yh3y_",
        "colab_type": "code",
        "outputId": "5a5fc139-8da5-400c-b5c5-de84de3d9b0a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "embed_dim = 300\n",
        "embeddings = nn.Embedding(len(r.vocab), embed_dim)\n",
        "model  = ConvClassifier(embeddings, len(r.labels), embed_dim)\n",
        "\n",
        "num_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "print(f\"Model has {num_params} parameters\") \n",
        "\n",
        "\n",
        "model.to('cuda:0')\n",
        "loss = torch.nn.NLLLoss()\n",
        "loss = loss.to('cuda:0')\n",
        "\n",
        "learnable_params = [p for p in model.parameters() if p.requires_grad]\n",
        "optimizer = torch.optim.Adadelta(learnable_params, lr=1.0)\n",
        "\n",
        "fit(model, r.labels, optimizer, loss, 10, 50, train, valid, test)\n"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model has 5442302 parameters\n",
            "EPOCH 1\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.5926248359558738, 'precision': 0.6179160630500119, 'recall': 0.6762583118388982, 'f1': 0.6457721335924437}\n",
            "Validation Results\n",
            "{'acc': 0.7110091743119266, 'precision': 0.7622950819672131, 'recall': 0.6283783783783784, 'f1': 0.6888888888888889}\n",
            "New best model 0.71\n",
            "EPOCH 2\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.6340224269435168, 'precision': 0.6503253333333333, 'recall': 0.7213611301734542, 'f1': 0.6840038593578207}\n",
            "Validation Results\n",
            "{'acc': 0.6089449541284404, 'precision': 0.5695006747638327, 'recall': 0.9504504504504504, 'f1': 0.7122362869198312}\n",
            "EPOCH 3\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.6637647639713621, 'precision': 0.6763477437133999, 'recall': 0.7433919401784236, 'f1': 0.7082868319298364}\n",
            "Validation Results\n",
            "{'acc': 0.7144495412844036, 'precision': 0.7506426735218509, 'recall': 0.6576576576576577, 'f1': 0.7010804321728692}\n",
            "New best model 0.71\n",
            "EPOCH 4\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.6873481373682775, 'precision': 0.6971548679277991, 'recall': 0.7613289476797842, 'f1': 0.7278300606279975}\n",
            "Validation Results\n",
            "{'acc': 0.7075688073394495, 'precision': 0.6556836902800659, 'recall': 0.8963963963963963, 'f1': 0.7573739295908659}\n",
            "EPOCH 5\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.7016800717246398, 'precision': 0.7109843018933928, 'recall': 0.7695165526870016, 'f1': 0.7390933781833471}\n",
            "Validation Results\n",
            "{'acc': 0.6490825688073395, 'precision': 0.597457627118644, 'recall': 0.9527027027027027, 'f1': 0.734375}\n",
            "EPOCH 6\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.7181819363054014, 'precision': 0.7250941083778342, 'recall': 0.783998674838496, 'f1': 0.7533967777512478}\n",
            "Validation Results\n",
            "{'acc': 0.7591743119266054, 'precision': 0.7378048780487805, 'recall': 0.8175675675675675, 'f1': 0.7756410256410255}\n",
            "New best model 0.76\n",
            "EPOCH 7\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.7320201140837567, 'precision': 0.7382341929658423, 'recall': 0.7932274781703306, 'f1': 0.7647434581251569}\n",
            "Validation Results\n",
            "{'acc': 0.7431192660550459, 'precision': 0.7370689655172413, 'recall': 0.7702702702702703, 'f1': 0.7533039647577091}\n",
            "EPOCH 8\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.7397253154194982, 'precision': 0.7452230704735008, 'recall': 0.7992380321351664, 'f1': 0.7712860095226134}\n",
            "Validation Results\n",
            "{'acc': 0.6788990825688074, 'precision': 0.6209439528023599, 'recall': 0.9481981981981982, 'f1': 0.7504456327985739}\n",
            "EPOCH 9\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.7485089850703603, 'precision': 0.7541725852272727, 'recall': 0.8040890697839513, 'f1': 0.7783313290958025}\n",
            "Validation Results\n",
            "{'acc': 0.7545871559633027, 'precision': 0.7281746031746031, 'recall': 0.8265765765765766, 'f1': 0.7742616033755274}\n",
            "EPOCH 10\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.7572016995621159, 'precision': 0.7618233111935491, 'recall': 0.8115431032442794, 'f1': 0.7858976121728768}\n",
            "Validation Results\n",
            "{'acc': 0.7591743119266054, 'precision': 0.7303149606299213, 'recall': 0.8355855855855856, 'f1': 0.7794117647058825}\n",
            "Final result\n",
            "{'acc': 0.7391543108182317, 'precision': 0.728421052631579, 'recall': 0.7612761276127613, 'f1': 0.7444862829478214}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.7391543108182317"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a54pDMAQjXY-",
        "colab_type": "text"
      },
      "source": [
        "Yikes, thats not very encouraging!  What about our LSTM?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f8s4lSIqjc6G",
        "colab_type": "code",
        "outputId": "0e1b7450-210d-46cf-9c7c-d7120991fe15",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "embed_dim = 300\n",
        "embeddings = nn.Embedding(len(r.vocab), embed_dim)\n",
        "model  = LSTMClassifier(embeddings, len(r.labels), embed_dim, 100, hidden_units=[100])\n",
        "\n",
        "num_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "print(f\"Model has {num_params} parameters\") \n",
        "\n",
        "\n",
        "model.to('cuda:0')\n",
        "loss = torch.nn.NLLLoss()\n",
        "loss = loss.to('cuda:0')\n",
        "\n",
        "learnable_params = [p for p in model.parameters() if p.requires_grad]\n",
        "optimizer = torch.optim.Adadelta(learnable_params, lr=1.0)\n",
        "\n",
        "fit(model, r.labels, optimizer, loss, 10, 50, train, valid, test)\n"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py:54: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.5 and num_layers=1\n",
            "  \"num_layers={}\".format(dropout, num_layers))\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Model has 5342502 parameters\n",
            "EPOCH 1\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.6416496667143099, 'precision': 0.648109286089027, 'recall': 0.7600511133722994, 'f1': 0.6996307873269656}\n",
            "Validation Results\n",
            "{'acc': 0.7488532110091743, 'precision': 0.7300613496932515, 'recall': 0.8040540540540541, 'f1': 0.765273311897106}\n",
            "New best model 0.75\n",
            "EPOCH 2\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.7467158690765453, 'precision': 0.7576038743550285, 'recall': 0.7921862798457133, 'f1': 0.7745092368734602}\n",
            "Validation Results\n",
            "{'acc': 0.7786697247706422, 'precision': 0.747534516765286, 'recall': 0.8536036036036037, 'f1': 0.7970557308096741}\n",
            "New best model 0.78\n",
            "EPOCH 3\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.7794467327607489, 'precision': 0.7900387712496272, 'recall': 0.8149033342009986, 'f1': 0.8022784456248252}\n",
            "Validation Results\n",
            "{'acc': 0.783256880733945, 'precision': 0.7958236658932715, 'recall': 0.7725225225225225, 'f1': 0.7839999999999999}\n",
            "New best model 0.78\n",
            "EPOCH 4\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.7937786671171113, 'precision': 0.8048943938623654, 'recall': 0.8242267919259803, 'f1': 0.8144458863830334}\n",
            "Validation Results\n",
            "{'acc': 0.7901376146788991, 'precision': 0.7713097713097713, 'recall': 0.8355855855855856, 'f1': 0.8021621621621621}\n",
            "New best model 0.79\n",
            "EPOCH 5\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.8033159652291421, 'precision': 0.814348632359759, 'recall': 0.8313258714120069, 'f1': 0.8227496809096125}\n",
            "Validation Results\n",
            "{'acc': 0.7878440366972477, 'precision': 0.8018648018648019, 'recall': 0.7747747747747747, 'f1': 0.7880870561282932}\n",
            "EPOCH 6\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.8122165772274269, 'precision': 0.8227892183038098, 'recall': 0.8386379232826143, 'f1': 0.8306379787184175}\n",
            "Validation Results\n",
            "{'acc': 0.7889908256880734, 'precision': 0.8125, 'recall': 0.7612612612612613, 'f1': 0.786046511627907}\n",
            "EPOCH 7\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.8170501942542326, 'precision': 0.8279907814791536, 'recall': 0.841666863863319, 'f1': 0.8347728126173487}\n",
            "Validation Results\n",
            "{'acc': 0.8027522935779816, 'precision': 0.7995594713656388, 'recall': 0.8175675675675675, 'f1': 0.8084632516703786}\n",
            "New best model 0.80\n",
            "EPOCH 8\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.8234690297683243, 'precision': 0.8332597224482206, 'recall': 0.8482453441870371, 'f1': 0.8406857571706653}\n",
            "Validation Results\n",
            "{'acc': 0.7924311926605505, 'precision': 0.8065268065268065, 'recall': 0.7792792792792793, 'f1': 0.7926689576174113}\n",
            "EPOCH 9\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.824742401995816, 'precision': 0.8359920588578769, 'recall': 0.846991173477839, 'f1': 0.8414556738839128}\n",
            "Validation Results\n",
            "{'acc': 0.7981651376146789, 'precision': 0.8116279069767441, 'recall': 0.786036036036036, 'f1': 0.7986270022883294}\n",
            "EPOCH 10\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.8271332233209028, 'precision': 0.8384886956115125, 'recall': 0.8486476253579119, 'f1': 0.8435375749735388}\n",
            "Validation Results\n",
            "{'acc': 0.7993119266055045, 'precision': 0.799554565701559, 'recall': 0.8085585585585585, 'f1': 0.8040313549832027}\n",
            "Final result\n",
            "{'acc': 0.8050521691378364, 'precision': 0.7991360691144709, 'recall': 0.8140814081408141, 'f1': 0.8065395095367847}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.8050521691378364"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tHtNnvvoj22N",
        "colab_type": "text"
      },
      "source": [
        "## Same model with pre-trained word embeddings\n",
        "\n",
        "The models below are identical to the ones above, the only difference is that we are going to initialize the embeddings using our previously defined `EmbeddingsReader`.  First lets take a look at our CNN model again.  Notice we only run 5 epochs here instead of 10!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cEzWqC2fkNSz",
        "colab_type": "code",
        "outputId": "eae932aa-fa84-41cd-e1b4-2791db35167b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 663
        }
      },
      "source": [
        "embeddings, embed_dim = EmbeddingsReader.from_binary(PRETRAINED_EMBEDDINGS_FILE, r.vocab)\n",
        "model  = ConvClassifier(embeddings, len(r.labels), embed_dim)\n",
        "num_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "print(f\"Model has {num_params} parameters\") \n",
        "\n",
        "\n",
        "model.to('cuda:0')\n",
        "loss = torch.nn.NLLLoss()\n",
        "loss = loss.to('cuda:0')\n",
        "\n",
        "learnable_params = [p for p in model.parameters() if p.requires_grad]\n",
        "optimizer = torch.optim.Adadelta(learnable_params, lr=1.0)\n",
        "\n",
        "fit(model, r.labels, optimizer, loss, 5, 50, train, valid, test)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:60: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Model has 5442302 parameters\n",
            "EPOCH 1\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.8329153727212484, 'precision': 0.8410831129054712, 'recall': 0.8577817742965995, 'f1': 0.8493503754817998}\n",
            "Validation Results\n",
            "{'acc': 0.8268348623853211, 'precision': 0.9080779944289693, 'recall': 0.7342342342342343, 'f1': 0.8119551681195517}\n",
            "New best model 0.83\n",
            "EPOCH 2\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.8798742220085498, 'precision': 0.8858578775128565, 'recall': 0.8967793842731726, 'f1': 0.8912851750373358}\n",
            "Validation Results\n",
            "{'acc': 0.8474770642201835, 'precision': 0.8373101952277657, 'recall': 0.8693693693693694, 'f1': 0.8530386740331491}\n",
            "New best model 0.85\n",
            "EPOCH 3\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.8955834773456686, 'precision': 0.9008221873462791, 'recall': 0.9100309993137556, 'f1': 0.9054031783402}\n",
            "Validation Results\n",
            "{'acc': 0.8486238532110092, 'precision': 0.8436123348017621, 'recall': 0.8626126126126126, 'f1': 0.8530066815144767}\n",
            "New best model 0.85\n",
            "EPOCH 4\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9059523654838165, 'precision': 0.9103606664948091, 'recall': 0.9192361390473035, 'f1': 0.91477687507359}\n",
            "Validation Results\n",
            "{'acc': 0.841743119266055, 'precision': 0.9090909090909091, 'recall': 0.7657657657657657, 'f1': 0.8312958435207825}\n",
            "EPOCH 5\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9139694130793519, 'precision': 0.9181293410925474, 'recall': 0.9258856101658818, 'f1': 0.9219911634756995}\n",
            "Validation Results\n",
            "{'acc': 0.8428899082568807, 'precision': 0.851258581235698, 'recall': 0.8378378378378378, 'f1': 0.844494892167991}\n",
            "Final result\n",
            "{'acc': 0.8725974739154311, 'precision': 0.8537095088819227, 'recall': 0.8987898789878987, 'f1': 0.8756698821007501}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.8725974739154311"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QcnajM3klDx9",
        "colab_type": "text"
      },
      "source": [
        "Much better!  And now the LSTM!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zqJ12jfAkxoX",
        "colab_type": "code",
        "outputId": "76774035-dad3-445c-a7ef-20005a636a8e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 683
        }
      },
      "source": [
        "model  = LSTMClassifier(embeddings, len(r.labels), embed_dim, 100, hidden_units=[100])\n",
        "\n",
        "num_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "print(f\"Model has {num_params} parameters\") \n",
        "\n",
        "\n",
        "model.to('cuda:0')\n",
        "loss = torch.nn.NLLLoss()\n",
        "loss = loss.to('cuda:0')\n",
        "\n",
        "learnable_params = [p for p in model.parameters() if p.requires_grad]\n",
        "optimizer = torch.optim.Adadelta(learnable_params, lr=1.0)\n",
        "\n",
        "fit(model, r.labels, optimizer, loss, 5, 50, train, valid, test)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py:54: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.5 and num_layers=1\n",
            "  \"num_layers={}\".format(dropout, num_layers))\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Model has 5342502 parameters\n",
            "EPOCH 1\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.870012084042567, 'precision': 0.8834494400722794, 'recall': 0.8792683215409736, 'f1': 0.8813539220569748}\n",
            "Validation Results\n",
            "{'acc': 0.8371559633027523, 'precision': 0.8081632653061225, 'recall': 0.8918918918918919, 'f1': 0.8479657387580299}\n",
            "New best model 0.84\n",
            "EPOCH 2\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.904562050908902, 'precision': 0.9189545934530094, 'recall': 0.9061028419981543, 'f1': 0.9124834677755669}\n",
            "Validation Results\n",
            "{'acc': 0.8646788990825688, 'precision': 0.9137055837563451, 'recall': 0.8108108108108109, 'f1': 0.8591885441527446}\n",
            "New best model 0.86\n",
            "EPOCH 3\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9171658372422395, 'precision': 0.9286806517895542, 'recall': 0.9197804018078989, 'f1': 0.9242090996635478}\n",
            "Validation Results\n",
            "{'acc': 0.8600917431192661, 'precision': 0.8340248962655602, 'recall': 0.9054054054054054, 'f1': 0.8682505399568036}\n",
            "EPOCH 4\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9252738399968815, 'precision': 0.9348261076703192, 'recall': 0.9286542511654322, 'f1': 0.9317299588076782}\n",
            "Validation Results\n",
            "{'acc': 0.856651376146789, 'precision': 0.8444924406047516, 'recall': 0.8806306306306306, 'f1': 0.8621830209481808}\n",
            "EPOCH 5\n",
            "=================================\n",
            "Training Results\n",
            "{'acc': 0.9308091111082236, 'precision': 0.9387294497766796, 'recall': 0.9350197591045695, 'f1': 0.9368709321762635}\n",
            "Validation Results\n",
            "{'acc': 0.8635321100917431, 'precision': 0.9156010230179028, 'recall': 0.8063063063063063, 'f1': 0.8574850299401198}\n",
            "Final result\n",
            "{'acc': 0.8802855573860516, 'precision': 0.9157641395908543, 'recall': 0.8371837183718371, 'f1': 0.8747126436781609}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.8802855573860516"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NpfM53E-lS9y",
        "colab_type": "text"
      },
      "source": [
        "#### A quick note about these models on this data\n",
        "\n",
        "Both of these models are surprisingly strong baselines and do fairly well on this dataset averaged over many runs.  Even with only 2-5 epochs of data it is quite common to see scores higher than in the Kim 2014 paper."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O4mLnxyIlM4u",
        "colab_type": "text"
      },
      "source": [
        "## Conclusions\n",
        "\n",
        "### Its not hard to get good performance with a Deep Learning model for Text Classification\n",
        "\n",
        "We saw above how to get good results on the SST-2 dataset using fairly simple deep learning models, even with very few training epochs.  This behavior is not limited to a single dataset -- these results have been shown over and over.  Also, using PyTorch, we were able to code an entire pipeline in this minimalistic notebook.\n",
        "\n",
        "### Pre-trained embeddings often help\n",
        "\n",
        "We can see that pre-trained embeddings can have a massive impact on the performance of our models, especially for smaller datasets.  The `word2vec` algorithm caused an explosion in the NLP community -- even though pre-training embeddings had been widely studied prior to that work, the results were reliable and fast.  `GloVe` and `fastText` embeddings came shortly thereafter, and all 3 models are in quite common use today.  The code above can load any of these flavors of embeddings and incorporate them into downstream models for large improvements.\n",
        "\n",
        "For large datasets, like those used in Language Modeling and Neural Machine Translation, models are typically trained from random embeddings, which are sufficient in those cases.\n",
        "\n",
        "### Incorporating pre-trained embeddings into your model is simple\n",
        "\n",
        "The file formats are very simple to read and can be incorporated with only a few lines of code.  In some cases, memory-mapping the file can increase the loading speed.  This is implemented in [Baseline](https://github.com/dpressel/baseline/)\n",
        "\n",
        "### Some Further Resources\n",
        "\n",
        "- [The Unreasonable Effectiveness of Recurrent Neural Networks, Karpathy, 2015](https://karpathy.github.io/2015/05/21/rnn-effectiveness/)\n",
        "- Tensorflow tutorial for word2vec: https://www.tensorflow.org/tutorials/representation/word2vec\n",
        "- Tensorflow docs on feature columns (for images above): https://www.tensorflow.org/guide/feature_columns\n",
        "- Xin Rong, wrote some amazing software to visualize word embeddings and the training process.  Sadly, Xin is no longer with us -- he was a great researcher and an awesome guy.  We miss him.\n",
        "  - https://ronxin.github.io/wevi/\n",
        "  - Accompanying talk from a2-dlearn 2015: https://www.youtube.com/channel/UCVdeq2cIxnujw2kTdzg2N5g\n",
        "  - https://ronxin.github.io/lamvi/dist/#model=word2vec&backend=browser&query_in=darcy&query_out=G_bennet,B_circumstances\n",
        "  - Accompanying paper for Lamvi [Visual Tools for Debugging Neural Language Models, Rong & Adar, 2016](http://www.cond.org/ICML16_NeuralVis.pdf)\n",
        "\n"
      ]
    }
  ]
}