1_Improved_Model.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "1_Improved_Model.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Doometnick/ConvNet-Workflow/blob/master/1_Improved_Model.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wMLksMoFX4Aj",
        "colab_type": "text"
      },
      "source": [
        "Remark: I recommend to run this in Google Colab. Click the link above.\n",
        "\n",
        "# Improving Model Accuracy\n",
        "\n",
        "In the first session, we have built a simple regularized convolutional network that does not seem to overfit. The accuracy, however, is too low. There are a many ways to improve this, four of these are briefly explained below.\n",
        "\n",
        "First, we need to import the data again and create the training and test sets. This is the same code than in the first session."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eksbtkDeYA_n",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import keras\n",
        "import tensorflow as tf\n",
        "\n",
        "import functools\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import os"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A3BgVjmoYCI8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "assert tf.test.is_gpu_available(), \"GPU not enabled. Enable under Runtime > Change runtime type\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nviL4hgIYDVV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "(x_train, y_train), (x_test, y_test) = keras.datasets.cifar10.load_data()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lBgVA370YEnT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x_train = x_train / 255.0\n",
        "x_test = x_test / 255.0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9AT5nhfmnDPy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def plot_training_history(history, title=None):\n",
        "    fig = plt.figure(figsize=(9,9))\n",
        "    plt.plot(history['acc'], label='acc')\n",
        "    plt.plot(history['val_acc'], label='val_acc')\n",
        "    plt.plot(history['loss'], label='loss')\n",
        "    plt.plot(history['val_loss'], label='val_loss')\n",
        "    plt.xlabel('Epoch')\n",
        "    plt.ylabel('Accuracy')\n",
        "    plt.legend(loc='best')\n",
        "    if title is not None:\n",
        "        plt.suptitle(title)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5w1jM7_dYGGh",
        "colab_type": "text"
      },
      "source": [
        "### 1. Data augmentation\n",
        "\n",
        "A way to avoid overfitting is by increasing the amount of data. We can easily increase the number of data points by copying and manipulating the images that we have. For example, rotating, zooming in and out, or cropping single images can generate new images that are slightly different than the originals. The increased number of data points should help to further increase the quality of our model.\n",
        "\n",
        "### 2. More complex network architecture\n",
        "\n",
        "The current model architecture is fairly simple with only two convolutional layers followed by max pooling and one hidden dense layer before the final output. More complicated architectures can be very powerful in detecting features. For example, convolutional networks applied on images tend to detect local detail features in shallow layers (close to the input layer) and global features in deeper layers. This effect can be seen in an illustrative way in a [neural style transfer paper by Gatys et al](https://arxiv.org/abs/1508.06576) where they use shallow and deep layers from the [vgg model](https://arxiv.org/abs/1409.1556) to extract local and global features to create artistic images.\n",
        "\n",
        "As the CIFAR-10 images we are woring with can also be partly distinguished by global features (for example shapes of animals, cars, etc.), we can make our network deeper to see if it increases the accuracy.\n",
        "\n",
        "### 3. Batch normalization\n",
        "\n",
        "Similar to normalizing input data, [batch normalization](https://arxiv.org/abs/1502.03167?context=cs) can significantly increase the training speed of a network by 'reducing covariate shift'. This means that weights of deeper networks are made more stable with respect to changes in weights in earlier layers. To achieve this, values of hidden layers are scaled to have zero mean and a standard deviation of one. This is normalization is done on mini-batches, which can lead to a regularizing effect depending on the size of the mini-batches. Furthermore, it can be applied either before or after the activation function with different effects. But we will not get deeper into those topics here. \n",
        "\n",
        "### 4. Dropout\n",
        "\n",
        "Dropout is another form of regularzation that can help to avoid overfitting. If Dropout is applied to a certain hidden layer in a network, a certain percentage of all nodes are assigned weights of 0 from the previous layer. The network is trained with the dropped out nodes for a short while and the weight of the nodes adjusted accordingly. Afterwards, the dropped nodes are reset and a new set of random nodes is removed for the next training step.\n",
        "\n",
        "The motivation of Dropout is to build different models. Each different model will fit towards certain features in the data - and each model might overfit towards these features. But the collection of all models, the average of it, will lead to a reguarized, not overfitted model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TCRgi3qAYIW2",
        "colab_type": "text"
      },
      "source": [
        "## New Improved Model\n",
        "With all that said, let's build our improved model containing all the points from above.\n",
        "\n",
        "\n",
        "First, let's augment our data set by utilizing keras' preprocessing feature ImageDataGenerator. This is a generator object that will return infinite sets of images that contain certain manipulations.\n",
        "\n",
        "The manipulations chosen in this example are flipping the images horizontally, rotating the angles, and shifting the picture in width and height.\n",
        "\n",
        "Such an image generator is not a data set that we have been using before, but a generator that yields one data set at a time. Therefore, we will have to change our model.fit() function to a model.fit_generator() function in a later step."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ej5YA4AJYL5b",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data_generator = keras.preprocessing.image.ImageDataGenerator(\n",
        "    horizontal_flip=True,\n",
        "    rotation_range=5)\n",
        "    # width_shift_range=0.05,\n",
        "    # height_shift_range=0.05,\n",
        "    # zoom_range=0.05)\n",
        "\n",
        "data_generator.fit(x_train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CyW8edKuYM_5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def build_improved_model():\n",
        "\n",
        "    n_filters = 64\n",
        "    weight_decay = 0.001\n",
        "\n",
        "    Activation = functools.partial(tf.keras.layers.Activation, activation=\"relu\")\n",
        "    Conv2D = functools.partial(tf.keras.layers.Conv2D, \n",
        "                               activation=None, \n",
        "                               padding=\"same\", \n",
        "                               kernel_regularizer=tf.keras.regularizers.l2(weight_decay))\n",
        "    MaxPool = functools.partial(tf.keras.layers.MaxPool2D, pool_size=(2,2))\n",
        "    Flatten = tf.keras.layers.Flatten\n",
        "    Dense = tf.keras.layers.Dense\n",
        "    BatchNormalization = tf.keras.layers.BatchNormalization\n",
        "    Dropout = tf.keras.layers.Dropout\n",
        "\n",
        "    model = tf.keras.Sequential([\n",
        "        \n",
        "        Conv2D(filters=n_filters*1, kernel_size=(3,3), strides=(1,1), input_shape=(32, 32, 3)),\n",
        "        BatchNormalization(),\n",
        "        Activation(),\n",
        "        Conv2D(filters=n_filters*1, kernel_size=(3,3), strides=(1,1)),\n",
        "        BatchNormalization(),\n",
        "        Activation(),\n",
        "        Conv2D(filters=n_filters*1, kernel_size=(3,3), strides=(1,1)),\n",
        "        BatchNormalization(),\n",
        "        Activation(),\n",
        "        MaxPool(),\n",
        "        \n",
        "\n",
        "        Conv2D(filters=n_filters*2, kernel_size=(3,3), strides=(1,1)),\n",
        "        BatchNormalization(),\n",
        "        Activation(),\n",
        "        Conv2D(filters=n_filters*2, kernel_size=(3,3), strides=(1,1)),\n",
        "        BatchNormalization(),\n",
        "        Activation(),\n",
        "        Conv2D(filters=n_filters*2, kernel_size=(3,3), strides=(1,1)),\n",
        "        BatchNormalization(),\n",
        "        Activation(),\n",
        "        MaxPool(),\n",
        "        \n",
        "\n",
        "        Conv2D(filters=n_filters*3, kernel_size=(3,3), strides=(1,1)),\n",
        "        BatchNormalization(),\n",
        "        Activation(),\n",
        "        Conv2D(filters=n_filters*3, kernel_size=(3,3), strides=(1,1)),\n",
        "        BatchNormalization(),\n",
        "        Activation(),\n",
        "        Conv2D(filters=n_filters*3, kernel_size=(3,3), strides=(1,1)),\n",
        "        BatchNormalization(),\n",
        "        Activation(),\n",
        "        MaxPool(),\n",
        "\n",
        "        Flatten(),\n",
        "\n",
        "        Dense(128, activation=None, kernel_regularizer=tf.keras.regularizers.l2(weight_decay)),\n",
        "        Dropout(0.5),\n",
        "        BatchNormalization(),\n",
        "        Activation(),\n",
        "\n",
        "        Dense(10, activation=\"softmax\")\n",
        "    ])\n",
        "\n",
        "    return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KLTZZWqoa2Hr",
        "colab_type": "text"
      },
      "source": [
        "There are four major blocks in this model: three stacks of convolutional layers followed by max pooling, and dense layers at the end. The dense layers transform the flattened output from the convolution into the ten labels describing the images. \n",
        "The convolutional stacks each consist of three convolutional layers without any max pooling in between. Such a structure could be replaced by only one convolutional layers that is larger (i.e. has a larger receptive field/kernel_size). But having three layers with smalle receptive fields leads to a lower number of total parameters used and has a more discriminative decision function since we apply the activation (relu) three times instead of one. This is described in more detail in a [paper by Simonyan & Zisserman](https://arxiv.org/abs/1409.1556).\n",
        "\n",
        "\n",
        "Our new model requires a lot more training than the old one, since it has many more parameters. Therefore, we will create two callback functions that are periodically called while training."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vl28u1nZaPe2",
        "colab_type": "text"
      },
      "source": [
        "### Reducing learning rate on plateau\n",
        "\n",
        "Since we will be training much longer than before, it might be beneficial to have a dynamic learning rate. If a certain metric of our model (here: validation loss) won't improve for a certain amount of periods, the learning rate will be automatically decreased by the factor of 10."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SifjR0cBYOAH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "lr_schedule = tf.keras.callbacks.ReduceLROnPlateau(monitor=\"val_loss\", \n",
        "                                                   factor=0.1,\n",
        "                                                   patience=5, \n",
        "                                                   min_lr=1e-08,\n",
        "                                                   verbose=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1LF4HOcHZljS",
        "colab_type": "text"
      },
      "source": [
        "### Saving training data\n",
        "In order to not lose the model's weights after a training session, we can save them by introducing a callback function that saves the weights every couple of iterations. If we decide to continue training, we can just pick up from where we left off. \n",
        "\n",
        "Note, however, that all saved data in colab will be removed after 24 hours. We can avoid this issue by saving data on the Google Drive, but this will not be covered here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bgIII8fUY-a7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "c0b323b7-9840-4248-ebbe-a0453b3237ab"
      },
      "source": [
        "# Saving training checkpoints, as training takes longer time now.\n",
        "checkpoint_path = \"training_1/cp.ckpt\"\n",
        "checkpoint_dir = os.path.dirname(checkpoint_path)\n",
        "\n",
        "# Create a callback that saves the model's weights.\n",
        "cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,\n",
        "                                                 save_weights_only=True,\n",
        "                                                 verbose=1,\n",
        "                                                 period=10)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:`period` argument is deprecated. Please use `save_freq` to specify the frequency in number of samples seen.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8wwSbFE3dT0t",
        "colab_type": "text"
      },
      "source": [
        "### Compile and train model\n",
        "Since we work with a [generator](https://wiki.python.org/moin/Generators) now instead of a normal variable that carries all the data, we need to change the model.fit function to model.fit_generator."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WsPCzTiIYO8R",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def compile_and_train_generator(model, data_generator, epochs=20):\n",
        "\n",
        "    # Batch normalization allows for larger learning rate.\n",
        "    optimizer = tf.keras.optimizers.Adam(learning_rate=0.001, amsgrad=True)\n",
        "\n",
        "    model.compile(optimizer=optimizer,\n",
        "                loss=\"sparse_categorical_crossentropy\",\n",
        "                metrics=[\"accuracy\"])\n",
        "    \n",
        "    # If we have saved weights, load them to continue training.\n",
        "    latest = tf.train.latest_checkpoint(checkpoint_dir)\n",
        "    if latest is not None:\n",
        "        model.load_weights(latest)\n",
        "\n",
        "    batch_size = 256\n",
        "    \n",
        "    history = model.fit_generator(\n",
        "        data_generator.flow(x_train, y_train, batch_size=batch_size), \n",
        "        steps_per_epoch=len(x_train) / batch_size, \n",
        "        epochs=epochs,\n",
        "        validation_data=(x_test, y_test),\n",
        "        callbacks=[lr_schedule, cp_callback])\n",
        "\n",
        "    return history"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E3lysXAFYP_d",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# In case we do multiple runs, save histories in a list and concatenate later.\n",
        "histories = []  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zOo_5SnxYRb0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "improved_model = build_improved_model()\n",
        "improved_history = compile_and_train_generator(improved_model, data_generator, epochs=50)\n",
        "histories.append(improved_history)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qj7k1tevYShd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "total_history = {}\n",
        "\n",
        "for history in histories:\n",
        "    for key in [\"acc\", \"val_acc\", \"loss\", \"val_loss\"]:\n",
        "        total_history.setdefault(key, []).extend(history.history[key].copy())\n",
        "\n",
        "plot_training_history(total_history, title=\"Regularized Model\")\n",
        "improved_model.evaluate(x_test, y_test)"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}