notebooks/1.0-ImageClassification.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/wiso/TutorialML-AtlasItalia2022/blob/main/notebooks/1.0-ImageClassification.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Prh_iK0wFuI8"
      },
      "source": [
        "# 1.0 Image Classification\n",
        "The goal of this tutorial is to familiarize with simple feed-forward neural networks. The goal is to classify images in one of the ten possible classes.\n",
        "\n",
        "The classification is based on a neural network. The first step is to flatten the input gray scale image (28x28) into a 1D array. In this way we can feed this input to a first layer of a neural network.\n",
        "\n",
        "The parameter of the neural network will tuned to minimize the loss, in this case the cross-entropy.\n",
        "\n",
        "The steps will be:\n",
        "\n",
        "   * load and preprocess the inputs, split in train and test dataset\n",
        "   * define the model\n",
        "   * define the loss\n",
        "   * train with the training dataset\n",
        "   * evaluate the performance with the test dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8qE5ezhTFuJC"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "from matplotlib import pyplot as plt\n",
        "import seaborn as sns"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fUbFJ3J-KFOc"
      },
      "source": [
        "## Load the dataset\n",
        "\n",
        "\n",
        "> Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. Han Xiao, Kashif Rasul, Roland Vollgraf. arXiv:1708.07747\n",
        "\n",
        "Images are 2D array 28x28, in grayscale. Each image is associated to a label, definig the class. There are 10 classes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "npj8BSJhFuJE"
      },
      "outputs": [],
      "source": [
        "fashion_mnist = tf.keras.datasets.fashion_mnist\n",
        "\n",
        "(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()\n",
        "class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',\n",
        "               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']\n",
        "nclasses = len(class_names)\n",
        "\n",
        "# summarize loaded dataset\n",
        "print('Train: X=%s, y=%s' % (train_images.shape, train_labels.shape))\n",
        "print('Test: X=%s, y=%s' % (test_images.shape, test_labels.shape))\n",
        "print(\"unique train labels=%s\" % np.unique(train_labels))\n",
        "print(\"range values first train img = %s, %s\" % (train_images[0].min(), train_images[0].max()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5YmTYS_LmOns"
      },
      "source": [
        "### Preprocess\n",
        "Normalize the images since each pixel is a number between 0 and 1."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "s2Oa5JOvmM4M"
      },
      "outputs": [],
      "source": [
        "train_images = train_images / 255.\n",
        "test_images = test_images / 255."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hcd7b1EjM7f-"
      },
      "source": [
        "## Example\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5kHTqioxFuJF"
      },
      "outputs": [],
      "source": [
        "plt.imshow(train_images[0], cmap='binary')\n",
        "plt.axis('off')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Iv7q5cpANlYr"
      },
      "source": [
        "### Check the frequencies of the labels\n",
        "The dataset is balanced"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "GZOj2qjqFuJF"
      },
      "outputs": [],
      "source": [
        "#@title\n",
        "fig, ax = plt.subplots()\n",
        "ax.hist(train_labels, bins=np.arange(nclasses + 1), label='train')\n",
        "ax.hist(test_labels, bins=np.arange(nclasses + 1), label='test')\n",
        "ax.set_xticks(np.arange(nclasses) + 0.5)\n",
        "ax.set_xticklabels(class_names, rotation=90)\n",
        "ax.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4DesorH3SLNC"
      },
      "source": [
        "### Display examples"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "jxgbjow2FuJG"
      },
      "outputs": [],
      "source": [
        "fig, axs = plt.subplots(10, 20, figsize=(13, 7), gridspec_kw=dict(wspace=0.0, hspace=0.0,\n",
        "                        top=1. - 0.5 / (10 + 1), bottom=0.5 / (10 + 1),\n",
        "                        left=0.5 / (20 + 1), right=1 - 0.5 / (20 + 1)),)\n",
        "for iclass, ax_row in enumerate(axs):\n",
        "    imgs = train_images[train_labels == iclass][:20]\n",
        "    label = class_names[iclass]\n",
        "    for ax, img in zip(ax_row, imgs):\n",
        "        ax.set_xticks([])\n",
        "        ax.set_yticks([])\n",
        "        ax.imshow(img, cmap='binary')\n",
        "        ax.axis('equal')\n",
        "        if ax.get_subplotspec().colspan.start == 0:\n",
        "            ax.set_ylabel(label, fontsize=12)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5mm6quwCTrxy"
      },
      "source": [
        "## Create the model\n",
        "### Using the functional API\n",
        "Note that here we are just defining the computational graph representing the model. Nothing is computed here."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yzJ06GScSvvM"
      },
      "outputs": [],
      "source": [
        "inputs = tf.keras.Input(shape=(28, 28), name='img')    # the input placeholder\n",
        "x = tf.keras.layers.Flatten()(inputs)                  # flatten to 1D array\n",
        "x = tf.keras.layers.Dense(128, activation='relu')(x)   # first dense layer + relu\n",
        "x = tf.keras.layers.Dense(32, activation='relu')(x)\n",
        "x = tf.keras.layers.Dense(nclasses)(x)                 # latest dense, no activation\n",
        "x = tf.keras.layers.Softmax()(x)                       # normalize the output\n",
        "\n",
        "model = tf.keras.Model(inputs=inputs, outputs=x)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nyYW0EZ8Ux3Y"
      },
      "source": [
        "### Inspect the model\n",
        "Note that the each output has an additional dimension with length `None`. This is the batch dimension, since we will feed the model with a batch of images. This very simple model has ~200k parameters to be optimized."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hpGPGpHLUvi7"
      },
      "outputs": [],
      "source": [
        "model.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fMdUnV-1fshN"
      },
      "outputs": [],
      "source": [
        "from keras.utils.vis_utils import plot_model\n",
        "plot_model(model, show_shapes=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5XZbrWcyV8qt"
      },
      "source": [
        "### Apply the model to one image\n",
        "Note that we need to expand the dimensions of the image. Also the output has one additional dimension."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zVMMGDCcVtkl"
      },
      "outputs": [],
      "source": [
        "example_image = train_images[0]\n",
        "print(\"image shape: \", example_image.shape)\n",
        "example_image = np.expand_dims(example_image, axis=0)\n",
        "print(\"image shape: \", example_image.shape)\n",
        "model(example_image)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oWGyc5SOXAAA"
      },
      "source": [
        "### Compile the model\n",
        "Add a loss and the optimizer. In TF these components are inside the same computational graph of the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PEGGXROmFuJG"
      },
      "outputs": [],
      "source": [
        "model.compile(optimizer='adam',\n",
        "              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "              metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ys27SnwLX6Ng"
      },
      "source": [
        "## Train\n",
        "Train for 10 epochs (the number of times the dataset is read). Use 1/3 of the train sample to check the loss and the metrics during the training. In particular use the validation sample for the early stopping."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_ASQTUerFuJI"
      },
      "outputs": [],
      "source": [
        "history = model.fit(train_images, train_labels,\n",
        "                    batch_size=256,\n",
        "                    epochs=30, validation_split=0.33,\n",
        "                    callbacks=[tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3)])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8d3sF0hgYUTZ"
      },
      "source": [
        "### Display the metric and the loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "7vWtVkd6FuJJ"
      },
      "outputs": [],
      "source": [
        "#@title\n",
        "fig, axs = plt.subplots(1, 2, figsize=(16, 5), sharex=True)\n",
        "for ax, quantity in zip(axs, ('accuracy', 'loss')):\n",
        "    ax.plot(history.history[quantity], label='train')\n",
        "    ax.plot(history.history[f'val_{quantity}'], label='validation')\n",
        "    ax.legend()\n",
        "    ax.set_xlabel('epoch', fontsize=15)\n",
        "    ax.set_ylabel(quantity, fontsize=15)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BHlI__liaZmG"
      },
      "source": [
        "## Run on the test sample"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "n8VACKXuFuJJ"
      },
      "outputs": [],
      "source": [
        "predictions = model.predict(test_images)\n",
        "predictions[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "t-Vc6l7VFuJO"
      },
      "outputs": [],
      "source": [
        "fig, axs = plt.subplots(5, 5, figsize=(7,7))\n",
        "for ax, img, prediction, label in zip(axs.flat, test_images, predictions, test_labels):\n",
        "    ax.set_xticks([])\n",
        "    ax.set_yticks([])\n",
        "    ax.imshow(img, cmap='binary')\n",
        "    predicted_class_id = np.argmax(prediction)\n",
        "    ax.set_title(\"%s (%.0f%%)\\n truth:%s\" % (class_names[predicted_class_id], prediction[predicted_class_id] * 100, class_names[label]),\n",
        "                 color='black' if predicted_class_id == label else \"red\")\n",
        "    \n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AUz-NTpRa46R"
      },
      "source": [
        "## Analyze the performances\n",
        "As first step compute the confusion matrix, defined as the frequencies for each true label and classified label:\n",
        "\n",
        "Confusion matrix = $N[\\text{true-label}, \\text{reco-label}]$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "nKS203ldFuJO"
      },
      "outputs": [],
      "source": [
        "confusion_matrix = tf.math.confusion_matrix(\n",
        "    test_labels,\n",
        "    np.argmax(predictions, axis=1),\n",
        "    num_classes=10,\n",
        ").numpy()\n",
        "\n",
        "with plt.style.context('seaborn-talk'):\n",
        "    fig, ax = plt.subplots(figsize=(10, 10))\n",
        "    sns.heatmap(confusion_matrix, xticklabels=class_names, annot=True, yticklabels=class_names, ax=ax, square=True, linewidths=0.1, fmt='d', cbar=False)\n",
        "    ax.set_xlabel('Prediction', fontsize=15)\n",
        "    ax.set_ylabel('Truth', fontsize=15)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mwa6qfpgec86"
      },
      "source": [
        "Compute the purity and the efficiency:\n",
        "\n",
        "efficiecy = $P[\\text{prediction}|\\text{truth}]=N[\\text{truth},\\text{prediction}] / N[\\text{truth}]$\n",
        "\n",
        "purity = $P[\\text{truth}|\\text{prediction}]=N[\\text{truth},\\text{prediction}] / N[\\text{prediction}]$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "2ZIVn2rSFuJP"
      },
      "outputs": [],
      "source": [
        "efficiency = confusion_matrix / np.sum(confusion_matrix, axis=1)  # divide by the truth\n",
        "purity = (confusion_matrix.T / np.sum(confusion_matrix, axis=0)).T  # divide by the reco\n",
        "\n",
        "with plt.style.context('seaborn-talk'):\n",
        "    fig, axs = plt.subplots(1, 2, figsize=(15, 7))\n",
        "    sns.heatmap(efficiency * 100, xticklabels=class_names, yticklabels=class_names, ax=axs[0], square=True, linewidths=0.1, annot=True, cmap='Blues', cbar=False)\n",
        "    sns.heatmap(purity * 100, xticklabels=class_names, yticklabels=class_names, ax=axs[1], square=True, linewidths=0.1, annot=True, cmap='Reds', cbar=False, vmin=0, vmax=100)\n",
        "\n",
        "    for ax in axs:\n",
        "        ax.set_xlabel('Prediction')\n",
        "        ax.set_ylabel('Truth')\n",
        "    axs[0].set_title('efficiecy = P[prediction|truth]', fontsize=15)\n",
        "    axs[1].set_title('purity = P[truth|prediction]', fontsize=15)\n",
        "    fig.subplots_adjust(wspace=0.5)\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z1VJQEhRfUXg"
      },
      "source": [
        "In machine learning you may encounter several new term, but they are basically the purity and the efficiency"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "j0E0MpfJFuJQ"
      },
      "outputs": [],
      "source": [
        "#Get the predictions for the test data\n",
        "predicted_classes = np.argmax(predictions, axis=1)\n",
        "correct = np.nonzero(predicted_classes==test_labels)[0]\n",
        "incorrect = np.nonzero(predicted_classes!=test_labels)[0]\n",
        "from sklearn.metrics import classification_report\n",
        "print(classification_report(test_labels, predicted_classes, target_names=class_names))"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "1.0-ImageClassification.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "venv3",
      "language": "python",
      "name": "venv3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}