01_Basics_NeuralNetworks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right\"><sub>This notebook is distributed under the <a href=\"https://creativecommons.org/licenses/by-sa/4.0/\" target=\"_blank\">Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license</a>.</sub></div>\n",
    "<h1>Hands on Machine Learning  <span style=\"font-size:10px;\"><i>by <a href=\"https://webgrec.ub.edu/webpages/000004/ang/dmaluenda.ub.edu.html\" target=\"_blank\">David Maluenda</a></i></span></h1>\n",
    "\n",
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td>\n",
    "    <a href=\"https://atenea.upc.edu/course/view.php?id=85709\" target=\"_blank\">\n",
    "      <img src=\"https://github.com/dmaluenda/hands_on_machine_learning/raw/master/resources/upc_logo_49px.png\" width=\"130\"/>\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "  </td>\n",
    "  <td>   <!-- gColab -->\n",
    "    <a href=\"https://colab.research.google.com/github/dmaluenda/hands_on_machine_learning/blob/master/01_Basics_NeuralNetworks.ipynb\" target=\"_blank\">\n",
    "      <img src=\"https://github.com/dmaluenda/hands_on_machine_learning/raw/master/resources/colab_logo_32px.png\" />\n",
    "      Run in Google Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>   <!-- github -->\n",
    "    <a href=\"https://github.com/dmaluenda/hands_on_machine_learning/blob/master/01_Basics_NeuralNetworks.ipynb\" target=\"_blank\">\n",
    "      <img src=\"https://github.com/dmaluenda/hands_on_machine_learning/raw/master/resources/github_logo_32px.png\" />\n",
    "      View source on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>   <!-- download -->\n",
    "    <a href=\"https://raw.githubusercontent.com/dmaluenda/hands_on_machine_learning/master/01_Basics_NeuralNetworks.ipynb\"  target=\"_blank\" download=\"01_Basics_NeuralNetworks\">\n",
    "      <img src=\"https://github.com/dmaluenda/hands_on_machine_learning/raw/master/resources/download_logo_32px.png\" />\n",
    "      Download notebook\n",
    "      </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# $\\text{I}$. Neural Networks with Pure Python\n",
    "\n",
    "Hands on \"Machine Learning on Classical and Quantum data\" course of\n",
    "[Master in Photonics - PHOTONICS BCN](https://photonics.masters.upc.edu/en/general-information)\n",
    "[[UPC](https://photonics.masters.upc.edu/en) +\n",
    "[UB](https://www.ub.edu/web/ub/en/estudis/oferta_formativa/master_universitari/fitxa/P/M0D0H/index.html?) +\n",
    "[UAB](https://www.uab.cat/web/estudiar/la-oferta-de-masteres-oficiales/informacion-general-1096480309770.html?param1=1096482863713) +\n",
    "[ICFO](https://www.icfo.eu/lang/studies/master-studies)].\n",
    "\n",
    "Tutorial 1\n",
    "\n",
    "This notebook shows how to:\n",
    "- implement the forward-pass (prediction, inference or evaluation) of a fully connected neural network in a few lines of pure python\n",
    "- understand the activation functions meanings and usages\n",
    "- do that efficiently using batches\n",
    "- illustrate the results for randomly initialized neural networks\n",
    "- understand the role of weights and biases in networks\n",
    "\n",
    "\n",
    "**References**:\n",
    "\n",
    "[1] [Machine Learning for Physicists](https://machine-learning-for-physicists.org/) by Florian Marquardt.<br>\n",
    "[2] [NumPy](https://numpy.org/doc/stable/user/whatisnumpy.html): the fundamental package for scientific computing in Python.<br>\n",
    "[3] [Matplotlib](https://matplotlib.org/stable/tutorials/introductory/usage.html): a comprehensive library for creating static, animated, and interactive visualizations in Python.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#0.-Imports:-only-numpy-and-matplotlib\" data-toc-modified-id=\"0.-Imports:-only-numpy-and-matplotlib-0\">0. Imports: only numpy and matplotlib</a></span></li><li><span><a href=\"#1.-A-very-simple-neural-network-(no-hidden-layer)\" data-toc-modified-id=\"1.-A-very-simple-neural-network-(no-hidden-layer)-1\">1. A very simple neural network (no hidden layer)</a></span><ul class=\"toc-item\"><li><span><a href=\"#1.1-Simple-implementation-(single-input-data-processing)\" data-toc-modified-id=\"1.1-Simple-implementation-(single-input-data-processing)-1.1\">1.1 Simple implementation (single input data processing)</a></span></li><li><span><a href=\"#1.2-More-compact-(working-with-functions)\" data-toc-modified-id=\"1.2-More-compact-(working-with-functions)-1.2\">1.2 More compact (working with functions)</a></span></li><li><span><a href=\"#1.3-Multiple-inputs\" data-toc-modified-id=\"1.3-Multiple-inputs-1.3\">1.3 Multiple inputs</a></span></li><li><span><a href=\"#1.4-Activation-functions\" data-toc-modified-id=\"1.4-Activation-functions-1.4\">1.4 Activation functions</a></span></li></ul></li><li><span><a href=\"#2.-Network-with-one-hidden-layer\" data-toc-modified-id=\"2.-Network-with-one-hidden-layer-2\">2. Network with one hidden layer</a></span></li><li><span><a href=\"#3.-'batch'-processing-of-Neural-Networks\" data-toc-modified-id=\"3.-'batch'-processing-of-Neural-Networks-3\">3. 'batch' processing of Neural Networks</a></span><ul class=\"toc-item\"><li><span><a href=\"#3.1-Matrix/Vector/Tensor-multiplication\" data-toc-modified-id=\"3.1-Matrix/Vector/Tensor-multiplication-3.1\">3.1 Matrix/Vector/Tensor multiplication</a></span></li><li><span><a href=\"#3.2-Batch-implementation-of-one-hidden-layer-network\" data-toc-modified-id=\"3.2-Batch-implementation-of-one-hidden-layer-network-3.2\">3.2 Batch implementation of one hidden layer network</a></span></li><li><span><a href=\"#3.3-Data-preprocessing-for-batch-implementation\" data-toc-modified-id=\"3.3-Data-preprocessing-for-batch-implementation-3.3\">3.3 Data preprocessing for batch implementation</a></span></li><li><span><a href=\"#3.4-A-network-with-MANY-hidden-layers\" data-toc-modified-id=\"3.4-A-network-with-MANY-hidden-layers-3.4\">3.4 A network with MANY hidden layers</a></span></li></ul></li><li><span><a href=\"#4.-Fancy-visualization-of-Neural-Networks-with-Pure-Python\" data-toc-modified-id=\"4.-Fancy-visualization-of-Neural-Networks-with-Pure-Python-4\">4. Fancy visualization of Neural Networks with Pure Python</a></span><ul class=\"toc-item\"><li><span><a href=\"#4.1-Some-internal-routines-for-fancy-plotting-the-network\" data-toc-modified-id=\"4.1-Some-internal-routines-for-fancy-plotting-the-network-4.1\">4.1 Some internal routines for fancy plotting the network</a></span></li><li><span><a href=\"#5.2-No-hidden-layer-NN\" data-toc-modified-id=\"5.2-No-hidden-layer-NN-4.2\">5.2 No hidden layer NN</a></span></li><li><span><a href=\"#5.3-Deep-dense-NN\" data-toc-modified-id=\"5.3-Deep-dense-NN-4.3\">5.3 Deep dense NN</a></span><ul class=\"toc-item\"><li><span><a href=\"#More-layers-and-some-activation-function-combination\" data-toc-modified-id=\"More-layers-and-some-activation-function-combination-4.3.1\">More layers and some activation function combination</a></span></li></ul></li><li><span><a href=\"#5.4-Something-not-random\" data-toc-modified-id=\"5.4-Something-not-random-4.4\">5.4 Something not random</a></span></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KlT_ZrS-R1rG"
   },
   "source": [
    "## 0. Imports: only numpy and matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 215,
     "status": "ok",
     "timestamp": 1626874792493,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "zRmzXX4wR1rG"
   },
   "outputs": [],
   "source": [
    "# \"numpy\" library for linear algebra\n",
    "import numpy as np\n",
    "\n",
    "# \"matplotlib\" for plotting\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib\n",
    "matplotlib.rcParams['figure.dpi'] = 300  # highres display\n",
    "from matplotlib.axes._axes import _log as mpl_ax_logger\n",
    "mpl_ax_logger.setLevel('ERROR')  # ignore warnings\n",
    "from mpl_toolkits.axes_grid1.inset_locator import inset_axes  # for nice inset\n",
    "\n",
    "# time control to count it and manage it\n",
    "from time import time\n",
    "\n",
    "# just to play like in a GUI\n",
    "from ipywidgets import interact, Text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "T98H_rSNR1rH"
   },
   "source": [
    "## 1. A very simple neural network (no hidden layer)\n",
    "\n",
    "The behavior of a neural network (NN) with $N_0$ input neurons and $N_1$ output neurons (no hidden layer) is\n",
    "\n",
    "\\begin{equation}\n",
    "z_i = \\sum_j^{N_0} w_{ij} x_j + b_i \\quad ; \\quad i=1\\dots N_1\n",
    "\\label{eq:simpleNN}\n",
    "\\end{equation}\n",
    "\n",
    "\\begin{equation}\n",
    "y_i = f(z_i)\n",
    "\\label{eq:actFunction}\n",
    "\\end{equation}\n",
    "\n",
    "where $x_j$ is the input value of the $j$-th input neuron,\n",
    "$y_i$ is the output value of the $i$-th output neuron,\n",
    "$w_{ij}$ is the weight of the connection between the $j$-th input neuron with the $i$-th output neuron,\n",
    "$b_i$ is the bias of the $i$-th output neuron,\n",
    "$z_i$ is the linear output of the layer (linear superposition between inputs and weights and biases), \n",
    "and $f(·)$ is the activation function (usually it is non-linear: for instance a sigmoid function).\n",
    "\n",
    "Notice that we can define a matrix $w$ of size $N_1\\times N_0$ (rows $\\times$ columns),\n",
    "which contains all $w_{ij}$ connection weights.\n",
    "In the same way, we can condensate all $b_j$ biases and $x_j$ input neurons in $b$ and $x$ column vectors of size $N_1$.\n",
    "Thus, Eq. (1) can be seen as a simple matrix multiplication and a vector sum."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Simple implementation (single input data processing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement a basic neural network of 3 input and 2 output neurons. Use the sigmoid function $$f_{sig}(z)=\\frac{1}{1+e^{-z}}$$ as activation function.\n",
    "\n",
    "Use random values for weights and biases, in such a way they fall in the range $[-3, 3]$.\n",
    "\n",
    "> You can use [`np.random.uniform(low, high, size)`](https://numpy.org/doc/stable/reference/random/generated/numpy.random.uniform.html) to generate the random numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate that neural network (defined by that weights, biases and activation function) for the input vector $x=(0.5, 0.3, 0.2)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the input vector $x$, the weights $w$, the biases $b$, the linear superposition $z$ and the final result $y$, as well as, their shapes (sizes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 226,
     "status": "ok",
     "timestamp": 1626874836630,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "KTpiYEIFR1rJ",
    "outputId": "d13d700c-1365-4dbf-d2e1-29aa014625ee",
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the shape of every array. Do they make sense?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xMfY7VTBR1rK"
   },
   "source": [
    "### 1.2 More compact (working with functions)\n",
    "\n",
    "It is very convenient to work with specific functions, instead of code snippers (small parts of the code that typically are copy/pasted everywhere).\n",
    "\n",
    "Let's still stay with the simplest network, thus you can use the code above, but define a function to evaluate the network the $y$ output vector (return) for a given (arguments) $x$ input vector, $w$ weights matrix, and $b$ biases vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 220,
     "status": "ok",
     "timestamp": 1626874872832,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "8ahzBFksR1rL"
   },
   "outputs": [],
   "source": [
    "def my_function(some_arguments):  # change the name and the arguments of this function\n",
    "    \n",
    "    # my code (same than a couple of cells above)\n",
    "    \n",
    "    return # return the result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's play with a different number of neurons. We want to visualize the behavior of having different weights and biases on a neural network.\n",
    "A good approach to that is working with 2 input and 1 output neurons.\n",
    "\n",
    "> **NOTE**: Neural Networks with 2 input neurons and 1 output neuron are ideal to illustrate how it\n",
    "works, because it allows us to visualize its behavior in a single picture. Let's see.\n",
    "\n",
    "Thus, define a weights and biases matrices to deal with 2 inputs and 1 output neural network, where the weights cover the range $[-20, 20]$, whereas biases cover the range $[-1, 1]$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 224,
     "status": "ok",
     "timestamp": 1626874886437,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "tzGCtU9uR1rL"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uses the function above to get the result for the input $x=(-0.2, 0.1)$ and print the result. Check the range and the size (shape) of the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 238,
     "status": "ok",
     "timestamp": 1626874895575,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "CsE8xEuPR1rL",
    "outputId": "e3396801-1459-4a7e-b070-ff414f33aec9"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Multiple inputs\n",
    "\n",
    "Let's see how the network acts with many different inputs $x$.\n",
    "\n",
    "Explore pairs of $x=(x_1, x_2)$ values from -0.5 to 0.5, having $m=300$ different values each, and apply the network to every pair of $x$ input values.\n",
    "\n",
    "The result have to be a $300\\times300$ array. Plot it with [`plt.imshow()`](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.imshow.html) and properly set the axis labels to cover the range $[-0.5, 0.5]$. Also, add a color bar to check the output range.\n",
    "\n",
    "> Initialize an array with the [`np.zeros(size)`](https://numpy.org/doc/stable/reference/generated/numpy.zeros.html) function and, then, you can use a couple of `for` loops to fill this numpy array ([indexing and slicing](https://numpy.org/doc/stable/user/basics.indexing.html))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 224,
     "status": "ok",
     "timestamp": 1626874936164,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "kFEaqMXaR1rM"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What represent this image? What represent the axis of that image and what represent the colors.\n",
    "\n",
    "Which kind of image is this? Does it has any specific orientation? Recall the sigmoid function.\n",
    "\n",
    "Print the weights and biases. Could you relate this values with the image?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 Activation functions\n",
    "\n",
    "Activation functions are a key point on neural networks, since they introduce nonlinearity on the computations. There are many types of activation functions, where some work very well for some problems and others for other problems kinds.\n",
    "\n",
    "Some info about activation functions:\n",
    "\n",
    "[https://en.wikipedia.org/wiki/Activation_function](https://en.wikipedia.org/wiki/Activation_function)\n",
    "\n",
    "[https://www.analyticsvidhya.com/blog/2021/04/activation-functions-and-their-derivatives-a-quick-complete-guide](https://www.analyticsvidhya.com/blog/2021/04/activation-functions-and-their-derivatives-a-quick-complete-guide)\n",
    "\n",
    "Make a new function to implement a neural network layer, like before, but using another activation function and check its behavior (basically, repeat the last work)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the differences between this resulting image and the one before. Orientation? Output range?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "adIF2dA2R1rM"
   },
   "source": [
    "## 2. Network with one hidden layer\n",
    "\n",
    "Let's increase the complexity of the neural network by adding a hidden layer with some inner neurons.\n",
    "\n",
    "The idea here is to have multiple weight matrices to connect one layer with the next (this is one weight matrix for each pair of subsequent layers).\n",
    "\n",
    "The function that \"applies a layer\" (i.e. goes from one layer to the next) is exactly the same as the function evaluating the simple network made before.\n",
    "\n",
    "Let's work with a three layers network, thus with one hidden layer in addition to the input and the output layers. Since we want to visualize the resulting application in an image like before, we will still dealing with 2 input and 1 output neurons. However, let's set **30 hidden neurons** in the inner layer.\n",
    "\n",
    "Generate the weights matrices and the biases vectors with random numbers, like before. Which size (shape) should they have?\n",
    "\n",
    "To easily visualize the behavior of the hidden layer, make the range of the first weights matrix larger than the rest of the numbers. Let's say $[-10, 10]$ for the weights connections from input to hidden layer, whereas leaving $[-1, 1]$ for the rest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 216,
     "status": "ok",
     "timestamp": 1626874970040,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "v5JXwyNNR1rN"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, now, to apply the whole neural network, we have to apply every layer, step by step.\n",
    "\n",
    "Thus, create a function to return the $y$ final result for a given (arguments) $x$ input vector, $w^{l=1}$ weights matrix from input to hidden layers, $b^{l=1}$ biases vector of the hidden layer, $w^{l=2}$ weights matrix from hidden to output layers, and $b^{l=2}$ biases vector of the output layer.\n",
    "\n",
    "Inside this function, you should use a couple of times the single layer function made before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 235,
     "status": "ok",
     "timestamp": 1626874973852,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "Cz1yN_CPR1rO"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the function you have just done above by applying the input vector $x=(0.3, -0.2)$, and print the result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, let's apply the network to a range of input pairs, like before.\n",
    "\n",
    "Generate an image showing the result of the network when applied to pairs of inputs from -0.5 to 0.5, using 300 samples in each direction, like before.\n",
    "\n",
    "Check the elapsed time to do fill the $y$ output array (the resulting image)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 218,
     "status": "ok",
     "timestamp": 1626874983951,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "6Tdm8QPIR1rO"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How long it took? What changed? The loop size or what is done in every iteration?\n",
    "\n",
    "This is just for one single hidden layer. Imagine what happen with larger networks. We will see how to deal with this time issue in the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the generated image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 790
    },
    "executionInfo": {
     "elapsed": 764,
     "status": "ok",
     "timestamp": 1626875087549,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "R40sBA-3R1rO",
    "outputId": "591f9ce8-c86d-4f6c-a798-5e4ef433788b"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MZCUJY6-R1rP"
   },
   "source": [
    "What differences are between this image and those produced by the no-hidden-layer network? Could you identify any orientation on it?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "s7oWq_SPR1rP"
   },
   "source": [
    "## 3. 'batch' processing of Neural Networks\n",
    "\n",
    "As we said before, the code above takes quite long, while the network is so small, i.e. it is not efficient.\n",
    "The reason is because we are looping over all the input values, and doing a matrix calculation for each iteration. This is not the most efficient way to operate.\n",
    "\n",
    "We will see how to do it avoiding looping, but using *batch* processing.\n",
    "\n",
    "**Goal: apply a network to many samples in parallel taking advantage of array computations.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HhfXVsoxR1rP"
   },
   "source": [
    "### 3.1 Matrix/Vector/Tensor multiplication\n",
    "\n",
    "For instance, create a no-hidden-layer network of $N_0=8$ input neurons, and $N_1=3$ output neurons.\n",
    "\n",
    "Let's try to process $M=50$ different inputs at once. Notice, **$M$ is usually called _batchsize_**.\n",
    "\n",
    "The idea is to have an extra dimension for the $x$ to be able to hold the different vector inputs.\n",
    "\n",
    "Thus,\n",
    "\n",
    " 1. Create an input array with 2 dimensions: one for every input neuron and one for every different input.\n",
    "\n",
    " 2. Create a weights and biases arrays. Should they have an extra dimension to hold that several inputs, as well?\n",
    "\n",
    " 3. Make the matrix multiplication in such a way to obtain the result prior to activation function, i.e. $z$. Which size (shape) should $z$ have?\n",
    " \n",
    "> Play with the dimension orders and multiplication order to get a valid matrix multiplication."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 2777,
     "status": "ok",
     "timestamp": 1626875178917,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "bojqaDqrR1rP",
    "outputId": "64950c99-8a19-424d-a447-f6f5c1a4b44f"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What represent each dimension of the $x$ array?\n",
    "\n",
    "What represent each dimension of the $z$ array?\n",
    "\n",
    "Do weights and bias depend on the $M$ *batchsize*?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vvu0gA4nR1rQ"
   },
   "source": [
    "### 3.2 Batch implementation of one hidden layer network\n",
    "\n",
    "Let's define the same network than before, with one hidden layer of 30 neurons, and 2-input/1-output neurons.\n",
    "\n",
    "Let's try to use the same weights and biases than before. How they have to be modified?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up a function to go from one layer to the next in such a way it is able to manage batch processing, according to what learned above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 231,
     "status": "ok",
     "timestamp": 1626875241146,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "WDnsidUPR1rR"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a function to apply a whole network (two layers in this case).\n",
    "\n",
    "Consider to use an argument for the $x$ input and just two additional arguments: one for the weights and the other for the biases.\n",
    "\n",
    "Take into account that for two layers network, we deal with two weights matrices and two biases vectors. However, for 10 layers network you would need many arguments (21). Moreover, you do not know how many layers a general network has. Then, consider to take a list of numpy array in the arguments corresponding to $[\\omega]$ weights and $[b]$ biases. Then, iterate over that lists inside the function. In this way, you may use that function for any network having an unknown number of layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 230,
     "status": "ok",
     "timestamp": 1626875244101,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "ee2uQFPYR1rR"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the implementation with a batchsize $M=m\\times m = 300 \\times 300 = 90000$, just to compare with the previous approach (looping).\n",
    "\n",
    "Thus, apply your network to an input random array. Which shape must this input have? And the result?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jt56KtwSR1rS"
   },
   "source": [
    "### 3.3 Data preprocessing for batch implementation\n",
    "\n",
    "Before, we had a two nested loops running all possible values for each input neuron.\n",
    "This was inefficient, but easy to understand how to calculate every combination of input values.\n",
    "Now, we have to think a way to put all that possibilities in a batch array to calculate them at once.\n",
    "How to do that?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `np.linspace()` and `np.meshgrid()` functions to create a couple of 2D arrays of $300\\times300$ holding values in range $[-0.5, 0.5]$, one vertically and the other horizontally. Check the documentation how that functions work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 223,
     "status": "ok",
     "timestamp": 1626875258337,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "J48VP4KBR1rS"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that one array increases with the horizontal direction and remains constant on the\n",
    "vertical, like the $X$ coordinate. While the other runs just on the opposite,\n",
    "like $Y$ coordinate. Then, we can use them just like simple Cartesian coordinates.\n",
    "\n",
    "Be careful, do not get confused between $X$-$Y$ coordinates of the space, with the $x$-$y$ input-output vectors of the network, defined before.\n",
    "\n",
    "Ok. Now we have two separated 2D-arrays ($300\\times300$, each), while the network expects an input array of `batchsize`$=M=90000$ for two input neurons.\n",
    "\n",
    "Then, use `np.flatten()` and `np.stack()` functions to transform two $300\\times300$ arrays to a single $90000\\times2$ array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 261,
     "status": "ok",
     "timestamp": 1626875272772,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "odaw2KOMR1rT",
    "outputId": "77d7659e-f0d4-4317-dafe-8496bbf522ba"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply that stacked array to the network.\n",
    "\n",
    "Check the elapsed time to do it and compare it with the two nested loops implementation done before.\n",
    "\n",
    "Check the shape of the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 216,
     "status": "ok",
     "timestamp": 1626875276327,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "8eRZpeBMR1rT"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What represent each dimension and component of this output vector.\n",
    "\n",
    "We want to visualize this output in an image, like before. However, we have a flattened array.\n",
    "\n",
    "Let's go back from a flattened array to a 2D array: $(M\\times 1) \\rightarrow (m\\times m)$ and `imshow` this image.\n",
    "\n",
    "> Check the `np.reshape()` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 790
    },
    "executionInfo": {
     "elapsed": 272,
     "status": "ok",
     "timestamp": 1626875281319,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "pM1hFhirR1rU",
    "outputId": "47d3a65a-63d3-4057-d079-a7153205f3a0"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is this image looking like before? Why? Which advantage have this approach with respect the two nested loops?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pLUulorJR1rU"
   },
   "source": [
    "### 3.4 A network with MANY hidden layers\n",
    "\n",
    "We can create a $[\\omega]$ list of weights containing many weight between many layers.\n",
    "Also, we can do the same for the $[b]$ biases.\n",
    "\n",
    "Thus, create a big neural network having, let's say, 20 hidden layers, with some number of neurons each in the range $[20, 40]$. You can randomly set the specific number of neurons for each layer.\n",
    "\n",
    "Set the weights matrices randomly in the range $[-5, 5]$ and the biases vectors in $[-1, 1]$.\n",
    "\n",
    "Let's set the input layer with 2 neurons and 1 single output neuron, to be able to visualize the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 222,
     "status": "ok",
     "timestamp": 1626875292158,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "W2_okOLTR1rU"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run this big neural network to the flatten and stacked input done before to cover $x=(x_1, x_2)$ input pairs in the range $[-0.5, 0.5]$, and `imshow` the result. Again, check the time consumed to apply this big network. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 246,
     "status": "ok",
     "timestamp": 1626875298627,
     "user": {
      "displayName": "David Maluenda",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhWqrU3JdB-d8lhWcsydkhDTsyMGlOXfplagiOHaw=s64",
      "userId": "14097055138958710373"
     },
     "user_tz": -120
    },
    "id": "kaPZGeWNR1rV"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comment the image obtained."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Fancy visualization of Neural Networks with Pure Python\n",
    "\n",
    "In this section you do not need to write code, just play with it to visualize different networks and activation functions to get familiar with the behavior of different hyper-parameters (number of layers, number of neurons, activation functions...) and to earn intuition.\n",
    "\n",
    "### 4.1 Some internal routines for fancy plotting the network\n",
    "\n",
    "This cell below contains code to show networks in tree plots where each branch color is proportional to its corresponding weight and the neuron color to its bias value. It is done in pure python/matplotlib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BLUE_COLOR = [0, 0.4, 0.8]  # RGB color for the full-range negative value\n",
    "ORANGE_COLOR = [1, 0.3, 0]  # RGB color for the full-range positive value\n",
    "\n",
    "def plot_connection_line(ax, X, Y, W, vmax=10, linewidth=3):\n",
    "    \"\"\" Draw a fancy line from (X[0], Y[0]) to (X[1], Y[1])\n",
    "        according to the weight W into the frame ax.\n",
    "    \"\"\"\n",
    "    t = np.linspace(0,1,20)  # free parameter to draw lines\n",
    "    \n",
    "    if W > 0:  # Color depending on the weight's sign\n",
    "        col = ORANGE_COLOR\n",
    "    else:\n",
    "        col = BLUE_COLOR\n",
    "    \n",
    "    # fancy line from (X0, Y0) to (X1, Y1)\n",
    "    xx = X[0] + t*(X[1] - X[0])  # Linear in horizontal\n",
    "    yy = Y[0] + (3*t**2 - 2*t**3) * (Y[1] - Y[0])  # Round borders\n",
    "    \n",
    "    # plotting the line according to the weight\n",
    "    ax.plot(xx, yy, alpha=min(1, abs(W)/vmax),\n",
    "            color=col, linewidth=linewidth)\n",
    "\n",
    "    \n",
    "def plot_neuron_alpha(ax, X, Y, B, size=100.0, vmax=10):\n",
    "    \"\"\" Draw a single neuron in position (X, Y) according to \n",
    "        the bias B, into the frame ax.\n",
    "    \"\"\"\n",
    "    if B > 0:\n",
    "        col = ORANGE_COLOR\n",
    "    else:\n",
    "        col = BLUE_COLOR\n",
    "        \n",
    "    ax.scatter([X], [Y], marker='o', color=col, alpha=min(1, abs(B)/vmax), \n",
    "               s=size, zorder=10)\n",
    "\n",
    "    \n",
    "def plot_neuron(ax, X, Y, B, size=100.0, vmax=10):\n",
    "    \"\"\" Draw a single neuron in position (X, Y) independently to \n",
    "        the bias B, into the frame ax.\n",
    "    \"\"\"\n",
    "    if B > 0:\n",
    "        col = ORANGE_COLOR\n",
    "    else:\n",
    "        col = BLUE_COLOR\n",
    "        \n",
    "    ax.scatter([X], [Y], marker='o', color=col, s=size, zorder=10)\n",
    "    \n",
    "    \n",
    "def visualize_network(weights, biases, activations, M=400,\n",
    "                      x0range=[-3,3], x1range=[-3,3],\n",
    "                      size=400.0, linewidth=5.0, maxv=1.):\n",
    "    \"\"\"\n",
    "    Visualize a neural network with 2 input \n",
    "    neurons and 1 output neuron (plot output vs input in a 2D plot)\n",
    "    \n",
    "    weights is a list of the weight matrices for the\n",
    "    layers, where weights[j] is the matrix for the connections\n",
    "    from layer j to layer j+1 (where j==0 is the input)\n",
    "    \n",
    "    weights[j][m,k] is the weight for input neuron k going to output neuron m\n",
    "    (note: internally, m and k are swapped, see the explanation of\n",
    "    batch processing in lecture 2)\n",
    "    \n",
    "    biases[j] is the vector of bias values for obtaining the neurons \n",
    "    in layer j+1, biases[j][k] is the bias for neuron k in layer j+1\n",
    "    \n",
    "    activations is a list of the activation functions for\n",
    "    the different layers: choose 'linear','sigmoid', \n",
    "    'jump' (i.e. step-function), and 'reLU'\n",
    "    \n",
    "    M is the resolution (MxM grid)\n",
    "    \n",
    "    x0range is the range of y0 neuron values (horizontal axis)\n",
    "    x1range is the range of y1 neuron values (vertical axis)\n",
    "    \"\"\"\n",
    "    \n",
    "    if type(weights) == str:\n",
    "        weights = eval(weights)\n",
    "    if type(biases) == str:\n",
    "        biases = eval(biases)\n",
    "\n",
    "    # Let's transpose the weight to be able the batch processing\n",
    "    swapped_weights = []\n",
    "    for j in range(len(weights)):\n",
    "        swapped_weights.append(np.transpose(weights[j]))\n",
    "        \n",
    "    # Let's create a set of input-pairs by means of a mesh grid\n",
    "    x0, x1 = np.meshgrid(np.linspace(x0range[0], x0range[1], M),\n",
    "                         np.linspace(x1range[0], x1range[1], M))\n",
    "    x = np.zeros([M*M, 2])\n",
    "    x[:, 0] = x0.flatten()\n",
    "    x[:, 1] = x1.flatten()\n",
    "    \n",
    "    # Let's apply the NN\n",
    "    y_out = apply_net(x, swapped_weights, biases, activations)\n",
    "\n",
    "    # We will plot a diagram at left and the result at right\n",
    "    fig, ax = plt.subplots(ncols=2, nrows=1, figsize=(8,4))\n",
    "    \n",
    "    \n",
    "    # For the diagram\n",
    "    \n",
    "    #  1: posX and posY are arrays containing the positions of neurons\n",
    "    posX = [[0, 0]]  # same column for both (at left)    \n",
    "    posY = [[-0.5, +0.5]]  # for 2 inputs, let's putted centered in high\n",
    "\n",
    "    \n",
    "    vmax = 0.0 # for finding the maximum weight\n",
    "    vmaxB = 0.0 # for maximum bias\n",
    "    for j in range(len(biases)):  # for every layer on the NN\n",
    "        n_neurons = len(biases[j])  # neurons in the current layer\n",
    "        \n",
    "        posX.append(np.full(n_neurons, j+1))  # next column to the previous one\n",
    "        posY.append(np.array(range(n_neurons)) - 0.5 * (n_neurons-1)) # spread\n",
    "        \n",
    "        vmax = maxv#np.maximum(vmax, np.max(np.abs(weights[j])))  # to get the maximum\n",
    "        vmaxB = maxv#np.maximum(vmaxB, np.max(np.abs(biases[j])))\n",
    "\n",
    "    #   2: plot connections\n",
    "    for j in range(len(biases)):  # for each layer\n",
    "        for k in range(len(posX[j])):  # for each neuron\n",
    "            for m in range(len(posX[j+1])):  # for each following neuron\n",
    "                plot_connection_line(ax[0],  # first column of the plot\n",
    "                                     [posX[j][k], posX[j+1][m]], # [X0,X1]\n",
    "                                     [posY[j][k], posY[j+1][m]], # [Y0,Y1]\n",
    "                                     swapped_weights[j][k,m],    # its weight\n",
    "                                     vmax=vmax,  # to get normalized plots\n",
    "                                     linewidth=linewidth)\n",
    "    \n",
    "    #   3: plot neurons\n",
    "    for k in range(len(posX[0])):  # input neurons (have no bias!)\n",
    "        plot_neuron(ax[0], posX[0][k], posY[0][k],\n",
    "                    vmaxB, vmax=vmaxB, size=size)\n",
    "        \n",
    "    for j in range(len(biases)): # all other neurons\n",
    "        for k in range(len(posX[j+1])):\n",
    "            plot_neuron_alpha(ax[0], posX[j+1][k], posY[j+1][k],\n",
    "                              biases[j][k], vmax=vmaxB, size=size)\n",
    "    \n",
    "    ax[0].axis('off')\n",
    "    \n",
    "    # now: the output of the network\n",
    "    img = ax[1].imshow(np.reshape(y_out, [M,M]),\n",
    "                       origin='lower',\n",
    "                       extent=[x0range[0],x0range[1],x1range[0],x1range[1]])\n",
    "    ax[1].set_xlabel('$x_1$')\n",
    "    ax[1].set_ylabel('$x_2$')\n",
    "    \n",
    "    axins1 = inset_axes(ax[1],\n",
    "                        width=\"40%\",  # width = 50% of parent_bbox width\n",
    "                        height=\"5%\",  # height : 5%\n",
    "                        loc='upper right')\n",
    "\n",
    "    imgmin = np.min(y_out)\n",
    "    imgmax = np.max(y_out)\n",
    "    color_bar = fig.colorbar(img, cax=axins1, orientation=\"horizontal\",\n",
    "                             ticks=np.linspace(imgmin,imgmax,3))\n",
    "    cbxtick_obj = plt.getp(color_bar.ax.axes, 'xticklabels')\n",
    "    plt.setp(cbxtick_obj, color=\"white\")\n",
    "    axins1.xaxis.set_ticks_position(\"bottom\")\n",
    "\n",
    "    ax[1].set_title(' , '.join(activations))\n",
    "    \n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the cell below we implement and example of a general network able to use different activation functions in each layer. Check it, but it should be very similar to your functions done above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def apply_layer(x, w, b, activation):\n",
    "    \"\"\" Batch processing of a single layer:\n",
    "           x: input values  -> shape: [batchsize, num_neurons_in]\n",
    "           w:    weight matrix -> shape: [n_neurons_in, n_neurons_out]\n",
    "           b:    bias vector   -> length: n_neurons_out\n",
    "    \n",
    "           activation is some string of the following ones:\n",
    "             - sigmoid\n",
    "             - jump\n",
    "             - linear\n",
    "             - reLU\n",
    "    \n",
    "           returns the values of the output neurons in the next layer \n",
    "              -> shape: [batchsize, n_neurons_out]\n",
    "    \"\"\"\n",
    "    z = np.dot(x, w) + b\n",
    "    if activation == 'sigmoid':\n",
    "        return 1 / (1 + np.exp(-z))\n",
    "    elif activation == 'jump':\n",
    "        return np.array(z>0, dtype='float')\n",
    "    elif activation == 'linear':\n",
    "        return z\n",
    "    elif activation == 'reLU':\n",
    "        return (z > 0) * z\n",
    "\n",
    "    \n",
    "def apply_net(x, weights, biases, activations):\n",
    "    \"\"\" Apply a whole network of multiple layers.\n",
    "          y_in: input values  -> shape: [batchsize, num_neurons_in]\n",
    "          weights, biases and activations must be any iterable \n",
    "          which length is the layers' number containing\n",
    "              weight matrix  -> shape: [n_neurons_in, n_neurons_out]\n",
    "              bias vector    -> length: n_neurons_out\n",
    "              activation str -> sigmoid, jump linear or reLU\n",
    "          Alternatively, they can be extended matrices\n",
    "          where a simple slicing generates the proper weight, \n",
    "          bias and activation.\n",
    "    \"\"\"\n",
    "    y = x.copy()\n",
    "    for j in range(len(biases)):\n",
    "        y = apply_layer(y, weights[j], biases[j], activations[j])\n",
    "    return y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 No hidden layer NN\n",
    "\n",
    "Let's visualize a simple network (no hidden layer) with different activation functions.\n",
    "\n",
    "Notice that no hidden layer means that weight, biases and activations are list of one single item.\n",
    "\n",
    "Play with different weights combinations to see its behavior. Do the same with the bias and the activation function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"You should see here three sliders and a text box to select the weights, \"\n",
    "      \"bias, and activation function.\\nIf you do not see them, try to restart \"\n",
    "      \"the Jupyter Notebook application.\")\n",
    "\n",
    "@interact(w1=(-10.,10.), w2=(-10.,10.), b=(-10.,10.), activation=[\"sigmoid\",\"jump\",\"linear\",\"reLU\"])\n",
    "def draw(w1=-3.4, w2=4.6, b=2.8, activation='sigmoid'):\n",
    "    weights=[ [      # a list of matrices (length 1 in this case)\n",
    "        [w1, w2]  # from 2 input neurons to a single output neuron: 1x2\n",
    "        ] ]\n",
    "\n",
    "    biases=[   # a list of vectors (length 1 in this case)\n",
    "        [b]  # bias for 1 single output neuron: 1 value\n",
    "        ]\n",
    "   \n",
    "    visualize_network(weights, biases, [activation], maxv=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How are the weights and bias related with the resulting image? And the activation function? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3 Deep dense NN\n",
    "\n",
    "Let's visualize a Neural Network of 1 hidden layer of 3 neurons using different activation functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@interact(activation_1=[\"sigmoid\",\"jump\",\"linear\",\"reLU\"],\n",
    "          activation_2=[\"sigmoid\",\"jump\",\"linear\",\"reLU\"])\n",
    "def draw(weights=\"[[[0.2, 0.9],[-0.5, 0.3],[0.8, -1]],[[-0.3,0.7,0.5]]]\",\n",
    "         biases=\"[[0.1, -0.5, -0.5],[-0.2]]\",\n",
    "         activation_1='jump', activation_2='sigmoid'):\n",
    "\n",
    "    visualize_network(weights, biases, [activation_1, activation_2], maxv=1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "Check the inclination of the lines having one hidden layer. How many lines are there when using a jump-sigmoid combination? And if the combination is sigmoid-jump? What about jump-jump? And sigmoid-sigmoid? How many lines in that combinations and how that lines are?\n",
    "\n",
    "What happens if we set the first activation function as linear?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### More layers and some activation function combination"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now more complicated, just for fun\n",
    "@interact(activation_1=[\"sigmoid\",\"jump\",\"linear\",\"reLU\"],\n",
    "          activation_2=[\"sigmoid\",\"jump\",\"linear\",\"reLU\"],\n",
    "          activation_3=[\"sigmoid\",\"jump\",\"linear\",\"reLU\"])\n",
    "def draw(w='[ [ [0.2, 0.9], [-0.5, 0.3], [0.8, -1.3],[-0.3, -0.9], [-0.8, -1.2]],'\n",
    "           '  [ [0.2, 0.8,-0.6, -0.9, 0.3], [0.5, 0.1, 0.3, -0.7,-0.9], [-0.3, 0.7, 0.5, -0.3, 0.4]],'\n",
    "           '  [ [-0.3, 0.7, -0.3] ]  ]',\n",
    "         b='[[0.1, -0.5, -0.5, 0.3, 0.2], [0.2,-0.5,0.3], [0.5]]',\n",
    "         activation_1='jump', activation_2='sigmoid', activation_3='reLU'):\n",
    "   \n",
    "    visualize_network(w, b, activations=[activation_1, activation_2, activation_3], maxv=1)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "Which is the behavior of the firsts weights-biases layer? And the lasts?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's apply a `factor` to scale all weights and biases!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@interact(activation_1=[\"sigmoid\",\"jump\",\"linear\",\"reLU\"],\n",
    "          activation_2=[\"sigmoid\",\"jump\",\"linear\",\"reLU\"],\n",
    "          factor=(0., 60.))\n",
    "def draw(weights=\"[[[0.2, 0.9],[-0.5, 0.3],[0.8, -1]],[[-0.3,0.7,0.5]]]\",\n",
    "         biases=\"[[0.1, -0.5, -0.5],[-0.2]]\",\n",
    "         activation_1='sigmoid', activation_2='linear',\n",
    "         factor=10):\n",
    "    \n",
    "    weights = eval(weights)\n",
    "    biases = eval(biases)\n",
    "\n",
    "    # this needs np.array(), because you cannot do factor*<python-list>\n",
    "    ws = [factor*np.array(matrix) for matrix in weights]\n",
    "    bs = [factor*np.array(vector) for vector in biases]\n",
    "    \n",
    "    visualize_network(ws, bs, [activation_1, activation_2], maxv=60)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "What happen when increasing the scale of weights and biases while using the sigmoid function?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4 Something not random\n",
    "\n",
    "Many superimposed lines can be used to construct arbitrary shapes, with only a single hidden layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@interact(factor=(1, 100), n_lines=(1, 20))\n",
    "def draw(factor=10, n_lines=3):\n",
    "    phi = np.linspace(0, 2*np.pi, n_lines+1)  # Angular variable\n",
    "    phi = phi[:-1]  # the last value is 2pi, which is equivalent to the 0\n",
    "\n",
    "    weight_hidden = np.zeros([n_lines, 2])   # comment this shape\n",
    "    weight_hidden[:,0] = factor*np.cos(phi)  # x=cos(phi)\n",
    "    weight_hidden[:,1] = factor*np.sin(phi)  # y=sin(phi)\n",
    "\n",
    "    bias_hidden = np.full(n_lines, factor*(+0.5))  # all neurons acts equally\n",
    "\n",
    "    visualize_network(weights=[ \n",
    "                                weight_hidden,           # from input to hidden\n",
    "                                np.full([1,n_lines],1.0) # from hidden to output\n",
    "                              ],\n",
    "                      biases=[ \n",
    "                                bias_hidden,\n",
    "                                [0.0]\n",
    "                             ],\n",
    "                      activations=['sigmoid',  # activation for hidden\n",
    "                                   'reLU'    # activation for output\n",
    "                                  ],\n",
    "                      size=30.0, maxv=60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Play with different `factor` above to see how sigmoid behavior changes.\n",
    "\n",
    "`n_lines` sets the number of lines on the figure, but what does it represent in the NN? Play with different number of lines.\n",
    "\n",
    "Why weights are made of sines and cosines?\n",
    "\n",
    "What are biases here? Play with it.\n",
    "\n",
    "Why the weights corresponding from hidden to output layer are full of ones?\n",
    "\n",
    "Why the output's activation function is linear?\n",
    "\n",
    "Draw a sharp and big six-pointed star using the code above.\n",
    "\n",
    "Draw a blurred and small circle using the code above"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "01_Basics_NeuralNetworksPython.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  },
  "toc": {
   "base_numbering": "0",
   "nav_menu": {
    "height": "326px",
    "width": "580px"
   },
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Table of Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "437.141px",
    "left": "43px",
    "top": "67.125px",
    "width": "296.297px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "position": {
    "height": "370.837px",
    "left": "987px",
    "right": "20px",
    "top": "120px",
    "width": "530px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}