10_Neural-machine-translation-with-attention-for-date-convert.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><div style=\"direction:rtl;font-family:B Lotus, B Nazanin, Tahoma\">به نام خدا</div></center>\n",
    "<img src=\"./logo.png\" alt=\"class.vision\" style=\"width: 200px;\"/>\n",
    "<h1><center><div style=\"direction:rtl;font-family:B Lotus, B Nazanin, Tahoma\">ترجمه ماشینی با توجه برای تبدیل تاریخ</div></center></h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Machine Translation\n",
    "\n",
    "\n",
    " (\"25th of June, 2009\")  --->  (\"2009-06-25\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.layers import Bidirectional, Concatenate, Permute, Dot, Input, LSTM, Multiply\n",
    "from keras.layers import RepeatVector, Dense, Activation, Lambda\n",
    "from keras.optimizers import Adam\n",
    "from keras.utils import to_categorical\n",
    "from keras.models import load_model, Model\n",
    "import keras.backend as K\n",
    "import numpy as np\n",
    "\n",
    "from faker import Faker\n",
    "import random\n",
    "from tqdm import tqdm\n",
    "from babel.dates import format_date\n",
    "from nmt_utils import *\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 - Translating human readable dates into machine readable dates\n",
    "\n",
    "The model you will build here could be used to translate from one language to another, such as translating from English to Hindi. However, language translation requires massive datasets and usually takes days of training on GPUs. To give you a place to experiment with these models even without using massive datasets, we will instead use a simpler \"date translation\" task. \n",
    "\n",
    "The network will input a date written in a variety of possible formats (*e.g. \"the 29th of August 1958\", \"03/30/1968\", \"24 JUNE 1987\"*) and translate them into standardized, machine readable dates (*e.g. \"1958-08-29\", \"1968-03-30\", \"1987-06-24\"*). We will have the network learn to output dates in the common machine-readable format YYYY-MM-DD. \n",
    "\n",
    "\n",
    "\n",
    "<!-- \n",
    "Take a look at [nmt_utils.py](./nmt_utils.py) to see all the formatting. Count and figure out how the formats work, you will need this knowledge later. !--> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 - Dataset\n",
    "\n",
    "We will train the model on a dataset of 10000 human readable dates and their equivalent, standardized, machine readable dates. Let's run the following cells to load the dataset and print some examples. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████| 10000/10000 [00:00<00:00, 13839.02it/s]\n"
     ]
    }
   ],
   "source": [
    "m = 10000\n",
    "dataset, src_vocab, dest_vocab, inv_dest_vocab = load_dataset(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('9 may 1998', '1998-05-09'),\n",
       " ('10.11.19', '2019-11-10'),\n",
       " ('9/10/70', '1970-09-10'),\n",
       " ('saturday april 28 1990', '1990-04-28'),\n",
       " ('thursday january 26 1995', '1995-01-26'),\n",
       " ('monday march 7 1983', '1983-03-07'),\n",
       " ('sunday may 22 1988', '1988-05-22'),\n",
       " ('08 jul 2008', '2008-07-08'),\n",
       " ('8 sep 1999', '1999-09-08'),\n",
       " ('thursday january 1 1981', '1981-01-01')]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You've loaded:\n",
    "- `dataset`: a list of tuples of (human readable date, machine readable date)\n",
    "- `src_vocab`: a python dictionary mapping all characters used in the human readable dates to an integer-valued index \n",
    "- `dest_vocab`: a python dictionary mapping all characters used in machine readable dates to an integer-valued index. These indices are not necessarily consistent with `src_vocab`. \n",
    "- `inv_dest_vocab`: the inverse dictionary of `dest_vocab`, mapping from indices back to characters. \n",
    "\n",
    "Let's preprocess the data and map the raw text data into the index values. We will also use Tx=30 (which we assume is the maximum length of the human readable date; if we get a longer input, we would have to truncate it) and Ty=10 (since \"YYYY-MM-DD\" is 10 characters long). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X.shape: (10000, 30)\n",
      "Y.shape: (10000, 10)\n",
      "Xoh.shape: (10000, 30, 37)\n",
      "Yoh.shape: (10000, 10, 11)\n"
     ]
    }
   ],
   "source": [
    "Tx = 30\n",
    "Ty = 10\n",
    "X, Y, Xoh, Yoh = preprocess_data(dataset, src_vocab, dest_vocab, Tx, Ty)\n",
    "\n",
    "print(\"X.shape:\", X.shape)\n",
    "print(\"Y.shape:\", Y.shape)\n",
    "print(\"Xoh.shape:\", Xoh.shape)\n",
    "print(\"Yoh.shape:\", Yoh.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([12,  0, 24, 13, 34,  0,  4, 12, 12, 11, 36, 36, 36, 36, 36, 36, 36,\n",
       "       36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2, 10, 10,  9,  0,  1,  6,  0,  1, 10])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You now have:\n",
    "- `X`: a processed version of the human readable dates in the training set, where each character is replaced by an index mapped to the character via `src_vocab`. Each date is further padded to $T_x$ values with a special character (< pad >). `X.shape = (m, Tx)`\n",
    "- `Y`: a processed version of the machine readable dates in the training set, where each character is replaced by the index it is mapped to in `dest_vocab`. You should have `Y.shape = (m, Ty)`. \n",
    "- `Xoh`: one-hot version of `X`, the \"1\" entry's index is mapped to the character thanks to `src_vocab`. `Xoh.shape = (m, Tx, len(src_vocab))`\n",
    "- `Yoh`: one-hot version of `Y`, the \"1\" entry's index is mapped to the character thanks to `dest_vocab`. `Yoh.shape = (m, Tx, len(dest_vocab))`. Here, `len(dest_vocab) = 11` since there are 11 characters ('-' as well as 0-9). \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets also look at some examples of preprocessed training examples. Feel free to play with `index` in the cell below to navigate the dataset and see how source/target dates are preprocessed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source date: 9 may 1998\n",
      "Target date: 1998-05-09\n",
      "\n",
      "Source after preprocessing (indices): [12  0 24 13 34  0  4 12 12 11 36 36 36 36 36 36 36 36 36 36 36 36 36 36\n",
      " 36 36 36 36 36 36]\n",
      "Target after preprocessing (indices): [ 2 10 10  9  0  1  6  0  1 10]\n",
      "\n",
      "Source after preprocessing (one-hot): [[0. 0. 0. ... 0. 0. 0.]\n",
      " [1. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " ...\n",
      " [0. 0. 0. ... 0. 0. 1.]\n",
      " [0. 0. 0. ... 0. 0. 1.]\n",
      " [0. 0. 0. ... 0. 0. 1.]]\n",
      "Target after preprocessing (one-hot): [[0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0.]\n",
      " [1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]\n",
      " [1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]]\n"
     ]
    }
   ],
   "source": [
    "index = 0\n",
    "print(\"Source date:\", dataset[index][0])\n",
    "print(\"Target date:\", dataset[index][1])\n",
    "print()\n",
    "print(\"Source after preprocessing (indices):\", X[index])\n",
    "print(\"Target after preprocessing (indices):\", Y[index])\n",
    "print()\n",
    "print(\"Source after preprocessing (one-hot):\", Xoh[index])\n",
    "print(\"Target after preprocessing (one-hot):\", Yoh[index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 - Neural machine translation with attention\n",
    "\n",
    "If you had to translate a book's paragraph from French to English, you would not read the whole paragraph, then close the book and translate. Even during the translation process, you would read/re-read and focus on the parts of the French paragraph corresponding to the parts of the English you are writing down. \n",
    "\n",
    "The attention mechanism tells a Neural Machine Translation model where it should pay attention to at any step. \n",
    "\n",
    "\n",
    "### 2.1 - Attention mechanism\n",
    "\n",
    "Here is a figure to remind you how the model works. The diagram on the left shows the attention model. The diagram on the right shows what one \"Attention\" step does to calculate the attention variables $\\alpha^{\\langle t, t' \\rangle}$, which are used to compute the context variable $context^{\\langle t \\rangle}$ for each timestep in the output ($t=1, \\ldots, T_y$). \n",
    "\n",
    "<table>\n",
    "<td> \n",
    "<img src=\"images/attn_model.png\" style=\"width:500;height:500px;\"> <br>\n",
    "</td> \n",
    "<td> \n",
    "<img src=\"images/attn_mechanism.png\" style=\"width:500;height:500px;\"> <br>\n",
    "</td> \n",
    "</table>\n",
    "<caption><center> **Figure 1**: Neural machine translation with attention</center></caption>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Here are some properties of the model that you may notice: \n",
    "\n",
    "- There are two separate LSTMs in this model (see diagram on the left). Because the one at the bottom of the picture is a Bi-directional LSTM and comes *before* the attention mechanism, we will call it *pre-attention* Bi-LSTM. The LSTM at the top of the diagram comes *after* the attention mechanism, so we will call it the *post-attention* LSTM. The pre-attention Bi-LSTM goes through $T_x$ time steps; the post-attention LSTM goes through $T_y$ time steps. \n",
    "\n",
    "- The post-attention LSTM passes $s^{\\langle t \\rangle}, c^{\\langle t \\rangle}$ from one time step to the next. In the slides, we were using only a basic RNN for the post-activation sequence model, so the state captured by the RNN output activations $s^{\\langle t\\rangle}$. But since we are using an LSTM here, the LSTM has both the output activation $s^{\\langle t\\rangle}$ and the hidden cell state $c^{\\langle t\\rangle}$. However, unlike text generation examples , in this model the post-activation LSTM at time $t$ does will not take the specific generated $y^{\\langle t-1 \\rangle}$ as input; it only takes $s^{\\langle t\\rangle}$ and $c^{\\langle t\\rangle}$ as input. We have designed the model this way, because (unlike language generation where adjacent characters are highly correlated) there isn't as strong a dependency between the previous character and the next character in a YYYY-MM-DD date. \n",
    "\n",
    "- We use $a^{\\langle t \\rangle} = [\\overrightarrow{a}^{\\langle t \\rangle}; \\overleftarrow{a}^{\\langle t \\rangle}]$ to represent the concatenation of the activations of both the forward-direction and backward-directions of the pre-attention Bi-LSTM. \n",
    "\n",
    "- The diagram on the right uses a `RepeatVector` node to copy $s^{\\langle t-1 \\rangle}$'s value $T_x$ times, and then `Concatenation` to concatenate $s^{\\langle t-1 \\rangle}$ and $a^{\\langle t \\rangle}$ to compute $e^{\\langle t, t'}$, which is then passed through a softmax to compute $\\alpha^{\\langle t, t' \\rangle}$. We'll explain how to use `RepeatVector` and `Concatenation` in Keras below. \n",
    "\n",
    "Lets implement this model. You will start by implementing two functions: `one_step_attention()` and `model()`.\n",
    "\n",
    "**1) `one_step_attention()`**: At step $t$, given all the hidden states of the Bi-LSTM ($[a^{<1>},a^{<2>}, ..., a^{<T_x>}]$) and the previous hidden state of the second LSTM ($s^{<t-1>}$), `one_step_attention()` will compute the attention weights ($[\\alpha^{<t,1>},\\alpha^{<t,2>}, ..., \\alpha^{<t,T_x>}]$) and output the context vector (see Figure  1 (right) for details):\n",
    "$$context^{<t>} = \\sum_{t' = 0}^{T_x} \\alpha^{<t,t'>}a^{<t'>}\\tag{1}$$ \n",
    "\n",
    "  \n",
    "**2) `model()`**: Implements the entire model. It first runs the input through a Bi-LSTM to get back $[a^{<1>},a^{<2>}, ..., a^{<T_x>}]$. Then, it calls `one_step_attention()` $T_y$ times (`for` loop). At each iteration of this loop, it gives the computed context vector $c^{<t>}$ to the second LSTM, and runs the output of the LSTM through a dense layer with softmax activation to generate a prediction $\\hat{y}^{<t>}$. \n",
    "\n",
    "\n",
    "The function `model()` will call the layers in `one_step_attention()` $T_y$ using a for-loop, and it is important that all $T_y$ copies have the same weights. I.e., it should not re-initiaiize the weights every time. In other words, all $T_y$ steps should have shared weights. Here's how you can implement layers with shareable weights in Keras:\n",
    "1. Define the layer objects (as global variables for examples).\n",
    "2. Call these objects when propagating the input.\n",
    "\n",
    "Please check the Keras documentation to make sure you understand what these layers are: [RepeatVector()](https://keras.io/layers/core/#repeatvector), [Concatenate()](https://keras.io/layers/merge/#concatenate), [Dense()](https://keras.io/layers/core/#dense), [Activation()](https://keras.io/layers/core/#activation), [Dot()](https://keras.io/layers/merge/#dot)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defined shared layers as global variables\n",
    "repeator = RepeatVector(Tx)\n",
    "concatenator = Concatenate(axis=-1)\n",
    "densor1 = Dense(10, activation = \"tanh\")\n",
    "densor2 = Dense(1, activation = \"relu\")\n",
    "activator = Activation(softmax, name='attention_weights') # We are using a custom softmax(axis = 1) loaded in this notebook\n",
    "dotor = Dot(axes = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#one_step_attention\n",
    "\n",
    "def one_step_attention(a, s_prev):\n",
    "    # repeat s_prev to be of shape (m, Tx, n_s) so that you can concatenate it with all hidden states \"a\" \n",
    "    s_prev = repeator(s_prev)\n",
    "    # concatenate a and s_prev on the last axis\n",
    "    concat = concatenator([a, s_prev])\n",
    "    # propagate concat through a small fully-connected neural network to compute the \"intermediate energies\" variable e.\n",
    "    e = densor1(concat)\n",
    "    # propagate e through a small fully-connected neural network to compute the \"energies\" variable energies. \n",
    "    energies = densor2(e)\n",
    "    # Use \"activator\" on \"energies\" to compute the attention weights \"alphas\" (≈ 1 line)\n",
    "    alphas = activator(energies)\n",
    "    # Use dotor together with \"alphas\" and \"a\" to compute the context vector to be given to the next (post-attention) LSTM-cell (≈ 1 line)\n",
    "    context = dotor([alphas, a])\n",
    "    \n",
    "    return context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_a = 32\n",
    "n_s = 64\n",
    "\n",
    "post_activation_LSTM_cell = LSTM(n_s, return_state = True)\n",
    "output_layer = Dense(len(dest_vocab), activation=softmax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can use these layers $T_y$ times in a `for` loop to generate the outputs, and their parameters will not be reinitialized. You will have to carry out the following steps: \n",
    "\n",
    "1. Propagate the input into a [Bidirectional](https://keras.io/layers/wrappers/#bidirectional) [LSTM](https://keras.io/layers/recurrent/#lstm)\n",
    "2. Iterate for $t = 0, \\dots, T_y-1$: \n",
    "    1. Call `one_step_attention()` on $[\\alpha^{<t,1>},\\alpha^{<t,2>}, ..., \\alpha^{<t,T_x>}]$ and $s^{<t-1>}$ to get the context vector $context^{<t>}$.\n",
    "    2. Give $context^{<t>}$ to the post-attention LSTM cell. Remember pass in the previous hidden-state $s^{\\langle t-1\\rangle}$ and cell-states $c^{\\langle t-1\\rangle}$ of this LSTM using `initial_state= [previous hidden state, previous cell state]`. Get back the new hidden state $s^{<t>}$ and the new cell state $c^{<t>}$.\n",
    "    3. Apply a softmax layer to $s^{<t>}$, get the output. \n",
    "    4. Save the output by adding it to the list of outputs.\n",
    "\n",
    "3. Create your Keras model instance, it should have three inputs (\"inputs\", $s^{<0>}$ and $c^{<0>}$) and output the list of \"outputs\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GRADED FUNCTION: model\n",
    "\n",
    "def model(Tx, Ty, n_a, n_s, src_vocab_size, dest_vocab_size):\n",
    "    \"\"\"\n",
    "    Arguments:\n",
    "    Tx -- length of the input sequence\n",
    "    Ty -- length of the output sequence\n",
    "    n_a -- hidden state size of the Bi-LSTM\n",
    "    n_s -- hidden state size of the post-attention LSTM\n",
    "    src_vocab_size -- size of the python dictionary \"src_vocab\"\n",
    "    dest_vocab_size -- size of the python dictionary \"dest_vocab\"\n",
    "\n",
    "    Returns:\n",
    "    model -- Keras model instance\n",
    "    \"\"\"\n",
    "    \n",
    "    # Define the inputs of your model with a shape (Tx,)\n",
    "    # Define s0 and c0, initial hidden state for the decoder LSTM of shape (n_s,)\n",
    "    X = Input(shape=(Tx, src_vocab_size))\n",
    "    s0 = Input(shape=(n_s,), name='s0')\n",
    "    c0 = Input(shape=(n_s,), name='c0')\n",
    "    s = s0\n",
    "    c = c0\n",
    "    \n",
    "    # Initialize empty list of outputs\n",
    "    outputs = []\n",
    "    \n",
    "    \n",
    "    # pre-attention Bi-LSTM.\n",
    "    a = Bidirectional(LSTM(n_a, return_sequences=True))(X)\n",
    "    \n",
    "    # Iterate for Ty steps\n",
    "    for t in range(Ty):\n",
    "    \n",
    "        #Perform one step of the attention mechanism to get back the context vector at step t \n",
    "        context = one_step_attention(a, s)\n",
    "        \n",
    "        #post-attention LSTM cell \n",
    "        s, _, c = post_activation_LSTM_cell(context, initial_state = [s, c])\n",
    "        \n",
    "        # Dense layer to the hidden state output of the post-attention LSTM\n",
    "        out = Dense(len(dest_vocab), activation=softmax)(s)\n",
    "        \n",
    "        outputs.append(out)\n",
    "        \n",
    "    model = Model(inputs=[X, s0, c0], outputs=outputs)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to create your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = model(Tx, Ty, n_a, n_s, len(src_vocab), len(dest_vocab))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get a summary of the model to check if it matches the expected output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            (None, 30, 37)       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "s0 (InputLayer)                 (None, 64)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_1 (Bidirectional) (None, 30, 64)       17920       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "repeat_vector_1 (RepeatVector)  (None, 30, 64)       0           s0[0][0]                         \n",
      "                                                                 lstm_1[0][0]                     \n",
      "                                                                 lstm_1[1][0]                     \n",
      "                                                                 lstm_1[2][0]                     \n",
      "                                                                 lstm_1[3][0]                     \n",
      "                                                                 lstm_1[4][0]                     \n",
      "                                                                 lstm_1[5][0]                     \n",
      "                                                                 lstm_1[6][0]                     \n",
      "                                                                 lstm_1[7][0]                     \n",
      "                                                                 lstm_1[8][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 30, 128)      0           bidirectional_1[0][0]            \n",
      "                                                                 repeat_vector_1[0][0]            \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 repeat_vector_1[1][0]            \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 repeat_vector_1[2][0]            \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 repeat_vector_1[3][0]            \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 repeat_vector_1[4][0]            \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 repeat_vector_1[5][0]            \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 repeat_vector_1[6][0]            \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 repeat_vector_1[7][0]            \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 repeat_vector_1[8][0]            \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 repeat_vector_1[9][0]            \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 30, 10)       1290        concatenate_1[0][0]              \n",
      "                                                                 concatenate_1[1][0]              \n",
      "                                                                 concatenate_1[2][0]              \n",
      "                                                                 concatenate_1[3][0]              \n",
      "                                                                 concatenate_1[4][0]              \n",
      "                                                                 concatenate_1[5][0]              \n",
      "                                                                 concatenate_1[6][0]              \n",
      "                                                                 concatenate_1[7][0]              \n",
      "                                                                 concatenate_1[8][0]              \n",
      "                                                                 concatenate_1[9][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 30, 1)        11          dense_1[0][0]                    \n",
      "                                                                 dense_1[1][0]                    \n",
      "                                                                 dense_1[2][0]                    \n",
      "                                                                 dense_1[3][0]                    \n",
      "                                                                 dense_1[4][0]                    \n",
      "                                                                 dense_1[5][0]                    \n",
      "                                                                 dense_1[6][0]                    \n",
      "                                                                 dense_1[7][0]                    \n",
      "                                                                 dense_1[8][0]                    \n",
      "                                                                 dense_1[9][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "attention_weights (Activation)  (None, 30, 1)        0           dense_2[0][0]                    \n",
      "                                                                 dense_2[1][0]                    \n",
      "                                                                 dense_2[2][0]                    \n",
      "                                                                 dense_2[3][0]                    \n",
      "                                                                 dense_2[4][0]                    \n",
      "                                                                 dense_2[5][0]                    \n",
      "                                                                 dense_2[6][0]                    \n",
      "                                                                 dense_2[7][0]                    \n",
      "                                                                 dense_2[8][0]                    \n",
      "                                                                 dense_2[9][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dot_1 (Dot)                     (None, 1, 64)        0           attention_weights[0][0]          \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 attention_weights[1][0]          \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 attention_weights[2][0]          \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 attention_weights[3][0]          \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 attention_weights[4][0]          \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 attention_weights[5][0]          \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 attention_weights[6][0]          \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 attention_weights[7][0]          \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 attention_weights[8][0]          \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "                                                                 attention_weights[9][0]          \n",
      "                                                                 bidirectional_1[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "c0 (InputLayer)                 (None, 64)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "lstm_1 (LSTM)                   [(None, 64), (None,  33024       dot_1[0][0]                      \n",
      "                                                                 s0[0][0]                         \n",
      "                                                                 c0[0][0]                         \n",
      "                                                                 dot_1[1][0]                      \n",
      "                                                                 lstm_1[0][0]                     \n",
      "                                                                 lstm_1[0][2]                     \n",
      "                                                                 dot_1[2][0]                      \n",
      "                                                                 lstm_1[1][0]                     \n",
      "                                                                 lstm_1[1][2]                     \n",
      "                                                                 dot_1[3][0]                      \n",
      "                                                                 lstm_1[2][0]                     \n",
      "                                                                 lstm_1[2][2]                     \n",
      "                                                                 dot_1[4][0]                      \n",
      "                                                                 lstm_1[3][0]                     \n",
      "                                                                 lstm_1[3][2]                     \n",
      "                                                                 dot_1[5][0]                      \n",
      "                                                                 lstm_1[4][0]                     \n",
      "                                                                 lstm_1[4][2]                     \n",
      "                                                                 dot_1[6][0]                      \n",
      "                                                                 lstm_1[5][0]                     \n",
      "                                                                 lstm_1[5][2]                     \n",
      "                                                                 dot_1[7][0]                      \n",
      "                                                                 lstm_1[6][0]                     \n",
      "                                                                 lstm_1[6][2]                     \n",
      "                                                                 dot_1[8][0]                      \n",
      "                                                                 lstm_1[7][0]                     \n",
      "                                                                 lstm_1[7][2]                     \n",
      "                                                                 dot_1[9][0]                      \n",
      "                                                                 lstm_1[8][0]                     \n",
      "                                                                 lstm_1[8][2]                     \n",
      "__________________________________________________________________________________________________\n",
      "dense_4 (Dense)                 (None, 11)           715         lstm_1[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "dense_5 (Dense)                 (None, 11)           715         lstm_1[1][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "dense_6 (Dense)                 (None, 11)           715         lstm_1[2][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "dense_7 (Dense)                 (None, 11)           715         lstm_1[3][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "dense_8 (Dense)                 (None, 11)           715         lstm_1[4][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "dense_9 (Dense)                 (None, 11)           715         lstm_1[5][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "dense_10 (Dense)                (None, 11)           715         lstm_1[6][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "dense_11 (Dense)                (None, 11)           715         lstm_1[7][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "dense_12 (Dense)                (None, 11)           715         lstm_1[8][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "dense_13 (Dense)                (None, 11)           715         lstm_1[9][0]                     \n",
      "==================================================================================================\n",
      "Total params: 59,395\n",
      "Trainable params: 59,395\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt = Adam(lr=0.005, beta_1=0.9, beta_2=0.999, decay=0.01)\n",
    "model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "### END CODE HERE ###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last step is to define all your inputs and outputs to fit the model:\n",
    "- You already have X of shape $(m = 10000, T_x = 30)$ containing the training examples.\n",
    "- You need to create `s0` and `c0` to initialize your `post_activation_LSTM_cell` with 0s.\n",
    "- Given the `model()` you coded, you need the \"outputs\" to be a list of 11 elements of shape (m, T_y). So that: `outputs[i][0], ..., outputs[i][Ty]` represent the true labels (characters) corresponding to the $i^{th}$ training example (`X[i]`). More generally, `outputs[i][j]` is the true label of the $j^{th}$ character in the $i^{th}$ training example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X.shape: (10000, 30)\n",
      "Y.shape: (10000, 10)\n",
      "Xoh.shape: (10000, 30, 37)\n",
      "Yoh.shape: (10000, 10, 11)\n"
     ]
    }
   ],
   "source": [
    "print(\"X.shape:\", X.shape)\n",
    "print(\"Y.shape:\", Y.shape)\n",
    "print(\"Xoh.shape:\", Xoh.shape)\n",
    "print(\"Yoh.shape:\", Yoh.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "s0 = np.zeros((m, n_s))\n",
    "c0 = np.zeros((m, n_s))\n",
    "outputs = list(Yoh.swapaxes(0,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 64)\n",
      "(10000, 64)\n",
      "(10, 10000, 11)\n"
     ]
    }
   ],
   "source": [
    "print(s0.shape)\n",
    "print(c0.shape)\n",
    "print(np.shape(outputs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now fit the model and run it for one epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "10000/10000 [==============================] - 69s 7ms/step - loss: 12.4757 - dense_4_loss: 0.8144 - dense_5_loss: 0.8247 - dense_6_loss: 1.6834 - dense_7_loss: 2.3235 - dense_8_loss: 0.1300 - dense_9_loss: 0.6678 - dense_10_loss: 2.2892 - dense_11_loss: 0.0953 - dense_12_loss: 1.3104 - dense_13_loss: 2.3370 - dense_4_accuracy: 0.5912 - dense_5_accuracy: 0.5823 - dense_6_accuracy: 0.1990 - dense_7_accuracy: 0.0984 - dense_8_accuracy: 0.9961 - dense_9_accuracy: 0.7391 - dense_10_accuracy: 0.1662 - dense_11_accuracy: 0.9900 - dense_12_accuracy: 0.3190 - dense_13_accuracy: 0.1011\n",
      "Epoch 2/30\n",
      " 4200/10000 [===========>..................] - ETA: 33s - loss: 11.5541 - dense_4_loss: 0.6167 - dense_5_loss: 0.6409 - dense_6_loss: 1.5992 - dense_7_loss: 2.3100 - dense_8_loss: 3.3439e-04 - dense_9_loss: 0.5649 - dense_10_loss: 2.2639 - dense_11_loss: 2.9461e-04 - dense_12_loss: 1.2471 - dense_13_loss: 2.3109 - dense_4_accuracy: 0.6169 - dense_5_accuracy: 0.6169 - dense_6_accuracy: 0.2210 - dense_7_accuracy: 0.1067 - dense_8_accuracy: 1.0000 - dense_9_accuracy: 0.7481 - dense_10_accuracy: 0.1676 - dense_11_accuracy: 1.0000 - dense_12_accuracy: 0.3226 - dense_13_accuracy: 0.1045"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-18-85d521a1ffdf>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mmodel\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mXoh\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0ms0\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mc0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0moutputs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mepochs\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m30\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m100\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m~\\Miniconda3\\envs\\tf2\\lib\\site-packages\\keras\\engine\\training.py\u001b[0m in \u001b[0;36mfit\u001b[1;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)\u001b[0m\n\u001b[0;32m   1237\u001b[0m                                         \u001b[0msteps_per_epoch\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0msteps_per_epoch\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1238\u001b[0m                                         \u001b[0mvalidation_steps\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mvalidation_steps\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1239\u001b[1;33m                                         validation_freq=validation_freq)\n\u001b[0m\u001b[0;32m   1240\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1241\u001b[0m     def evaluate(self,\n",
      "\u001b[1;32m~\\Miniconda3\\envs\\tf2\\lib\\site-packages\\keras\\engine\\training_arrays.py\u001b[0m in \u001b[0;36mfit_loop\u001b[1;34m(model, fit_function, fit_inputs, out_labels, batch_size, epochs, verbose, callbacks, val_function, val_inputs, shuffle, initial_epoch, steps_per_epoch, validation_steps, validation_freq)\u001b[0m\n\u001b[0;32m    194\u001b[0m                     \u001b[0mins_batch\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mins_batch\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtoarray\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    195\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 196\u001b[1;33m                 \u001b[0mouts\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mfit_function\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mins_batch\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    197\u001b[0m                 \u001b[0mouts\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mto_list\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mouts\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    198\u001b[0m                 \u001b[1;32mfor\u001b[0m \u001b[0ml\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mo\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mzip\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mout_labels\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mouts\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Miniconda3\\envs\\tf2\\lib\\site-packages\\tensorflow_core\\python\\keras\\backend.py\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, inputs)\u001b[0m\n\u001b[0;32m   3738\u001b[0m         \u001b[0mvalue\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mmath_ops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcast\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtensor\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdtype\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3739\u001b[0m       \u001b[0mconverted_inputs\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 3740\u001b[1;33m     \u001b[0moutputs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_graph_fn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0mconverted_inputs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   3741\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3742\u001b[0m     \u001b[1;31m# EagerTensor.numpy() will often make a copy to ensure memory safety.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Miniconda3\\envs\\tf2\\lib\\site-packages\\tensorflow_core\\python\\eager\\function.py\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m   1079\u001b[0m       \u001b[0mTypeError\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mFor\u001b[0m \u001b[0minvalid\u001b[0m \u001b[0mpositional\u001b[0m\u001b[1;33m/\u001b[0m\u001b[0mkeyword\u001b[0m \u001b[0margument\u001b[0m \u001b[0mcombinations\u001b[0m\u001b[1;33m.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1080\u001b[0m     \"\"\"\n\u001b[1;32m-> 1081\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_call_impl\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1082\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1083\u001b[0m   \u001b[1;32mdef\u001b[0m \u001b[0m_call_impl\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcancellation_manager\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Miniconda3\\envs\\tf2\\lib\\site-packages\\tensorflow_core\\python\\eager\\function.py\u001b[0m in \u001b[0;36m_call_impl\u001b[1;34m(self, args, kwargs, cancellation_manager)\u001b[0m\n\u001b[0;32m   1119\u001b[0m       raise TypeError(\"Keyword arguments {} unknown. Expected {}.\".format(\n\u001b[0;32m   1120\u001b[0m           list(kwargs.keys()), list(self._arg_keywords)))\n\u001b[1;32m-> 1121\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_call_flat\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcaptured_inputs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcancellation_manager\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1122\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1123\u001b[0m   \u001b[1;32mdef\u001b[0m \u001b[0m_filtered_call\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Miniconda3\\envs\\tf2\\lib\\site-packages\\tensorflow_core\\python\\eager\\function.py\u001b[0m in \u001b[0;36m_call_flat\u001b[1;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[0;32m   1222\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mexecuting_eagerly\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1223\u001b[0m       flat_outputs = forward_function.call(\n\u001b[1;32m-> 1224\u001b[1;33m           ctx, args, cancellation_manager=cancellation_manager)\n\u001b[0m\u001b[0;32m   1225\u001b[0m     \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1226\u001b[0m       \u001b[0mgradient_name\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_delayed_rewrite_functions\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mregister\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Miniconda3\\envs\\tf2\\lib\\site-packages\\tensorflow_core\\python\\eager\\function.py\u001b[0m in \u001b[0;36mcall\u001b[1;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[0;32m    509\u001b[0m               \u001b[0minputs\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    510\u001b[0m               \u001b[0mattrs\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"executor_type\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mexecutor_type\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m\"config_proto\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mconfig\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 511\u001b[1;33m               ctx=ctx)\n\u001b[0m\u001b[0;32m    512\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    513\u001b[0m           outputs = execute.execute_with_cancellation(\n",
      "\u001b[1;32m~\\Miniconda3\\envs\\tf2\\lib\\site-packages\\tensorflow_core\\python\\eager\\execute.py\u001b[0m in \u001b[0;36mquick_execute\u001b[1;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[0;32m     59\u001b[0m     tensors = pywrap_tensorflow.TFE_Py_Execute(ctx._handle, device_name,\n\u001b[0;32m     60\u001b[0m                                                \u001b[0mop_name\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0minputs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mattrs\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 61\u001b[1;33m                                                num_outputs)\n\u001b[0m\u001b[0;32m     62\u001b[0m   \u001b[1;32mexcept\u001b[0m \u001b[0mcore\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_NotOkStatusException\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     63\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "model.fit([Xoh, s0, c0], outputs, epochs=30, batch_size=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While training you can see the loss as well as the accuracy on each of the 10 positions of the output. The table below gives you an example of what the accuracies could be if the batch had 2 examples: \n",
    "\n",
    "<img src=\"images/table.png\" style=\"width:700;height:200px;\"> <br>\n",
    "<caption><center>Thus, `dense_2_acc_8: 0.89` means that you are predicting the 7th character of the output correctly 89% of the time in the current batch of data. </center></caption>\n",
    "\n",
    "\n",
    "We have run this model for longer, and saved the weights. Run the next cell to load our weights. (By training a model for several minutes, you should be able to obtain a model of similar accuracy, but loading our model will save you time.) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr/>\n",
    "<hr/>\n",
    "You can Download our weights:\n",
    "    \n",
    "http://deepnn.ir/class.vision/andrewng/machine-translation-with-attention/models.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.load_weights('models/model.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now see the results on new examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "EXAMPLES = ['3 May 1979', '5 April 09', '21th of August 2016', 'Tue 10 Jul 2007', 'Saturday May 9 2018', 'March 3 2001', 'March 3rd 2001', '1 March 2001']\n",
    "for example in EXAMPLES:\n",
    "    \n",
    "    source = string_to_int(example, Tx, src_vocab)\n",
    "    source = np.array(list(map(lambda x: to_categorical(x, num_classes=len(src_vocab)), source))).swapaxes(0,1)\n",
    "    prediction = model.predict([source, s0, c0])\n",
    "    prediction = np.argmax(prediction, axis = -1)\n",
    "    output = [inv_dest_vocab[int(i)] for i in prediction]\n",
    "    \n",
    "    print(\"source:\", example)\n",
    "    print(\"output:\", ''.join(output))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also change these examples to test with your own examples. The next part will give you a better sense on what the attention mechanism is doing--i.e., what part of the input the network is paying attention to when generating a particular output character. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 - Visualizing Attention (Optional / Ungraded)\n",
    "\n",
    "Since the problem has a fixed output length of 10, it is also possible to carry out this task using 10 different softmax units to generate the 10 characters of the output. But one advantage of the attention model is that each part of the output (say the month) knows it needs to depend only on a small part of the input (the characters in the input giving the month). We can  visualize what part of the output is looking at what part of the input.\n",
    "\n",
    "Consider the task of translating \"Saturday 9 May 2018\" to \"2018-05-09\". If we visualize the computed $\\alpha^{\\langle t, t' \\rangle}$ we get this: \n",
    "\n",
    "<img src=\"images/date_attention.png\" style=\"width:600;height:300px;\"> <br>\n",
    "<caption><center> **Figure 8**: Full Attention Map</center></caption>\n",
    "\n",
    "Notice how the output ignores the \"Saturday\" portion of the input. None of the output timesteps are paying much attention to that portion of the input. We see also that 9 has been translated as 09 and May has been correctly translated into 05, with the output paying attention to the parts of the input it needs to to make the translation. The year mostly requires it to pay attention to the input's \"18\" in order to generate \"2018.\" \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 - Getting the activations from the network\n",
    "\n",
    "Lets now visualize the attention values in your network. We'll propagate an example through the network, then visualize the values of $\\alpha^{\\langle t, t' \\rangle}$. \n",
    "\n",
    "To figure out where the attention values are located, let's start by printing a summary of the model ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Navigate through the output of `model.summary()` above. You can see that the layer named `attention_weights` outputs the `alphas` of shape (m, 30, 1) before `dot_2` computes the context vector for every time step $t = 0, \\ldots, T_y-1$. Lets get the activations from this layer.\n",
    "\n",
    "The function `attention_map()` pulls out the attention values from your model and plots them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "attention_map = plot_attention_map(model, src_vocab, inv_dest_vocab, \"Tuesday 09 Oct 1993\", num = 7, n_s = 64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the generated plot you can observe the values of the attention weights for each character of the predicted output. Examine this plot and check that where the network is paying attention makes sense to you.\n",
    "\n",
    "In the date translation application, you will observe that most of the time attention helps predict the year, and hasn't much impact on predicting the day/month."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Congratulations!\n",
    "\n",
    "\n",
    "You have come to the end of this assignment \n",
    "\n",
    "<font color='blue'> **Here's what you should remember from this notebook**:\n",
    "\n",
    "- Machine translation models can be used to map from one sequence to another. They are useful not just for translating human languages (like French->English) but also for tasks like date format translation. \n",
    "- An attention mechanism allows a network to focus on the most relevant parts of the input when producing a specific part of the output. \n",
    "- A network using an attention mechanism can translate from inputs of length $T_x$ to outputs of length $T_y$, where $T_x$ and $T_y$ can be different. \n",
    "- You can visualize attention weights $\\alpha^{\\langle t,t' \\rangle}$ to see what the network is paying attention to while generating each output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations on finishing this assignment! You are now able to implement an attention model and use it to learn complex mappings from one sequence to another. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<div style=\"direction:rtl;text-align:right;font-family:B Lotus, B Nazanin, Tahoma\"> دوره پیشرفته یادگیری عمیق<br>علیرضا اخوان پور<br>  آبان و آذر 1399<br>\n",
    "</div>\n",
    "<a href=\"http://class.vision\">Class.Vision</a> - <a href=\"http://AkhavanPour.ir\">AkhavanPour.ir</a> - <a href=\"https://github.com/Alireza-Akhavan/\">GitHub</a>\n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "nlp-sequence-models",
   "graded_item_id": "n16CQ",
   "launcher_item_id": "npjGi"
  },
  "kernelspec": {
   "display_name": "tf2-GPU",
   "language": "python",
   "name": "tf2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}