Neural_Machine_Translation_tf_1L_8H.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Neural_Machine_Translation_tf_1L_8H.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "#### Mounting Google Drive"
      ],
      "metadata": {
        "id": "0dqNGaFLKKAJ"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bVenoImvJZs9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "00e6c985-278f-41a4-997d-fd3a0f1885d5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ]
        }
      ],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "cd \"/content/drive/MyDrive/IASNLP\""
      ],
      "metadata": {
        "id": "iNpRDc1-KMj8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "27c2053a-bb61-4f9a-8932-bc856f598df5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/drive/MyDrive/IASNLP\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Importing Necessary Libraries"
      ],
      "metadata": {
        "id": "OlMuBUdIApMX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install sentencepiece\n",
        "!pip install sacrebleu\n",
        "!pip install tensorflow-gpu # ---> For GPU"
      ],
      "metadata": {
        "id": "Iwi6TDKsB0St"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers\n",
        "from tensorflow.keras import Model\n",
        "from tensorflow.keras import Sequential\n",
        "from tensorflow.keras.layers import Dense, Lambda, Layer, Embedding, LayerNormalization, Input\n",
        "import sentencepiece as spm\n",
        "from sacrebleu.metrics import BLEU, CHRF, TER\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from collections import Counter"
      ],
      "metadata": {
        "id": "DXSYYy2b-jMM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Loading Data"
      ],
      "metadata": {
        "id": "ToLMSNAHBnvJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We load the `train_data`, `train_dev_data`, `test_val_data` and `test_data` as well as the byte-pair encoder tokenizer for English and Bengali i.e. `sp_en_bpe` and `s_ben_bpe`."
      ],
      "metadata": {
        "id": "zXIf-FL7IG7f"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "train_data, train_dev_data = pd.read_csv(\"train_data.csv\")[['src', 'tgt']], pd.read_csv(\"train_dev.csv\")[['src', 'tgt']] \n",
        "test_val_data, test_data = pd.read_csv(\"test_val.csv\")[['src', 'tgt']], pd.read_csv(\"test_data.csv\")[['src', 'tgt']]\n",
        "sp_en_bpe, sp_ben_bpe = spm.SentencePieceProcessor(), spm.SentencePieceProcessor()\n",
        "sp_en_bpe.load('eng_bpe.model'); sp_ben_bpe.load('ben_bpe.model');"
      ],
      "metadata": {
        "id": "IXsCg4mBBeoo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Data Preprocessing"
      ],
      "metadata": {
        "id": "auf4TlPFhNtn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We decide on some formats of the data and batches. We take the maximum tokens we can have per tokenized English and Bengali sentence to be `MAX_TOKENS = 60`. We use the vocabulary size of `32000`(i.e. `ENCODER_VOCAB = DECODER_VOCAB = 32000`). A batch size of `256` was choosen for our Mini-Batch Gradient Descent which is used to train our Model."
      ],
      "metadata": {
        "id": "3z7BUnn_G3WL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MAX_TOKENS = 60\n",
        "VOCAB_SIZE = 32000\n",
        "BATCH_SIZE = 256\n",
        "BUFFER_SIZE = BATCH_SIZE*4"
      ],
      "metadata": {
        "id": "7jTuQSqOAybO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The following two functions help in tokenizing an detokenizing the data based on our sentencepiece byte-pair encoding model."
      ],
      "metadata": {
        "id": "NlEW7GndbCfw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def tokenize(sentence, sp_model, add_bos = True, add_eos = True):\n",
        "    bos = [sp_model.bos_id()] if add_bos else []\n",
        "    eos = [sp_model.eos_id()] if add_eos else []\n",
        "    inputs = bos + sp_model.encode_as_ids(sentence) + eos\n",
        "    return np.reshape(np.array(inputs), [1, -1])\n",
        "def detokenize(tokenized, sp_model, is_bos = True, is_eos = True):\n",
        "    integers = np.squeeze(tokenized).tolist()\n",
        "    if is_eos:\n",
        "        return sp_model.DecodeIdsWithCheck(integers[int(is_bos):integers.index(sp_model.eos_id())])\n",
        "    else:\n",
        "        if sp_model.pad_id() in tokenized:\n",
        "            return sp_model.DecodeIdsWithCheck(integers[int(is_bos):integers.index(sp_model.pad_id())])\n",
        "        else:\n",
        "            return sp_model.DecodeIdsWithCheck(integers[int(is_bos):])"
      ],
      "metadata": {
        "id": "VC-6hav2BrJJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We have tokenized the data."
      ],
      "metadata": {
        "id": "5-76mwBnbuzp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Tokenization\n",
        "src_train_data_enc = [np.squeeze(tokenize(train_data['src'].iloc[i], sp_en_bpe)) for i in range(train_data.shape[0])]\n",
        "tgt_train_data_enc = [np.squeeze(tokenize(train_data['tgt'].iloc[i], sp_ben_bpe)) for i in range(train_data.shape[0])]\n",
        "src_train_dev_data_enc = [np.squeeze(tokenize(train_dev_data['src'].iloc[i], sp_en_bpe)) for i in range(train_dev_data.shape[0])]\n",
        "tgt_train_dev_data_enc = [np.squeeze(tokenize(train_dev_data['tgt'].iloc[i], sp_ben_bpe)) for i in range(train_dev_data.shape[0])]\n",
        "src_test_val_data_enc = [np.squeeze(tokenize(test_val_data['src'].iloc[i], sp_en_bpe)) for i in range(test_val_data.shape[0])]\n",
        "tgt_test_val_data_enc = [np.squeeze(tokenize(test_val_data['tgt'].iloc[i], sp_ben_bpe)) for i in range(test_val_data.shape[0])]"
      ],
      "metadata": {
        "id": "nT0QpepuDoFF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We have padded the data with post-truncation and post padding upto MAX_TOKENS number of tokens."
      ],
      "metadata": {
        "id": "wdyEjbm5hWXg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Padding\n",
        "# The reason why we add MAX_TOKENS+1 after-pad length for Bengali is that when we form the tf.Dataset later, we will shift the target sentence once and will truncate the last token once\n",
        "# Hence, it will get adjusted\n",
        "train_src = tf.keras.preprocessing.sequence.pad_sequences(src_train_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "train_tgt = tf.keras.preprocessing.sequence.pad_sequences(tgt_train_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "train_dev_src = tf.keras.preprocessing.sequence.pad_sequences(src_train_dev_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "train_dev_tgt = tf.keras.preprocessing.sequence.pad_sequences(tgt_train_dev_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "test_val_src = tf.keras.preprocessing.sequence.pad_sequences(src_test_val_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "test_val_tgt = tf.keras.preprocessing.sequence.pad_sequences(tgt_test_val_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')"
      ],
      "metadata": {
        "id": "NT63vC8PcK-2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below, we discard all those pairs where the tokenized source or target sentences are larger than MAX_TOKENS number of tokens."
      ],
      "metadata": {
        "id": "FvxtnfKniMRA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Removing all pairs whose either the target sentence or source setence exceeds MAX_TOKENS number of tokens\n",
        "same_idx = list(set([i for i in range(train_src.shape[0]) if train_src[i][-1] == 1 or train_src[i][-1] == 0]).intersection(set([i for i in range(train_tgt.shape[0]) if train_tgt[i][-1] == 1 or train_tgt[i][-1] == 0])))\n",
        "train_src = train_src[same_idx]\n",
        "train_tgt = train_tgt[same_idx]\n",
        "same_idx = list(set([i for i in range(train_dev_src.shape[0]) if train_dev_src[i][-1] == 1 or train_dev_src[i][-1] == 0]).intersection(set([i for i in range(train_dev_tgt.shape[0]) if train_dev_tgt[i][-1] == 1 or train_dev_tgt[i][-1] == 0])))\n",
        "train_dev_src = train_dev_src[same_idx]\n",
        "train_dev_tgt = train_dev_tgt[same_idx]\n",
        "same_idx = list(set([i for i in range(test_val_src.shape[0]) if test_val_src[i][-1] == 1 or test_val_src[i][-1] == 0]).intersection(set([i for i in range(test_val_tgt.shape[0]) if test_val_tgt[i][-1] == 1 or test_val_tgt[i][-1] == 0])))\n",
        "test_val_src = test_val_src[same_idx]\n",
        "test_val_tgt = test_val_tgt[same_idx]"
      ],
      "metadata": {
        "id": "L6-cV6zpcO5e"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_labels = np.zeros(train_tgt.shape)\n",
        "train_labels[:,0:train_tgt.shape[1] -1] = train_tgt[:,1:]\n",
        "train_dev_labels = np.zeros(train_dev_tgt.shape)\n",
        "train_dev_labels[:,0:train_dev_tgt.shape[1] -1] = train_dev_tgt[:,1:]\n",
        "test_val_labels = np.zeros(test_val_tgt.shape)\n",
        "test_val_labels[:,0:test_val_tgt.shape[1] -1] = test_val_tgt[:,1:]"
      ],
      "metadata": {
        "id": "JGMvPvIxOx-0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"*** Preprocessed Data ***\")\n",
        "print(\"Train Data Shape(Source): \", train_src.shape)\n",
        "print(\"Train Data Shape(Target): \", train_tgt.shape)\n",
        "print(\"Train Labels Shape(Target): \", train_labels.shape)\n",
        "print(\"Train Dev Data Shape(Source): \", train_dev_src.shape)\n",
        "print(\"Train Dev Data Shape(Target): \", train_dev_tgt.shape)\n",
        "print(\"Train Dev Labels Shape(Target): \", train_dev_labels.shape)\n",
        "print(\"Test Val Data Shape(Source): \", test_val_src.shape)\n",
        "print(\"Test Val Data Shape(Target): \", test_val_tgt.shape)\n",
        "print(\"Test Labels Shape(Target): \", test_val_labels.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rO5HD33XOGxf",
        "outputId": "5ef7538d-5d8c-4bc2-9d2c-4be89f10b5f0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "*** Preprocessed Data ***\n",
            "Train Data Shape(Source):  (154836, 60)\n",
            "Train Data Shape(Target):  (154836, 60)\n",
            "Train Labels Shape(Target):  (154836, 60)\n",
            "Train Dev Data Shape(Source):  (3649, 60)\n",
            "Train Dev Data Shape(Target):  (3649, 60)\n",
            "Train Dev Labels Shape(Target):  (3649, 60)\n",
            "Test Val Data Shape(Source):  (1194, 60)\n",
            "Test Val Data Shape(Target):  (1194, 60)\n",
            "Test Labels Shape(Target):  (1194, 60)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below we prepare the data in the form ready to feed to the Model."
      ],
      "metadata": {
        "id": "xtsMxbAYzePz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "train_data_prep = tf.data.Dataset.from_tensor_slices((train_src, train_tgt, train_labels))\n",
        "train_dev_data_prep = tf.data.Dataset.from_tensor_slices((train_dev_src, train_dev_tgt, train_dev_labels))\n",
        "test_val_data_prep = tf.data.Dataset.from_tensor_slices((test_val_src, test_val_tgt, test_val_labels))"
      ],
      "metadata": {
        "id": "_OkR5tbtyBQj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below, we can see how our data looks"
      ],
      "metadata": {
        "id": "jSGBKDhc2sc-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for encoder_inputs, decoder_inputs, labels in train_data_prep.take(1):\n",
        "    print(f'encoder_inputs.shape: {encoder_inputs.shape}')\n",
        "    print(f'decoder_inputs.shape: {decoder_inputs.shape}')\n",
        "    print(f\"labels.shape: {labels.shape}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ao1TX6GHzWn7",
        "outputId": "de4de7dd-a2e0-41f9-cf88-2343d1c136dd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "encoder_inputs.shape: (60,)\n",
            "decoder_inputs.shape: (60,)\n",
            "labels.shape: (60,)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Model"
      ],
      "metadata": {
        "id": "zjXk6pt-28kL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will use the Transformer Architechture with the following hyperparameters."
      ],
      "metadata": {
        "id": "a9UFHuFO3dHe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Transformer parameters\n",
        "d_model = 256 # 512 in the original paper\n",
        "d_k = 32 # 64 in the original paper\n",
        "d_v = 32 # 64 in the original paper\n",
        "n_heads = 8 # 8 in the original paper\n",
        "n_encoder_layers = 1 # 6 in the original paper\n",
        "n_decoder_layers = 1 # 6 in the original paper\n",
        "\n",
        "max_token_length = MAX_TOKENS # 512 in the original paper"
      ],
      "metadata": {
        "id": "v9ULcpgbJ9WP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class SingleHeadAttention(Layer):\n",
        "  def __init__(self, input_shape=(3, -1, d_model), dropout=.0, masked=None):\n",
        "    super(SingleHeadAttention, self).__init__()\n",
        "    self.q = Dense(d_k, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.normalize_q = Lambda(lambda x: x / np.sqrt(d_k))\n",
        "    self.k = Dense(d_k, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.v = Dense(d_v, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.dropout = dropout\n",
        "    self.masked = masked\n",
        "  \n",
        "  # Inputs: [query, key, value]\n",
        "  def call(self, inputs, training=None):\n",
        "    assert len(inputs) == 3\n",
        "    # We use a lambda layer to divide vector q by sqrt(d_k) according to the equation\n",
        "    q = self.normalize_q(self.q(inputs[0]))\n",
        "    k = self.k(inputs[1])\n",
        "    # The dimensionality of q is (batch_size, query_length, d_k) and that of k is (batch_size, key_length, d_k)\n",
        "    # So we will do a matrix multication by batch after transposing last 2 dimensions of k\n",
        "    # tf.shape(attn_weights) = (batch_size, query_length, key_length)\n",
        "    attn_weights = tf.matmul(q, tf.transpose(k, perm=[0,2,1]))\n",
        "    if self.masked: # Prevent future attentions in decoding self-attention\n",
        "      # Create a matrix where the strict upper triangle (not including main diagonal) is filled with -inf and 0 elsewhere\n",
        "      length = tf.shape(attn_weights)[-1]\n",
        "      #attn_mask = np.triu(tf.fill((length, length), -np.inf), k=1) # We need to use tensorflow functions instead of numpy\n",
        "      attn_mask = tf.fill((length, length), -np.inf)\n",
        "      attn_mask = tf.linalg.band_part(attn_mask, 0, -1) # Get upper triangle\n",
        "      attn_mask = tf.linalg.set_diag(attn_mask, tf.zeros((length))) # Set diagonal to zeros to avoid operations with infinity\n",
        "      # This matrix is added to the attention weights so all future attention will have -inf logits (0 after softmax)\n",
        "      attn_weights += attn_mask\n",
        "    # Softmax along the last dimension\n",
        "    attn_weights = tf.nn.softmax(attn_weights, axis=-1)\n",
        "    if training: # Attention dropout included in the original paper. This is possibly to encourage multihead diversity.\n",
        "      attn_weights = tf.nn.dropout(attn_weights, rate=self.dropout)\n",
        "    v = self.v(inputs[2])\n",
        "    return tf.matmul(attn_weights, v)\n",
        "\n",
        "class MultiHeadAttention_my(Layer):\n",
        "  def __init__(self, dropout=.0, masked=None):\n",
        "    super(MultiHeadAttention_my, self).__init__()\n",
        "    self.attn_heads = list()\n",
        "    for i in range(n_heads): \n",
        "      self.attn_heads.append(SingleHeadAttention(dropout=dropout, masked=masked))\n",
        "    self.linear = Dense(d_model, input_shape=(-1, n_heads * d_v), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    \n",
        "  def call(self, x, training=None):\n",
        "    attentions = [self.attn_heads[i](x, training=training) for i in range(n_heads)]\n",
        "    concatenated_attentions = tf.concat(attentions, axis=-1)\n",
        "    return self.linear(concatenated_attentions)\n",
        "\n",
        "class TransformerEncoder(Layer):\n",
        "  def __init__(self, dropout=.1, attention_dropout=.0, **kwargs):\n",
        "    super(TransformerEncoder, self).__init__(**kwargs)\n",
        "    self.dropout_rate = dropout\n",
        "    self.attention_dropout_rate = attention_dropout\n",
        "  def build(self, input_shape):\n",
        "    self.multihead_attention = MultiHeadAttention_my(dropout=self.attention_dropout_rate)\n",
        "    self.dropout1 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization1 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.linear1 = Dense(input_shape[-1] * 4, input_shape=input_shape, activation='relu',\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.linear2 = Dense(input_shape[-1], input_shape=self.linear1.compute_output_shape(input_shape),\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.dropout2 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization2 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    super(TransformerEncoder, self).build(input_shape)\n",
        "  def call(self, x, training=None):\n",
        "    sublayer1 = self.multihead_attention((x, x, x), training=training)\n",
        "    sublayer1 = self.dropout1(sublayer1, training=training)\n",
        "    layernorm1 = self.layer_normalization1(x + sublayer1)\n",
        "    \n",
        "    sublayer2 = self.linear2(self.linear1(layernorm1))\n",
        "    sublayer1 = self.dropout2(sublayer2, training=training)\n",
        "    layernorm2 = self.layer_normalization2(layernorm1 + sublayer2)\n",
        "    return layernorm2\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape\n",
        "\n",
        "class TransformerDecoder(Layer):\n",
        "  def __init__(self, dropout=.0, attention_dropout=.0, **kwargs):\n",
        "    super(TransformerDecoder, self).__init__(**kwargs)\n",
        "    self.dropout_rate = dropout\n",
        "    self.attention_dropout_rate = attention_dropout\n",
        "  def build(self, input_shape):\n",
        "    self.multihead_self_attention = MultiHeadAttention_my(dropout=self.attention_dropout_rate, masked=True)\n",
        "    self.dropout1 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization1 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.multihead_encoder_attention = MultiHeadAttention_my(dropout=self.attention_dropout_rate)\n",
        "    self.dropout2 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization2 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.linear1 = Dense(input_shape[-1] * 4, input_shape=input_shape, activation='relu',\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.linear2 = Dense(input_shape[-1], input_shape=self.linear1.compute_output_shape(input_shape),\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.dropout3 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization3 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    super(TransformerDecoder, self).build(input_shape)\n",
        "  def call(self, x, hidden, training=None):\n",
        "    sublayer1 = self.multihead_self_attention((x, x, x))\n",
        "    sublayer1 = self.dropout1(sublayer1, training=training)\n",
        "    layernorm1 = self.layer_normalization1(x + sublayer1)\n",
        "    \n",
        "    sublayer2 = self.multihead_encoder_attention((x, hidden, hidden))\n",
        "    sublayer2 = self.dropout2(sublayer2, training=training)\n",
        "    layernorm2 = self.layer_normalization2(layernorm1 + sublayer2)\n",
        "    \n",
        "    sublayer3 = self.linear2(self.linear1(layernorm1))\n",
        "    sublayer3 = self.dropout3(sublayer3, training=training)\n",
        "    layernorm3 = self.layer_normalization2(layernorm2 + sublayer3)\n",
        "    return layernorm3\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape\n",
        "\n",
        "class SinusoidalPositionalEncoding(Layer): # This is a TensorFlow implementation of https://github.com/graykode/nlp-tutorial/blob/master/5-1.Transformer/Transformer_Torch.ipynb\n",
        "  def __init__(self):\n",
        "    super(SinusoidalPositionalEncoding, self).__init__()\n",
        "    self.sinusoidal_encoding = np.array([self.get_positional_angle(pos) for pos in range(max_token_length)], dtype=np.float32)\n",
        "    self.sinusoidal_encoding[:, 0::2] = np.sin(self.sinusoidal_encoding[:, 0::2])\n",
        "    self.sinusoidal_encoding[:, 1::2] = np.cos(self.sinusoidal_encoding[:, 1::2])\n",
        "    self.sinusoidal_encoding = tf.cast(self.sinusoidal_encoding, dtype=tf.float32) # Casting the array to Tensor for slicing\n",
        "  def call(self, x):\n",
        "    return x + self.sinusoidal_encoding[:tf.shape(x)[1]]\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape\n",
        "  def get_angle(self, pos, dim):\n",
        "    return pos / np.power(10000, 2 * (dim // 2) / d_model)\n",
        "  def get_positional_angle(self, pos):\n",
        "    return [self.get_angle(pos, dim) for dim in range(d_model)]\n",
        "\n",
        "class Transformer(Model):\n",
        "  def __init__(self, dropout=.1, attention_dropout=.0, **kwargs):\n",
        "    super(Transformer, self).__init__(**kwargs)\n",
        "    self.encoding_embedding = Embedding(VOCAB_SIZE, d_model)\n",
        "    self.decoding_embedding = Embedding(VOCAB_SIZE, d_model)\n",
        "    self.pos_encoding = SinusoidalPositionalEncoding()\n",
        "    self.encoder = [TransformerEncoder(dropout=dropout, attention_dropout=attention_dropout) for i in range(n_encoder_layers)]\n",
        "    self.decoder = [TransformerDecoder(dropout=dropout, attention_dropout=attention_dropout) for i in range(n_decoder_layers)]\n",
        "    self.decoder_final = Dense(VOCAB_SIZE, input_shape=(None, d_model))\n",
        "  def call(self, inputs, training=None): # Source_sentence and decoder_input\n",
        "    source_sentence, decoder_input = inputs\n",
        "    embedded_source = self.encoding_embedding(source_sentence)\n",
        "    encoder_output = self.pos_encoding(embedded_source)\n",
        "    for encoder_unit in self.encoder:\n",
        "      encoder_output = encoder_unit(encoder_output, training=training)\n",
        "    \n",
        "    embedded_target = self.decoding_embedding(decoder_input)\n",
        "    decoder_output = self.pos_encoding(embedded_target)\n",
        "    for decoder_unit in self.decoder:\n",
        "      decoder_output = decoder_unit(decoder_output, encoder_output, training=training)\n",
        "    if training:\n",
        "      decoder_output = self.decoder_final(decoder_output)\n",
        "      decoder_output = tf.nn.softmax(decoder_output, axis=-1)\n",
        "    else:\n",
        "      decoder_output = self.decoder_final(decoder_output[:, -1:, :])\n",
        "      decoder_output = tf.nn.softmax(decoder_output, axis=-1)\n",
        "    return decoder_output\n",
        "  def build_graph(self):\n",
        "    input1 = Input(shape = (60,))\n",
        "    input2 = Input(shape = (60,))\n",
        "    return Model(inputs=[input1, input2], outputs=self.call((input1, input2), True))"
      ],
      "metadata": {
        "id": "KvsIOzPnzt4F"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "transformer = Transformer(dropout=0.3) # Instantiating a new transformer model\n",
        "train = [tf.cast(train_src, dtype=tf.float32), tf.cast(train_tgt, dtype=tf.float32)] # Cast the tuples to tensors\n",
        "validation = [tf.cast(train_dev_src, dtype=tf.float32), tf.cast(train_dev_tgt, dtype=tf.float32)]"
      ],
      "metadata": {
        "id": "eRsGRYTR6XQW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "transformer.build([train_src.shape, train_tgt.shape])\n",
        "transformer.summary()"
      ],
      "metadata": {
        "id": "IHXUpv-0eh-b"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tf.keras.utils.plot_model(transformer.build_graph(), to_file=\"model-1L-8H.png\", expand_nested=True, show_shapes=True)"
      ],
      "metadata": {
        "id": "ps0of4pGT9C1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training"
      ],
      "metadata": {
        "id": "0JkChx5Ndp1s"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "transformer.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])"
      ],
      "metadata": {
        "id": "InXkWf-LawJi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "EPOCHS = 10\n",
        "checkpoint_filepath = './Models/'\n",
        "model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n",
        "    filepath=checkpoint_filepath,\n",
        "    save_weights_only=True,\n",
        "    monitor = \"accuracy\",\n",
        "    mode='max',\n",
        "    save_best_only=True)"
      ],
      "metadata": {
        "id": "LMttctIwaNxG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "with tf.device('/device:GPU:0'):\n",
        "    history = transformer.fit(train, tf.cast(train_labels, dtype=tf.float32), batch_size=256, epochs=EPOCHS, callbacks=[model_checkpoint_callback])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bzdsT8kmBEMo",
        "outputId": "90244a89-d827-4b8a-a277-85f70a74eb0e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "605/605 [==============================] - 335s 534ms/step - loss: 1.8682 - accuracy: 0.7975\n",
            "Epoch 2/10\n",
            "605/605 [==============================] - 327s 541ms/step - loss: 1.4482 - accuracy: 0.8136\n",
            "Epoch 3/10\n",
            "605/605 [==============================] - 328s 542ms/step - loss: 1.2953 - accuracy: 0.8212\n",
            "Epoch 4/10\n",
            "605/605 [==============================] - 328s 543ms/step - loss: 1.1797 - accuracy: 0.8274\n",
            "Epoch 5/10\n",
            "605/605 [==============================] - 329s 543ms/step - loss: 1.0881 - accuracy: 0.8331\n",
            "Epoch 6/10\n",
            "605/605 [==============================] - 329s 543ms/step - loss: 1.0119 - accuracy: 0.8385\n",
            "Epoch 7/10\n",
            "605/605 [==============================] - 330s 546ms/step - loss: 0.9458 - accuracy: 0.8438\n",
            "Epoch 8/10\n",
            "605/605 [==============================] - 330s 545ms/step - loss: 0.8881 - accuracy: 0.8488\n",
            "Epoch 9/10\n",
            "605/605 [==============================] - 330s 545ms/step - loss: 0.8374 - accuracy: 0.8537\n",
            "Epoch 10/10\n",
            "605/605 [==============================] - 330s 545ms/step - loss: 0.7921 - accuracy: 0.8583\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "transformer.load_weights(\"model_weights-1L-8H\")\n",
        "#transformer.save_weights(\"model_weights-1L-8H\")\n",
        "# transformer.save(\"./Models/\")"
      ],
      "metadata": {
        "id": "6tmeafiFdZ_U",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "426d0b69-ef6c-4816-c3b4-94a0d663c672"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x7f1e91bef3d0>"
            ]
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(10, 10))\n",
        "plt.plot(history.history['accuracy'])\n",
        "plt.title('model accuracy')\n",
        "plt.ylabel('accuracy')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train'], loc='upper left')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "JbSCoj0cft6O",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 621
        },
        "outputId": "4e4588cb-7545-4845-e819-3625f79cd541"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 720x720 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmcAAAJcCAYAAAC8DwN/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3hUdfr+8fshBEIg9CK99w4hYF0RcVkUG66KBZGmoqDYV7Huupbvrm0VC0qRDoqCig0XXQsKCQkQeicBhIQSSCD98/sj4/4iCxIgkzMzeb+ui2tn5szJ3HExuf2cc55jzjkBAAAgMJTxOgAAAAD+P8oZAABAAKGcAQAABBDKGQAAQAChnAEAAAQQyhkAAEAAoZwBCHpmNtnM/lbE924zs4v9nQkAThflDAAAIIBQzgAgQJhZWa8zAPAe5QxAifAdTnzAzFaaWYaZvWtmdczsMzM7bGaLzKxaofdfbmarzeygmX1jZm0LbetqZst9+82WFHHMZ11mZgm+fX80s05FzHipmcWb2SEzSzKzJ4/Zfp7v6x30bR/ie72Cmf3TzLabWZqZfe977UIzSz7OP4eLfY+fNLP3zWyamR2SNMTMYsxsie8zdpvZa2ZWrtD+7c3sKzPbb2Z7zOwRMzvLzI6YWY1C7+tmZilmFl6U7x1A4KCcAShJAyX1ldRK0gBJn0l6RFItFfw8GiNJZtZK0kxJ9/i2LZT0sZmV8xWVjyRNlVRd0lzf15Vv366SJkq6TVINSW9JWmBm5YuQL0PSYElVJV0q6Q4zu9L3dRv78v7Ll6mLpATffv+Q1F3SOb5MD0rKL+I/kyskve/7zOmS8iSNlVRT0tmS+kga5csQJWmRpM8l1ZPUQtLXzrlfJH0j6dpCX/dmSbOcczlFzAEgQFDOAJSkfznn9jjndkr6TtLPzrl451ympA8ldfW97zpJnzrnvvKVi39IqqCC8tNLUrikl51zOc659yUtK/QZIyW95Zz72TmX55ybIinLt9/vcs5945xb5ZzLd86tVEFB/INv8w2SFjnnZvo+d59zLsHMykgaKulu59xO32f+6JzLKuI/kyXOuY98n3nUORfnnPvJOZfrnNumgnL5a4bLJP3inPuncy7TOXfYOfezb9sUSTdJkpmFSRqkggILIMhQzgCUpD2FHh89zvNKvsf1JG3/dYNzLl9SkqT6vm07nXOu0L7bCz1uLOk+32HBg2Z2UFJD336/y8x6mtli3+HANEm3q2AFS76vsfk4u9VUwWHV420riqRjMrQys0/M7Bffoc6/FyGDJM2X1M7MmqpgdTLNObf0NDMB8BDlDEAg2qWCkiVJMjNTQTHZKWm3pPq+137VqNDjJEnPOOeqFvoT6ZybWYTPnSFpgaSGzrkqkt6U9OvnJElqfpx9UiVlnmBbhqTIQt9HmAoOiRbmjnn+hqR1klo65yqr4LBv4QzNjhfct/o4RwWrZzeLVTMgaFHOAASiOZIuNbM+vhPa71PBockfJS2RlCtpjJmFm9nVkmIK7TtB0u2+VTAzs4q+E/2jivC5UZL2O+cyzSxGBYcyfzVd0sVmdq2ZlTWzGmbWxbeqN1HSi2ZWz8zCzOxs3zluGyRF+D4/XNI4SSc79y1K0iFJ6WbWRtIdhbZ9Iqmumd1jZuXNLMrMehba/p6kIZIuF+UMCFqUMwABxzm3XgUrQP9SwcrUAEkDnHPZzrlsSVeroITsV8H5afMK7RsraYSk1yQdkLTJ996iGCXpaTM7LOlxFZTEX7/uDkn9VVAU96vgYoDOvs33S1qlgnPf9kt6XlIZ51ya72u+o4JVvwxJv7l68zjuV0EpPKyCojm7UIbDKjhkOUDSL5I2SupdaPsPKrgQYblzrvChXgBBxH572gYAIJiZ2b8lzXDOveN1FgCnh3IGACHCzHpI+koF58wd9joPgNPDYU0ACAFmNkUFM9DuoZgBwY2VMwAAgADCyhkAAEAACZmb7NasWdM1adLE6xgAAAAnFRcXl+qcO3buoaQQKmdNmjRRbGys1zEAAABOysxOOO6Gw5oAAAABhHIGAAAQQChnAAAAASRkzjk7npycHCUnJyszM9PrKH4XERGhBg0aKDw83OsoAADgDIR0OUtOTlZUVJSaNGkiM/M6jt8457Rv3z4lJyeradOmXscBAABnIKQPa2ZmZqpGjRohXcwkycxUo0aNUrFCCABAqPNrOTOzfma23sw2mdnDx9neyMwWm1m8ma00s/6FtnUysyVmttrMVplZxGlmOJNvIWiUlu8TAIBQ57fDmmYWJul1SX0lJUtaZmYLnHNrCr1tnKQ5zrk3zKydpIWSmphZWUnTJN3snFthZjUk5fgrKwAAQKDw58pZjKRNzrktzrlsSbMkXXHMe5ykyr7HVSTt8j2+RNJK59wKSXLO7XPO5fkxq98cPHhQ48ePP+X9+vfvr4MHD/ohEQAACGT+LGf1JSUVep7se62wJyXdZGbJKlg1G+17vZUkZ2ZfmNlyM3vweB9gZiPNLNbMYlNSUoo3fTE5UTnLzc393f0WLlyoqlWr+isWAAAIUF5fEDBI0mTnXANJ/SVNNbMyKjjcep6kG33/e5WZ9Tl2Z+fc2865aOdcdK1ax709lecefvhhbd68WV26dFGPHj10/vnn6/LLL1e7du0kSVdeeaW6d++u9u3b6+233/7vfk2aNFFqaqq2bdumtm3basSIEWrfvr0uueQSHT161KtvBwAA+Jk/R2nslNSw0PMGvtcKGyapnyQ555b4TvqvqYJVtv8451IlycwWSuom6evTDfPUx6u1Zteh0939uNrVq6wnBrT/3fc899xzSkxMVEJCgr755htdeumlSkxM/O/Ii4kTJ6p69eo6evSoevTooYEDB6pGjRq/+RobN27UzJkzNWHCBF177bX64IMPdNNNNxXr9wIAAAKDP1fOlklqaWZNzaycpOslLTjmPTsk9ZEkM2srKUJSiqQvJHU0s0jfxQF/kLRGISAmJuY3s8heffVVde7cWb169VJSUpI2btz4P/s0bdpUXbp0kSR1795d27ZtK6m4AACghPlt5cw5l2tmd6mgaIVJmuicW21mT0uKdc4tkHSfpAlmNlYFFwcMcc45SQfM7EUVFDwnaaFz7tMzyXOyFa6SUrFixf8+/uabb7Ro0SItWbJEkZGRuvDCC487q6x8+fL/fRwWFsZhTQAAQphf7xDgnFuoghP9C7/2eKHHaySde4J9p6lgnEZQi4qK0uHDh4+7LS0tTdWqVVNkZKTWrVunn376qYTTAQCAQBPSt28KBDVq1NC5556rDh06qEKFCqpTp85/t/Xr109vvvmm2rZtq9atW6tXr14eJgUAAIHACo4iBr/o6GgXGxv7m9fWrl2rtm3bepSo5JW27xcAgGBlZnHOuejjbfN6lAYAAAAKoZwBAAAEkJAvZ6Fy2PZkSsv3CQBAqAvpchYREaF9+/aFfHFxzmnfvn2KiIjwOgoAADhDIX21ZoMGDZScnKxAve9mcYqIiFCDBg28jgEAQFBzzik33yk8zLv1q5AuZ+Hh4b+Zxg8AAHAiew9n6tEPE1U7qryeuaqjZzlC+rAmAADAyTjnND9hpy556T/6dkOKmtas6OkpUSG9cgYAAPB7Ug5nadxHq/TF6j3q0rCq/vHnzmpRu5KnmShnAACgVPpk5S499lGiMrLz9PCf2mjE+c0UVsa8jkU5AwAApcu+9Cw9Pn+1Pl21W50bVNE//txZLetEeR3rvyhnAACg1Phs1W6N+yhRhzNz9WC/1hp5fjOV9fDKzOOhnAEAgJC3PyNbj89P1Ccrd6tj/YLVstZnBc5qWWGUMwAAENI+T/xF4z5apbSjObqvbyvdfmFzT+eYnQzlDAAAhKSDR7L15ILV+ihhl9rVraypw3qqbd3KXsc6KcoZAAAIOV+t2aNHPlylAxnZGntxK43qHdirZYVRzgAAQMhIO5Kjpz5erXnxO9XmrChNvrWH2ter4nWsU0I5AwAAIeHf6/boL/NWKTU9W2MuaqG7LmqpcmWDY7WsMMoZAAAIamlHc/S3T9ZoblyyWteJ0juDe6hjg+BaLSuMcgYAAILWN+v36uEPViklPUt39W6h0X1aqHzZMK9jnRHKGQAACDqHMnP0zCdrNTs2SS1rV9JbN3dX54ZVvY5VLChnAAAgqHy3MUUPvb9SvxzK1B0XNtfdfVoqIjy4V8sKo5wBAICgkJ6Vq2c+XauZS3eoea2K+uCOc9S1UTWvYxU7yhkAAAh4P2xK1YPvr9SutKO67YJmGtu3VUitlhVGOQMAAAErIytXz322TlN/2q5mNSvq/dvPVvfG1b2O5VeUMwAAEJCWbN6nBz9YoeQDRzX8vKa6/4+tQ3a1rDDKGQAACChHsnP1/GfrNGXJdjWpEak5t52tHk1Ce7WsMMoZAAAIGD9v2acH3l+pHfuP6NZzm+jBP7ZRhXKhv1pWGOUMAAB47mh2nl74Yp0m/7hNDatFatbIXurVrIbXsTxBOQMAAJ6K3bZfD7y/UltTM3TL2Y310J/aKLJc6a0opfc7BwAAnsrMydM/vlivd3/YqvpVK2jGiJ46p3lNr2N5jnIGAABKXNz2A3pg7gptSc3QTb0a6S9/aquK5aklEuUMAACUoMycPL301QZN+G6L6lapoOnDe+rcFqyWFUY5AwAAJSIh6aDum5OgzSkZGhTTSI/0b6OoiHCvYwUcyhkAAPCrrNw8vbxoo976drPOqhyh94bG6IJWtbyOFbAoZwAAwG9WJh/U/XNXaMOedF0X3VCPXtZWlVkt+12UMwAAUOyycvP0r6836Y1vN6tWpfKadGsP9W5d2+tYQYFyBgAAilXizjTdP3eF1v1yWNd0b6DHLmunKhVYLSsqyhkAACgW2bn5em3xJo1fvEnVK5bTxCHRuqhNHa9jBR3KGQAAOGNrdh3SfXNXaO3uQ7q6a309MaC9qkSyWnY6KGcAAOC05eTla/zizfrXvzeqWsVymjA4Wn3bsVp2JihnAADgtKz75ZDum7NCq3cd0hVd6unJAe1VrWI5r2MFPcoZAAA4Jbl5+Xrz28165euNqlIhXG/e1F39OpzldayQQTkDAABFtmHPYd0/d4VWJqdpQOd6eury9qrOalmxopwBAICTys93evf7rfq/L9arUkRZjb+xm/p3rOt1rJBEOQMAAL8r+cAR3T93hX7asl+XtKujv1/dUTUrlfc6VsiinAEAgONyzunD+J16Yv5qOUn/d00nXdO9gczM62ghjXIGAAD+x4GMbI37KFGfrtqtHk2q6cVru6hh9UivY5UKlDMAAPAb325I0QNzV+jAkWw91K+NRl7QTGFlWC0rKZQzAAAgSTqanafnPlurKUu2q1WdSpp0aw+1r1fF61ilDuUMAABoZfJB3TM7QVtSMjTsvKZ64I+tFREe5nWsUolyBgBAKZabl6/x32zWq19vVK2o8po+vKfObVHT61ilGuUMAIBSaltqhsbOSVD8joO6sks9PXVFB1WpwM3KvUY5AwCglHHOaebSJP31kzUKDzP9a1BXDehcz+tY8KGcAQBQiqQcztLDH6zU1+v26rwWNfV/f+6kulUqeB0LhVDOAAAoJb5Y/Yv+Mm+VMrJy9eSAdhp8dhOVYURGwKGcAQAQ4tKzcvX0x6s1JzZZ7etV1svXdVHLOlFex8IJUM4AAAhhsdv2a+ycBO08cFR39W6hMX1aqlzZMl7Hwu+gnAEAEIKyc/P18qINevPbzWpQLVJzbjtb0U2qex0LRUA5AwAgxGzYc1j3zErQmt2HdH2Phhp3WTtVKs+v/GDB/1MAAISI/HynyT9u03Ofr1NU+bKaMDhafdvV8ToWThHlDACAELA77ajun7tCP2zapz5tauu5gZ1UK6q817FwGihnAAAEufkJO/XYR4nKzXd67uqOuq5HQ5kxIiNYUc4AAAhSaUdy9Nj8RC1YsUtdG1XVS9d2UZOaFb2OhTNEOQMAIAj9sClV981ZodT0LN1/SSvd/ofmKhvGiIxQQDkDACCIZObk6YXP12viD1vVvFZFTRh8rjo2qOJ1LBQjyhkAAEEicWeaxs5O0Ma96RpyThM91K+NKpQL8zoWihnlDACAAJeX7/Tmt5v18qINql6xnN4bGqMLWtXyOhb8hHIGAEAA27HviO6dk6DY7Qd0ace6euaqDqoaWc7rWPAjyhkAAAHIOae5scl66uPVKlPG9PJ1XXRFl3qMyCgFKGcAAASYfelZ+su8VfpyzR6d3ayG/nFtZ9WvWsHrWCghlDMAAALI12v36KEPVurQ0VyNu7Sthp7bVGXKsFpWmvh1IIqZ9TOz9Wa2ycwePs72Rma22MzizWylmfX3vd7EzI6aWYLvz5v+zAkAgNcysnL1l3mrNGxKrGpWKq8Fo8/V8PObUcxKIb+tnJlZmKTXJfWVlCxpmZktcM6tKfS2cZLmOOfeMLN2khZKauLbttk518Vf+QAACBTLdxzQvbMTtH3/Ed32h2a6t28rlS/LiIzSyp+HNWMkbXLObZEkM5sl6QpJhcuZk1TZ97iKpF1+zAMAQEDJycvXv77eqNcWb1LdKhU0a0Qv9WxWw+tY8Jg/y1l9SUmFnidL6nnMe56U9KWZjZZUUdLFhbY1NbN4SYckjXPOfXfsB5jZSEkjJalRo0bFlxwAAD/bnJKusbMTtDI5TQO7NdCTl7dTVES417EQALy+IGCQpMnOuX+a2dmSpppZB0m7JTVyzu0zs+6SPjKz9s65Q4V3ds69LeltSYqOjnYlHR4AgFPlnNPUn7br7wvXqkJ4mN64sZv+1LGu17EQQPxZznZKaljoeQPfa4UNk9RPkpxzS8wsQlJN59xeSVm+1+PMbLOkVpJi/ZgXAAC/2nMoUw+8v1L/2ZCiC1vX0gsDO6l25QivYyHA+LOcLZPU0syaqqCUXS/phmPes0NSH0mTzaytpAhJKWZWS9J+51yemTWT1FLSFj9mBQDArxau2q1HPlylzJw8/fXKDrqpZyMGyuK4/FbOnHO5ZnaXpC8khUma6JxbbWZPS4p1zi2QdJ+kCWY2VgUXBwxxzjkzu0DS02aWIylf0u3Ouf3+ygoAgL8cyszRk/NXa178TnVuUEUvXddFzWpV8joWApg5FxqnakVHR7vYWI56AgACx09b9um+OSv0y6FM3dW7he66qIXCw/w6YhRBwszinHPRx9vm9QUBAACEnKzcPP3zyw2a8N0WNalRUe/ffra6NqrmdSwECcoZAADFaO3uQxo7O0HrfjmsG3s20qOXtlVkOX7douj42wIAQDHIz3ea+MNWvfD5elWuEK5JQ3qod5vaXsdCEKKcAQBwhlIOZ+n+uSv07YYU9W1XR88P7KTqFct5HQtBinIGAMAZ+G5jisbOXqFDmTmMyECxoJwBAHAacvLy9Y8v1+utb7eoZe1KmjY8Rm3OqnzyHYGToJwBAHCKtu/L0JhZCVqRdFA39Gykxy5tpwrlwryOhRBBOQMA4BTMT9ipRz9MVBkT98WEX1DOAAAogoysXD0+f7U+WJ6sHk2q6eXru6p+1Qpex0IIopwBAHASiTvTNHpmfMHhzD4tNeaiFirLpH/4CeUMAIATcM7p3e+36vnP16lGxfKaMaKXejWr4XUshDjKGQAAx5GanqUH5q7Q4vUFs8teGNhJ1ZhdhhJAOQMA4Bjfb0zV2DkJSjuao79e0V439WrM7DKUGMoZAAA+OXn5+ueXG/TWfzarRa1KmjqM2WUoeZQzAAAk7dh3RKNnxWtF0kENimmkxy9jdhm8QTkDAJR6hWeXjb+xm/ozuwweopwBAEqtjKxcPblgtebGJSu6cTW9fH0XNagW6XUslHKUMwBAqZS4M01jZsZr674Mjbmohcb0acnsMgQEyhkAoFRxzmniD9v0/GfrVL1iOc0Y3ktnN2d2GQIH5QwAUGrsS8/S/cwuQ4CjnAEASoUfNqXqntkFs8uevqK9bmZ2GQIU5QwAENJy8vL14lcb9Oa3m9W8ViW9NzRGbesyuwyBi3IGAAhZSfuPaPTMeCUkHdSgmIZ67LJ2iizHrz4ENv6GAgBC0oIVu/TovFWSSa/f0E2XdmJ2GYID5QwAEFKOZOfqifkFs8u6N66mV5hdhiBDOQMAhIzEnWkaMyteW1MzNPqiFrqb2WUIQpQzAEDQc85p0g/b9Nxn61StYrimD++pc5rX9DoWcFooZwCAoLYvPUsPvL9S/163Vxe3ra0Xrums6swuQxCjnAEAgtaPvtllB4/m6KnL22vw2cwuQ/CjnAEAgk5OXr5e+mqD3vh2s5rVrKjJt8aoXT1mlyE0UM4AAEElaf8RjZkVr/gdB3V9j4Z6fACzyxBa+NsMAAgaH6/YpUd8s8teu6GrLutUz+tIQLGjnAEAAt6R7Fw9uWC15sQmq2ujqnr1+q5qWJ3ZZQhNlDMAQEBbvStNo2cWzC67q3cL3X1xS4UzuwwhjHIGAAhIzjlN/nGbnl3I7DKULpQzAEDA2Z+RrQfmrtDXzC5DKUQ5AwAElP/OLjuSoycHtNMt5zRhdhlKFcoZACAg5OTl6+VFGzT+m81qWrOiJt3aQ+3rVfE6FlDiKGcAAM8Vnl12XXRDPXE5s8tQevE3HwDgqU9W7tJf5q2SnPSvQV01oDOzy1C6Uc4AAJ44kp2rpz9eo1nLkphdBhRCOQMAlLg1uw5p9Mzl2pKaoTt7N9c9F7didhngQzkDAJQY55ym/LhNf1+4TlUjwzV9WE+d04LZZUBhlDMAQIk4eCRb989dqUVr9+iiNrX1f9d0Uo1K5b2OBQQcyhkAwO9WJB3UqOnLtfdwph6/rJ1uPZfZZcCJUM4AAH7jnNPUn7brb5+sVa2o8ppz29nq2qia17GAgEY5AwD4RXpWrh7+YKU+WblbF7WprRev7ayqkdyCCTgZyhkAoNit++WQRk1brm37MvRgv9a6/YLmKlOGw5hAUVDOAADFam5skh6bn6jKEeGaMaKXejWr4XUkIKhQzgAAxeJodp4en5+ouXHJOqd5Db1yfVfViuJqTOBUUc4AAGdsS0q6Rk1frvV7DmvMRS1098WtFMZhTOC0UM4AAGfkk5W79ND7K1WubBlNGtJDF7au7XUkIKhRzgAApyUrN09//3StpizZrm6Nquq1G7qpXtUKXscCgh7lDABwypL2H9FdM5ZrRXKahp/XVA/9qQ33xgSKCeUMAHBKvl67R/fOWaF85/TmTd3Vr8NZXkcCQgrlDABQJLl5+fq/L9frrW+3qH29yhp/Yzc1rlHR61hAyKGcAQBOas+hTI2eEa+l2/brhp6N9Phl7RQRHuZ1LCAkUc4AAL/rh02puntWvI5k5+nl67royq71vY4EhDTKGQDguPLznV5bvEkvLdqgFrUqadbIbmpRO8rrWEDIo5wBAP7HvvQs3TM7Qd9tTNVVXevrmas6KLIcvzKAksC/aQCA34jdtl93zYjX/iPZevbqjrq+R0OZMe0fKCmUMwCAJMk5p3e+26rnP1+n+tUqaN4d56hD/SpexwJKHcoZAEBpR3P0wNwV+nLNHvVrf5Ze+HMnVY4I9zoWUCpRzgCglFuVnKZRM+K0+2CmHrusnYae24TDmICHKGcAUEo55zT95x16+uM1qlGpnGbfdra6N67mdSyg1KOcAUAplJGVq0c+XKX5Cbv0h1a19NJ1XVS9YjmvYwEQ5QwASp0New7rjmlx2pqaofsvaaVRF7ZQmTIcxgQCBeUMAEqRecuT9eiHiapYvqymDe+pc5rX9DoSgGNQzgCgFMjMydOTC1Zr1rIk9WxaXf8a1FW1K0d4HQvAcVDOACDEbUvN0Kjpy7Vm9yHd2bu5xl7cSmXDyngdC8AJUM4AIIR9tmq3Hnh/pcqGmSYN6aHebWp7HQnASVDOACAEZefm69nP1mrSD9vUpWFVvX5jN9WvWsHrWACKwK/r2mbWz8zWm9kmM3v4ONsbmdliM4s3s5Vm1v8429PN7H5/5gSAULLz4FFd+9YSTfphm249t4nm3HY2xQwIIn5bOTOzMEmvS+orKVnSMjNb4JxbU+ht4yTNcc69YWbtJC2U1KTQ9hclfeavjAAQahav36uxsxOUm+c0/sZu6t+xrteRAJwifx7WjJG0yTm3RZLMbJakKyQVLmdOUmXf4yqSdv26wcyulLRVUoYfMwJASMjNy9dLizbo9cWb1bZuZY2/sZua1qzodSwAp8Gf5ay+pKRCz5Ml9TzmPU9K+tLMRkuqKOliSTKzSpIeUsGq2wkPaZrZSEkjJalRo0bFlRsAgsrew5kaMzNeP23Zr+t7NNSTl7dXRHiY17EAnCavr6UeJGmyc66BpP6SpppZGRWUtpecc+m/t7Nz7m3nXLRzLrpWrVr+TwsAAWbJ5n3q/8r3Skg6qH/+ubOeG9iJYgYEOX+unO2U1LDQ8wa+1wobJqmfJDnnlphZhKSaKlhhu8bMXpBUVVK+mWU6517zY14ACBr5+U5vfLtZ//xyvZrWrKjpw3uq9VlRXscCUAz8Wc6WSWppZk1VUMqul3TDMe/ZIamPpMlm1lZShKQU59z5v77BzJ6UlE4xA4ACBzKyNXZOgr5Zn6LLO9fTs1d3VMXyTEYCQoXf/m12zuWa2V2SvpAUJmmic261mT0tKdY5t0DSfZImmNlYFVwcMMQ55/yVCQCC3fIdB3TX9OVKTc/W367soBt7NpIZNy0HQomFSheKjo52sbGxXscAAL9wzmniD9v07MK1qls1QuNv6K6ODap4HQvAaTKzOOdc9PG2sQ4OAAHuUGaOHnp/pT5L/EV929XRP67prCqR4V7HAuAnlDMACGCrd6XpzunLlXTgqB7t31bDz2/KYUwgxFHOACAAOec0a1mSnliwWtUjy2n2yF6KblLd61gASgDlDAACzJHsXI37MFHz4nfq/JY19fJ1XVSjUnmvYwEoIZQzAAggm/Ye1h3TlmtTSrrGXtxKd13UQmFlOIwJlCaUMwAIEB+v2KWHPlipCuFhmjq0p85rWdPrSAA8QDkDAI/l5OXr2YXrNPGHrYpuXE2v3dBNZ1WJ8DoWAI9QzgDAQ3sPZerOGcu1bNsB3XpuEz3Sv63Cw7y+7TEAL1HOAMAjS7fu150zlp1CsFYAACAASURBVCs9M1evXN9FV3Sp73UkAAGAcgYAJcw5p3e/36pnP1unxtUjNX14T7Wqw03LARSgnAFACcrIytWDH6zUpyt364/t6+gff+6sqAim/QP4/yhnAFBCNu1N1+3T4rQlJV0P/6mNbrugGdP+AfwPyhkAlIDPVu3W/XNXKCI8TNOG9dQ5LRiTAeD4KGcA4Ee5efl64Yv1evs/W9SlYVW9cVM31a1SwetYAAIY5QwA/CTlcJZGz1yun7bs1829GmvcZW1VvmyY17EABDjKGQD4Qdz2Axo1PU5pR3P04rWddXW3Bl5HAhAkKGcAUIycc3pvyXb97dM1qlulgubdEaN29Sp7HQtAEKGcAUAxOZKdq0fmrdJHCbvUp01tvXhtF1WJZEwGgFNDOQOAYrA1NUO3T43Thr2Hdf8lrTTqwhYqU4YxGQBOHeUMAM7Ql6t/0X1zVigszDTl1hhd0KqW15EABDHKGQCcprx8pxe/Wq/XF29WpwZVNP7GbmpQLdLrWACCHOUMAE7DvvQs3T0rQd9vStWgmIZ6YkB7RYQzJgPAmaOcAcApSkg6qFHT4pSaka0XBnbStT0aeh0JQAihnAFAETnnNGPpDj21YI1qVy6veXecow71q3gdC0CIoZwBQBFk5uRp3EeJej8uWX9oVUsvX9dF1SqW8zoWgBBEOQOAk9ix74hunxanNbsP6e4+LTWmT0uFMSYDgJ9QzgDgdyxet1d3z4qXJE0a0kO929T2OBGAUEc5A4DjyMt3euXrjXr1641qV7ey3rypuxrVYEwGAP+jnAHAMQ4eydbdsxL07YYUDezWQM9c1YExGQBKDOUMAApJ3Jmm26fFae+hLD1zVQfdENNIZpxfBqDkUM4AwGfOsiSNm5+omhXLac7tZ6tLw6peRwJQClHOAJR6mTl5eurj1Zq5NEnntaipV67vohqVynsdC0ApRTkDUKolHziiO6Yt16qdabqzd3Pd27c1YzIAeIpyBqDU+s+GFI2ZFa+8PKe3b+6uS9qf5XUkAKCcASh98vOdXl+8SS8u2qBWtaP05s3d1bRmRa9jAYAkyhmAUibtaI7unZ2gr9ft1RVd6unZqzsqshw/CgEEDn4iASg11uw6pDumx2nngaN66vL2Gnx2Y8ZkAAg4lDMApcK85cl65MNVqlIhXLNv66Xujat7HQkAjotyBiCkZefm66+frNHUn7arZ9Pqeu2GbqoVxZgMAIGLcgYgZO1OO6o7pi1XQtJBjbygmR78Y2uVDSvjdSwA+F2UMwAh6cdNqRo9M16ZOXkaf2M39e9Y1+tIAFAklDMAIcU5p7f+s0UvfL5OzWpV0ps3dVeL2pW8jgUARUY5AxAyDmfm6P65K/TF6j26tFNdvTCwkyqW58ccgODCTy0AIWHDnsO6fWqctu8/onGXttWw85oyJgNAUKKcAQh68xN26uEPVqli+bKaMbynejar4XUkADhtlDMAQSsnL19/X7hWk37YpujG1fT6jd1Up3KE17EA4IxQzgAEpb2HMjVq+nLFbj+goec21V/6t1E4YzIAhADKGYCg8/OWfbpzRrwysnL16qCuurxzPa8jAUCxoZwBCBrOOb37/VY9+9k6Na4eqRkjeqpVnSivYwFAsaKcAQgKR7Pz9OAHK/Xxil36Y/s6+sefOysqItzrWABQ7ChnAALe7rSjGvlenBJ3pemBP7bWqAubMyYDQMiinAEIaMt3HNDI9+J0NDtXE26O1sXt6ngdCQD8inIGIGB9EJesv8xbpbOqRHB+GYBSg3IGIODk5Ts9//k6vf2fLTq7WQ2Nv7GbqlUs53UsACgRlDMAAeVQZo7GzIzXN+tTdHOvxnp8QDvmlwEoVShnAALG1tQMDZ+yTNv3HdHfruygm3o19joSAJQ4yhmAgPD9xlTdOWO5ypg0bXhP9eL+mABKKcoZAE855zT5x23626dr1aJWJb1zS7QaVo/0OhYAeIZyBsAz2bn5enx+omYtS1LfdnX00nVdVKk8P5YAlG78FATgidT0LN0xLU7Lth3QXb1b6N6+rVSmDINlAYByBqDErdl1SCPei1VqehY3LgeAY1DOAJSozxN3a+zsFapSIVxzbz9bnRpU9ToSAAQUyhmAEuGc06tfb9JLizaoS8Oqevvm7qpdOcLrWAAQcChnAPzuSHauHpi7Up+u2q2ru9XX36/qqIjwMK9jAUBAopwB8KudB49q5HuxWrP7kB7p30Yjzm8mM078B4AToZwB8Ju47ft129Q4ZeXka+ItPdS7TW2vIwFAwKOcAfCLubFJevTDRNWrGqFZI6PVonaU15EAIChQzgAUq9y8fD372Tq9+/1Wndeipl67oauqRpbzOhYABA3KGYBik3Y0R6Nnxus/G1I05JwmGndpW5UNK+N1LAAIKpQzAMVic0q6RkyJVdKBI3r26o4aFNPI60gAEJQoZwDO2LcbUnTXjOUKDyuj6cN7KaZpda8jAUDQopwBOG3OOb37/Vb9feFataoTpXduiVaDapFexwKAoObXk0HMrJ+ZrTezTWb28HG2NzKzxWYWb2Yrzay/7/UYM0vw/VlhZlf5MyeAU5eVm6cH31+pv326Vn3b1dEHd5xDMQOAYlCklTMzmyfpXUmfOefyi7hPmKTXJfWVlCxpmZktcM6tKfS2cZLmOOfeMLN2khZKaiIpUVK0cy7XzOpKWmFmHzvncov6jQHwn5TDWbp9Wpzith/QmD4tdU+flipThsGyAFAcirpyNl7SDZI2mtlzZta6CPvESNrknNvinMuWNEvSFce8x0mq7HtcRdIuSXLOHSlUxCJ87wMQABJ3pumK177X6l1pev2Gbrq3byuKGQAUoyKVM+fcIufcjZK6SdomaZGZ/Whmt5pZ+Al2qy8pqdDzZN9rhT0p6SYzS1bBqtnoXzeYWU8zWy1plaTbj7dqZmYjzSzWzGJTUlKK8q0AOAMLV+3Wn99cIifp/dvP0aWd6nodCQBCTpHPOTOzGpKGSBouKV7SKyooa1+dwecPkjTZOddAUn9JU82sjCQ55352zrWX1EPSX8ws4tidnXNvO+einXPRtWrVOoMYAH5Pfr7Ti19t0Kjpy9W2bpQW3HWeOtSv4nUsAAhJRT3n7ENJrSVNlTTAObfbt2m2mcWeYLedkhoWet7A91phwyT1kyTn3BJfAaspae+vb3DOrTWzdEkdJJ3oswD4SUZWru6bs0Kfr/5F13RvoGeu6qDyZcO8jgUAIauoozRedc4tPt4G51z0CfZZJqmlmTVVQSm7XgXnrRW2Q1IfSZPNrK0Kzi9L8e2T5LsgoLGkNio4nAqgBCUfOKLhU2K1Yc9hjbu0rYad11RmnF8GAP5U1HLWzszinXMHJcnMqkka5Jwbf6IdfMXqLklfSAqTNNE5t9rMnpYU65xbIOk+SRPMbKwKTvof4pxzZnaepIfNLEdSvqRRzrnU0/4uAZyyZdv26/apccrOy9ekW2P0h1acOgAAJcGcO/mFkGaW4Jzrcsxr8c65rn5Ldoqio6NdbCxHPYHiMGvpDj02P1ENq0Vqwi3Ral6rkteRACCkmFnciY4+FnXlLMzMzPmanG+GWbniCgggMOTm5etvn67V5B+36fyWNfXaoG6qEnmiC7IBAP5Q1HL2uQpO/n/L9/w232sAQkTakRzdNXO5vtuYqmHnNdVf/tRGZcP8ehMRAMBxFLWcPaSCQnaH7/lXkt7xSyIAJW7T3nQNn7JMOw8e1QsDO+naHg1PvhMAwC+KVM58t2x6w/cHQAhZvH6vxsyIV/nwMpo5opeim1T3OhIAlGpFnXPWUtKzktqpYNyFJMk518xPuQD4mXNO73y3Vc9+tlZtzqqsCbdEq37VCl7HAoBSr6iHNSdJekLSS5J6S7pVp3B3AQCBJTMnT498uErzlu/UpR3r6v/+3EmR5Yr64wAA4E9F/WlcwTn3te+Kze2SnjSzOEmP+zEbAD/YeyhTt02LU/yOgxp7cSuN6dOCwbIAEECKWs6yfPe83OgbLLtTEoOPgCCzKjlNI96LVdrRHL1xYzf9qSM3LgeAQFPUQ5N3S4qUNEZSd0k3SbrFX6EAFL+PV+zSn9/6UWFlTB/ccQ7FDAAC1ElXznwDZ69zzt0vKV0F55sBCBL5+U4vfrVBry3epB5NqumNm7qrZqXyXscCAJzAScuZcy7Pd69LAEEmIytXY2cn6Ms1e3RddEP99coOKleWa3kAIJAV9ZyzeDNbIGmupIxfX3TOzfNLKgBnLGn/EY14L1Yb9hzWEwPaacg5TTjxHwCCQFHLWYSkfZIuKvSak0Q5AwLQT1v2adT05crNy9eUoTE6v2UtryMBAIqoqHcI4DwzIEjM+HmHHp+fqMY1IvXOLT3UtGZFryMBAE5BUe8QMEkFK2W/4ZwbWuyJAJyWvHynpz9erSlLtuvC1rX06qCuqhwR7nUsAMApKuphzU8KPY6QdJWkXcUfB8DpOJqdp9Ez47Vo7R6NOL+pHv5TW4WV4fwyAAhGRT2s+UHh52Y2U9L3fkkE4JTsz8jWsCnLlJB0UE9f0V6Dz27idSQAwBk43ZvptZRUuziDADh1O/Yd0S2TlmrXwaN648bu6tfhLK8jAQDOUFHPOTus355z9oukh/ySCECRrEpO062Tlyo332n68J6KblLd60gAgGJQ1MOaUf4OAqDovlm/V6OmL1e1yHKaNTRGLWpzq1sACBVFGhVuZleZWZVCz6ua2ZX+iwXgRObGJmnYlFg1qVFRH446h2IGACGmqPdxecI5l/brE+fcQUlP+CcSgONxzulfX2/UA++v1DnNa2j2bb1Uu3KE17EAAMWsqBcEHK/Ene7FBABOUW5evh6bv1ozl+7Q1d3q67mrO3GPTAAIUUUtWLFm9qKk133P75QU559IAAo7kp2r0TPi9fW6vbqzd3Pdf0lr7pEJACGsqP/pPVpStqTZkmZJylRBQQPgR/vSs3TDhJ+1eP1e/fXKDnrgj20oZgAQ4op6tWaGpIf9nAVAIdv3ZeiWiUu1Oy1Tb9zUXX9szwwzACgNinq15ldmVrXQ82pm9oX/YgGl24qkgxr4xo86eDRHM0b0pJgBQClS1HPOavqu0JQkOecOmBl3CAD8YPG6ghlmNSqV05ShMWpei1EZAFCaFPWcs3wza/TrEzNrot/eMQBAMZizLEnD34tVs1oVNW/UORQzACiFirpy9qik783sW0km6XxJI/2WCihlnHN69etNemnRBp3fsqbeuKm7KpVnWg0AlEZFvSDgczOLVkEhi5f0kaSj/gwGlBYFM8wSNXNpkq7uVl/PD+yk8DBmmAFAaVXUG58Pl3S3pAaSEiT1krRE0kX+iwaEviPZubprRrz+vW6v7urdQvdd0opRGQBQyhX1P8/vltRD0nbnXG9JXSUd/P1dAPye1PQsDXr7J32zfq+euaqD7v8jw2UBAEU/5yzTOZdpZjKz8s65dWbW2q/JgBC2LTVDt0xaqj2HMvXWzdHq266O15EAAAGiqOUs2Tfn7CNJX5nZAUnb/RcLCF0JSQc1bPIy5Tun6cN7qXvjal5HAgAEkKJeEHCV7+GTZrZYUhVJn/stFRCi/r1uj+6cHq+aUeU05dYYNWNUBgDgGKd8rb5z7lt/BAFC3aylO/ToR4lqV7ey3h0SrdpREV5HAgAEIAYpAX7mnNPLizbqla836g+tamn8jd1UkRlmAIAT4DcE4Ec5efka92GiZscm6ZruDfTs1R2ZYQYA+F2UM8BPMrJydeeM5fpmfYrGXNRCY/sywwwAcHKUM8APUtOzNHTyMiXuTNMzV3XQjT0bex0JABAkKGdAMSs8w+ztm6N1MTPMAACngHIGFKP4HQc0bEqsJGnmiF7q2ogZZgCAU0M5A4rJ12v36M4Zy1U7KkJThsaoac2KXkcCAAQhyhlQDGb8vEPjPlql9vWqaOKQHqoVVd7rSACAIEU5A86Ac04vfbVBr/57ky5sXUuv38AMMwDAmeG3CHCacvLy9ci8VZobl6xroxvomauYYQYAOHOUM+A0ZGTlatT05fp2Q4rG9GmpsRe3ZIYZAKBYUM6AU5RyuGCG2epdaXr26o4aFNPI60gAgBBCOQNOwZaUdN0yaalSD2drwuBo9WnLDDMAQPGinAFFtHzHAQ2bvExmppkje6lLw6peRwIAhCDKGVAEX63Zo9Ezl6tO5QhNuTVGTZhhBgDwE8oZcBLTf96uxz5KVIf6BTPMalZihhkAwH8oZ8AJOOf0zy836LXFm9S7dS29xgwzAEAJ4DcNcBw5efl6+INV+mB5sq6LbqhnruqgsswwAwCUAMoZcIx03wyz/2xI0T0Xt9TdfZhhBgAoOZQzoJC9hzM1dPIyrd19WM8P7KjrejDDDABQsihngM/mlHTdMnGp9qVna8Lg7rqoDTPMAAAlj3IGSIrbfkDDpixTmJlmjeylzswwAwB4hHKGUu+L1b9ozMx41a0SoSlDY9S4BjPMAADeoZyhVJv603Y9MT9RHRtU1cRbolWDGWYAAI9RzlAqOef0f1+s1/hvNuuiNrX12g1dFVmOfx0AAN7jtxFKnezcfD08b6XmLd+pQTEN9dcrmGEGAAgclDOUKulZubpjWpy+25iqe/u20uiLWjDDDAAQUChnKDVS07N0y8SlWvfLYb0wsJOu7dHQ60gAAPwPyhlKhcOZORoyaak2p6TrnVui1bt1ba8jAQBwXJQzhLys3DzdNjVOa3cf1oTB3SlmAICAxlnQCGl5+U73zl6hHzfv0wsDOzH1HwAQ8ChnCFnOOT25YLU+XbVbj/Zvq4HdG3gdCQCAk6KcIWS98vVGTf1pu277QzONuKCZ13EAACgSyhlC0rSftuvlRRt1TfcGerhfG6/jAABQZJQzhJyFq3brsfmJ6tOmtp67uiNzzAAAQcWv5czM+pnZejPbZGYPH2d7IzNbbGbxZrbSzPr7Xu9rZnFmtsr3vxf5MydCx4+bUnXPrAR1b1RNr93Qjcn/AICg47dRGmYWJul1SX0lJUtaZmYLnHNrCr1tnKQ5zrk3zKydpIWSmkhKlTTAObfLzDpI+kJSfX9lRWhYlZymEe/FqmnNinr3lh6qUC7M60gAAJwyfy4rxEja5Jzb4pzLljRL0hXHvMdJqux7XEXSLklyzsU753b5Xl8tqYKZlfdjVgS5rakZGjJpqapGltOUoTGqEhnudSQAAE6LP8tZfUlJhZ4n639Xv56UdJOZJatg1Wz0cb7OQEnLnXNZx24ws5FmFmtmsSkpKcWTGkFn76FM3fzuz3KS3hsWo7OqRHgdCQCA0+b1CTmDJE12zjWQ1F/SVDP7byYzay/peUm3HW9n59zbzrlo51x0rVq1SiQwAkva0RwNnrhU+zOyNWlIDzWvVcnrSAAAnBF/lrOdkgrfWbqB77XChkmaI0nOuSWSIiTVlCQzayDpQ0mDnXOb/ZgTQSozJ08jpsRqc0q63rq5uzo3rOp1JAAAzpg/y9kySS3NrKmZlZN0vaQFx7xnh6Q+kmRmbVVQzlLMrKqkTyU97Jz7wY8ZEaRy8/I1ema8lm3frxev7aLzW7JyCgAIDX4rZ865XEl3qeBKy7UquCpztZk9bWaX+952n6QRZrZC0kxJQ5xzzrdfC0mPm1mC7w93q4akgtsyPfphor5as0dPDmivAZ3reR0JAIBiYwVdKPhFR0e72NhYr2OgBLzw+TqN/2azxlzUQvde0trrOAAAnDIzi3PORR9vm9cXBACn5N3vt2r8N5s1KKaRxvZt5XUcAACKHeUMQeOj+J366ydr1K/9WfrblR24LRMAICRRzhAUvlm/V/fPXaFezarr5eu7KKwMxQwAEJooZwh48TsO6I5py9WqTpQmDI5WRDi3ZQIAhC7KGQLapr2HdevkZaoVVV6Th/ZQVAS3ZQIAhDbKGQLWroNHNfjdpSpbpoymDotR7ShuywQACH2UMwSkAxnZGjxxqQ5n5mrK0B5qXKOi15EAACgRZb0OABzrSHauhk5Zph37j2jKrTFqX6+K15EAACgxrJwhoOTk5WvU9OVakXRQr17fRWc3r+F1JAAAShQrZwgY+flOD76/Ut+sT9GzV3dUvw51vY4EAECJY+UMAcE5p78vXKsP43fq/ktaaVBMI68jAQDgCcoZAsJb/9mid77fqiHnNNGdvVt4HQcAAM9QzuC5ObFJeu6zdRrQuZ4ev6wdt2UCAJRqlDN4atGaPfrLvFU6v2VN/fPPnVWG2zIBAEo5yhk8s2zbft05Y7k61KusN2/qrnJl+esIAAC/DeGJdb8c0tDJy1S/agVNHNJDFctz4TAAABLlDB5I2n9Eg99dqshyYXpvWIxqVCrvdSQAAAIG5QwlKjU9S4MnLlVmTp7eG9pTDapFeh0JAICAwrEklJj0rFzdOmmZdh08qunDe6r1WVFeRwIAIOBQzlAisnLzdNvUWK3ZfUhv39xd0U2qex0JAICAxGFN+F1evtO9c1boh0379PzATurTto7XkQAACFiUM/iVc05Pfbxan67crUf6t9E13Rt4HQkAgIBGOYNf/evfm/Teku0aeUEzjbyguddxAAAIeJQz+M20n7brxa826Opu9fVwvzZexwEAIChQzuAXC1ft1mPzE3VRm9p6fmAnbssEAEARUc5Q7H7clKp7ZiWoW6Nqev2GbgoP468ZAABFxW9NFKvEnWkaOTVOTWpG6t1bolWhXJjXkQAACCqUMxSbbakZGjJpqapUCNeUoTGqGlnO60gAAAQdyhmKxd5Dmbp54s/Ky3eaMjRGdatU8DoSAABBiTsE4IwdyszRLZOWaV96tmaM6KUWtSt5HQkAgKDFyhnOSGZOnoZPidWmvYf15k3d1aVhVa8jAQAQ1Fg5w2nLzcvXmJnxWrp1v165vosuaFXL60gAAAQ9Vs5wWpxzGvdRor5cs0dPDGinK7rU9zoSAAAhgXKG0/KPL9dr1rIk3dW7hW49t6nXcQAACBmUM5yyST9s1euLN2tQTEPdd0krr+MAABBSKGc4JfMTduqpj9foj+3r6K9XdJAZt2UCAKA4Uc5QZN9uSNF9c1aoZ9PqeuX6rirLbZkAACh2/HZFkSQkHdQd0+LUsk6UJtwSrYhwbssEAIA/UM5wUpv2puvWSUtVs1J5TRnaQ5Ujwr2OBABAyKKc4XftTjuqwe/+rLAypqnDYlQ7KsLrSAAAhDSG0OKEDh7J1uB3l+pQZq5mjeylxjUqeh0JAICQx8oZjutodp6GTl6m7fuO6O3B3dWhfhWvIwEAUCqwcob/kZOXr1HT4xSfdFDjb+imc5rX9DoSAAClBitn+I38fKeH3l+pxetT9LcrO+hPHet6HQkAgFKFcobfePaztZoXv1P39m2lG3s29joOAAClDuUM/zXlx22a8N1WDT67sUZf1MLrOAAAlEqUM0iSMnPy9PKiDTq/ZU09MaA9t2UCAMAjlDNIKrhn5oEjObqzd4v/196dB8dZ33cc/3yty/ctX7J1UCBgG2yMsLDUpk1NEq4AzZDWBhPLk5n8Q9IkQ6dNOkmbyd+dtJ0pTXM0lg0EAwZSlyElIaFukORD2AZiY8DYq8uXfFuyZV3f/rGbGcUhiVbS6vc8u+/XX7vPPrv7kZ+x9qPn+K7yxlHMAAAIhXIGubs21id0w7wpqqqYGToOAAA5jXIG7TxyRgePX9SGmnIOZwIAEBjlDKqrT2j6xALdv7wkdBQAAHIe5SzHtZ29pJ8eOK61K0s1viAvdBwAAHIe5SzHPbGjWWamdbcz0wwAgCignOWwyz392rKrVZ9cMlcl0yeEjgMAAEQ5y2k/3teu85d7VVtdEToKAABIoZzlqOT4jCNasmCqbiufEToOAABIoZzlqMYPTuu9E52qrWZ8BgAAUUI5y1EbGxKaOalQn1q2IHQUAAAwCOUsB7WeuaRX3zmhhxifAQBA5FDOctDmxoTGMT4DAIBIopzlmK4rfdqyu1V3LZ2nedPGh44DAACuQjnLMS/ubdfF7j5tqCkPHQUAAHwIylkOcXfVNSR0U8k0rShlfAYAAFFEOcsh9YdO69BJxmcAABBllLMcsrH+iGZPLtS9y+aHjgIAAH4HylmOSJzq0i/ePamHqspUlM/4DAAAoopyliM2NzYrz0zrqkpDRwEAAL8H5SwHdF7p03NNrbrn5vmaM5XxGQAARBnlLAe8sKdNF6/0qba6PHQUAADwB1DOstzAQHJ8xrJF03UL4zMAAIg8ylmW++WhUzrc0aUN7DUDACAWKGdZrq7+iIqnFOnumxifAQBAHGS0nJnZnWb2rpkdMrOvfsjjpWb2mpntNbO3zOzu1PJZqeWdZvZvmcyYzQ53dOq1dzv0cFWpCvPp4QAAxEHGPrHNLE/S45LukrRY0lozW3zVal+X9Ky73yJpjaR/Ty3vlvQNSX+TqXy5YHNjswryTA9XlYWOAgAAhiiTu1NWSjrk7ofdvUfSFkn3X7WOS5qauj1N0lFJcvcud39dyZKGYbjY3avnmlr1qZsXqHhKUeg4AABgiDJZzkoktQ6635ZaNtg3Ja0zszZJL0v6YjpvYGafN7MmM2vq6OgYSdass/WNNnX19Gs9FwIAABAroU9EWiupzt0XSrpb0hNmNuRM7v49d69098ri4uKMhYybgQHXpoaEVpRO17JF00PHAQAAachkOWuXtGjQ/YWpZYN9TtKzkuTujZLGS5qdwUw5Yft7HUqcvqTamorQUQAAQJoyWc52S7rOzCrMrFDJE/63XbVOi6TVkmRmNypZzjg+OUIbGxKaO7VIdy2dFzoKAABIU36mXtjd+8zsC5JekZQn6Yfuvt/MviWpyd23SXpM0vfN7CtKXhxQ6+4uSWaWUPJigUIze0DSJ9z9QKbyZotDJzv1f+916LGPX6+CvNBHrQEAQLoyVs4kyd1fVvJE/8HL/mHQ7QOSan7Hc8szmS1bbWpIqDBvnNZWlYaOAgAAOX5iLQAADrhJREFUhoFdK1nk/OVePb+nTfctX6DZkxmfAQBAHFHOsshzTa261NOvWsZnAAAQW5SzLNE/4Nrc2Kzbymdoacm00HEAAMAwUc6yxGsHT6rlzCXVVjM+AwCAOKOcZYm6hoTmTxuvTyyZGzoKAAAYAcpZFnj/xEW9fuiU1t1exvgMAABijk/yLFDXkFBh/jitXcn4DAAA4o5yFnPnL/XqhT3temD5As2cVBg6DgAAGCHKWcw909Siy739XAgAAECWoJzFWP+Aa1NDs6oqZmrxgqmh4wAAgFFAOYuxV985ofZzl7Whpjx0FAAAMEooZzFWV59QyfQJuuNGxmcAAJAtKGcxdfD4BTUePq1HVpUpn/EZAABkDT7VY2pTQ0LjC8ZpzW2LQkcBAACjiHIWQ2e7evTi3nb9xS0lmj6R8RkAAGQTylkMPdPUqu7eAa2vLg8dBQAAjDLKWcz09Q9oc0NCq66ZpRvmMT4DAIBsQzmLmZ8dOKGj57sZnwEAQJainMXMxoaEFs6YoNWMzwAAICtRzmJk/9Hz2nXkjNavKlfeOAsdBwAAZADlLEY2NSQ0oSBPf1nJ+AwAALIV5SwmznT16Mf7jurTK0o0bWJB6DgAACBDKGcx8fSuFvX0DaiW8RkAAGQ1ylkM9PYP6Mkdzfrja2frurlTQscBAAAZRDmLgVf2H9ex893sNQMAIAdQzmKgrj6h0pkT9bEb5oSOAgAAMoxyFnFvt51XU/NZra9mfAYAALmAchZxdQ0JTSzM02cqF4aOAgAAxgDlLMJOdV7Rf795VA/eulBTxzM+AwCAXEA5i7Cnd7aop39An11VHjoKAAAYI5SziOrtH9ATO5r10euLde2cyaHjAACAMUI5i6if/Oq4Tl68og2MzwAAIKdQziJqY/0Rlc+aqD+9vjh0FAAAMIYoZxG0r/Wc9rac0/rqco1jfAYAADmFchZBmxoSmlyUrwdvZXwGAAC5hnIWMScvduult5LjM6YwPgMAgJxDOYuYH+1sUW+/az0XAgAAkJMoZxHS0zegJ3e06GMfKVbF7Emh4wAAgAAoZxHy8tvHdKrzimprKkJHAQAAgVDOImRjQ0LXFE/Sn1w7O3QUAAAQCOUsIva0nNWbredUy/gMAAByGuUsIurqE5pSlK9Pr2B8BgAAuYxyFgEnLnTr5beP6TOVizS5KD90HAAAEBDlLAKe2tGsfnetry4LHQUAAARGOQvsSl+/ntrZotU3zFHZLMZnAACQ6yhngb305jGd7upRbTXjMwAAAOUsKHdXXUNC186ZrJprZ4WOAwAAIoByFtCelrN6u/28aqvLZcb4DAAAQDkLamN9QlPG5+vTK0pCRwEAABFBOQvk2PnL+smvjmvNbYs0sZDxGQAAIIlyFsiTO5rl7vrsqvLQUQAAQIRQzgLo7u3Xj3a26I4b52rRzImh4wAAgAihnAWw7c2jOnupV7U15aGjAACAiKGcjTF3V119Qh+ZO0WrrmF8BgAA+E2UszG2O3FWB45dUG0N4zMAAMBvo5yNsbqGI5o2oUAPLGd8BgAA+G2UszHUfu6yXtl/QmtWLtKEwrzQcQAAQARRzsbQE43J8RmP3F4WOgoAAIgoytkYudzTry27W/SJxfO0cAbjMwAAwIejnI2R/9rXrnOMzwAAAH8A5WwMuLvqGhK6cf5UVVXMDB0HAABEGOVsDOw4fEYHj1/UhmrGZwAAgN+PcjYG6hqOaMbEAt23fEHoKAAAIOIoZxnWeuaSfnbghNauLNX4AsZnAACA349ylmFP7miWmWkd4zMAAMAQUM4y6FJPn57e1aI7l8zTgukTQscBAAAxQDnLoBf3tutCdx/jMwAAwJBRzjLE3VVXn9CSBVNVWTYjdBwAABATlLMMafjgtN4/2akNNRWMzwAAAENGOcuQjfUJzZpUqHtvnh86CgAAiBHKWQa0nL6knx88oYeqGJ8BAADSQznLgM2NCeWZ6eEqxmcAAID0UM5GWdeVPj3T1Kq7bpqvedPGh44DAABiJqPlzMzuNLN3zeyQmX31Qx4vNbPXzGyvmb1lZncPeuxrqee9a2afzGTO0fTC3nZd7O5TbXV56CgAACCG8jP1wmaWJ+lxSR+X1CZpt5ltc/cDg1b7uqRn3f07ZrZY0suSylO310haImmBpFfN7Hp3789U3tEwMOCqqz+imxdO04rS6aHjAACAGMrknrOVkg65+2F375G0RdL9V63jkqambk+TdDR1+35JW9z9irsfkXQo9XqR9vqhU/qgo0u11eWMzwAAAMOSyXJWIql10P221LLBvilpnZm1KbnX7ItpPFdm9nkzazKzpo6OjtHKPWx1DQnNnlykexifAQAAhin0BQFrJdW5+0JJd0t6wsyGnMndv+fule5eWVxcnLGQQ3HkVJd+cfCkHq4qVVE+4zMAAMDwZOycM0ntkhYNur8wtWywz0m6U5LcvdHMxkuaPcTnRsrmxoQK8kwPV5WGjgIAAGIsk3vOdku6zswqzKxQyRP8t121Touk1ZJkZjdKGi+pI7XeGjMrMrMKSddJ2pXBrCPSeaVPzzW16Z6b5mvOVMZnAACA4cvYnjN37zOzL0h6RVKepB+6+34z+5akJnffJukxSd83s68oeXFArbu7pP1m9qykA5L6JD0a5Ss1n3+jTZ1X+lRbUxE6CgAAiDlLdqH4q6ys9KampjF/34EB1x3f3q6pEwr040drxvz9AQBA/JjZG+5e+WGPhb4gIPa2v9+hw6e6tKGmPHQUAACQBShnI1RXn1DxlCLdtZTxGQAAYOQoZyPwQUentr/XoXVVZSrM558SAACMHI1iBDY3JFSYN04PMT4DAACMEsrZMF3o7tXWN9p077L5Kp5SFDoOAADIEpSzYdra1Kaunn5tqGZ8BgAAGD2Us2EYGHBtakzo1rIZumnhtNBxAABAFqGcDcP/vndSzacvqba6PHQUAACQZShnw7CxPqG5U4t059J5oaMAAIAsQzlL0/snLuqX75/SI7eXqSCPfz4AADC6aBdp2tSYUGH+OK1dyfgMAAAw+ihnaTh/uVfPv9Gu+5Yt0KzJjM8AAACjj3KWhueaWnW5t58LAQAAQMZQzoaoPzU+Y2X5TC0tYXwGAADIDMrZEJ240K0JBXmqrSkPHQUAAGSx/NAB4mLB9Al65csflXvoJAAAIJtRztJgZjILnQIAAGQzDmsCAABECOUMAAAgQihnAAAAEUI5AwAAiBDKGQAAQIRQzgAAACKEcgYAABAhlDMAAIAIoZwBAABECOUMAAAgQihnAAAAEUI5AwAAiBDKGQAAQIRQzgAAACKEcgYAABAhlDMAAIAIoZwBAABECOUMAAAgQihnAAAAEUI5AwAAiBDKGQAAQIRQzgAAACLE3D10hlFhZh2SmsfgrWZLOjUG74PMYPvFH9sw/tiG8cc2HLkydy/+sAeyppyNFTNrcvfK0DkwPGy/+GMbxh/bMP7YhpnFYU0AAIAIoZwBAABECOUsfd8LHQAjwvaLP7Zh/LEN449tmEGccwYAABAh7DkDAACIEMoZAABAhFDOhsjM7jSzd83skJl9NXQepMfMFpnZa2Z2wMz2m9mXQmdC+swsz8z2mtlLobMgfWY23cy2mtlBM3vHzFaFzoT0mNlXUr9Df2VmT5vZ+NCZshHlbAjMLE/S45LukrRY0lozWxw2FdLUJ+kxd18s6XZJj7INY+lLkt4JHQLD9q+S/sfdb5C0TGzLWDGzEkl/LanS3ZdKypO0Jmyq7EQ5G5qVkg65+2F375G0RdL9gTMhDe5+zN33pG5fVPJDoSRsKqTDzBZKukfSD0JnQfrMbJqkj0r6T0ly9x53Pxc2FYYhX9IEM8uXNFHS0cB5shLlbGhKJLUOut8mPthjy8zKJd0iaWfYJEjTv0j6W0kDoYNgWCokdUjamDo0/QMzmxQ6FIbO3dsl/ZOkFknHJJ1395+GTZWdKGfIKWY2WdLzkr7s7hdC58HQmNm9kk66+xuhs2DY8iWtkPQdd79FUpckzt+NETOboeRRowpJCyRNMrN1YVNlJ8rZ0LRLWjTo/sLUMsSImRUoWcyecvcXQudBWmok3WdmCSVPK/hzM3sybCSkqU1Sm7v/eo/1ViXLGuLjDklH3L3D3XslvSCpOnCmrEQ5G5rdkq4zswozK1TyBMhtgTMhDWZmSp7r8o67fzt0HqTH3b/m7gvdvVzJ/3+/cHf+Yo8Rdz8uqdXMPpJatFrSgYCRkL4WSbeb2cTU79TV4qKOjMgPHSAO3L3PzL4g6RUlr075obvvDxwL6amR9Iikt81sX2rZ37v7ywEzAbnmi5KeSv2Re1jShsB5kAZ332lmWyXtUfIK+L3ia5wygq9vAgAAiBAOawIAAEQI5QwAACBCKGcAAAARQjkDAACIEMoZAABAhFDOAGCEzOzPzOyl0DkAZAfKGQAAQIRQzgDkDDNbZ2a7zGyfmX3XzPLMrNPM/tnM9pvZz82sOLXucjPbYWZvmdmLqe8VlJlda2avmtmbZrbHzP4o9fKTzWyrmR00s6dSE9QBIG2UMwA5wcxulPRXkmrcfbmkfkkPS5okqcndl0jaLukfU0/ZLOnv3P1mSW8PWv6UpMfdfZmS3yt4LLX8FklflrRY0jVKfisFAKSNr28CkCtWS7pV0u7UTq0Jkk5KGpD0TGqdJyW9YGbTJE139+2p5ZskPWdmUySVuPuLkuTu3ZKUer1d7t6Wur9PUrmk1zP/YwHINpQzALnCJG1y96/9xkKzb1y13nC/0+7KoNv94vcrgGHisCaAXPFzSQ+a2RxJMrOZZlam5O/BB1PrPCTpdXc/L+msmf1Javkjkra7+0VJbWb2QOo1isxs4pj+FACyHn/ZAcgJ7n7AzL4u6admNk5Sr6RHJXVJWpl67KSS56VJ0npJ/5EqX4clbUgtf0TSd83sW6nX+MwY/hgAcoC5D3cPPgDEn5l1uvvk0DkA4Nc4rAkAABAh7DkDAACIEPacAQAARAjlDAAAIEIoZwAAABFCOQMAAIgQyhkAAECE/D9z8+xwaNyI4AAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(10, 10))\n",
        "plt.plot(history.history['loss'])\n",
        "plt.title('model loss')\n",
        "plt.ylabel('loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train'], loc='upper left')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "9dIpZk6AK8Zl",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 621
        },
        "outputId": "8fdb87a8-5742-4aea-a50a-ba55a93c88cf"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 720x720 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAJcCAYAAACxEXM4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3jW9b3/8dfnzk7IICE3I+yVgGzClJU4irgXLtwKqFVbrT3t6ek45/S0WlurVAXBvXDWPYvsLSAgyN5hZUAWJCHj8/sjaX9oGQFy53OP5+O6cjXJ93vfecXrKtcr38/n+/4aa60AAADQuDyuAwAAAIQiShgAAIADlDAAAAAHKGEAAAAOUMIAAAAcoIQBAAA4QAkDEBKMMS8aY35fz3O3G2POPdP3AYAToYQBAAA4QAkDAABwgBIGwG/ULQM+ZIxZbYw5ZIx5zhjT3BjzmTGmxBgzwxjT9KjzLzHGrDXGFBpjZhtjuh11rK8xZkXd696UFP2Dn3WRMWZl3WsXGmN6nWbmO40xm40xB4wxHxpjWtV93xhj/mqMyTXGFBtjvjXG9Kg7NsYY811dtt3GmJ+d1n8wAAGNEgbA31wp6TxJXSVdLOkzSf8pKVW1/2bdJ0nGmK6Spkv6Sd2xTyV9ZIyJNMZESnpf0iuSkiW9Xfe+qnttX0nPS5ogKUXSM5I+NMZEnUpQY0y2pD9KGiuppaQdkt6oO3y+pBF1v0di3TkFdceekzTBWhsvqYekmafycwEEB0oYAH/zN2vtfmvtbknzJC2x1n5jrS2X9J6kvnXnXSPpE2vtP6y1lZL+LClG0lBJgyVFSHrcWltprX1H0tdH/Yzxkp6x1i6x1lZba1+SVFH3ulNxg6TnrbUrrLUVkn4paYgxpr2kSknxkjIkGWvtOmvt3rrXVUrqboxJsNYetNauOMWfCyAIUMIA+Jv9R31edoyvm9R93kq1V54kSdbaGkm7JKXVHdttrbVHvXbHUZ+3k/Rg3VJkoTGmUFKbutedih9mKFXt1a40a+1MSU9KekpSrjFmqjEmoe7UKyWNkbTDGDPHGDPkFH8ugCBACQMQqPaotkxJqt2DpdoitVvSXklpdd/7p7ZHfb5L0v9Za5OO+oi11k4/wwxxql3e3C1J1tpJ1tr+krqrdlnyobrvf22tvVSSV7XLpm+d4s8FEAQoYQAC1VuSLjTGnGOMiZD0oGqXFBdKWiSpStJ9xpgIY8wVkgYe9dppkiYaYwbVbaCPM8ZcaIyJP8UM0yXdaozpU7ef7A+qXT7dbowZUPf+EZIOSSqXVFO3Z+0GY0xi3TJqsaSaM/jvACBAUcIABCRr7QZJ4yT9TVK+ajfxX2ytPWKtPSLpCkm3SDqg2v1jfz/qtcsk3ana5cKDkjbXnXuqGWZI+rWkd1V79a2TpGvrDieotuwdVO2SZYGkR+uO3ShpuzGmWNJE1e4tAxBizPe3TAAAAKAxcCUMAADAAUoYAACAA5QwAAAAByhhAAAADoS7DnCqmjVrZtu3b+86BgAAwEktX74831qbeqxjAVfC2rdvr2XLlrmOAQAAcFLGmB3HO8ZyJAAAgAOUMAAAAAcoYQAAAA4E3J6wY6msrFROTo7Ky8tdR/G56OhotW7dWhEREa6jAACAMxAUJSwnJ0fx8fFq3769jDGu4/iMtVYFBQXKyclRhw4dXMcBAABnICiWI8vLy5WSkhLUBUySjDFKSUkJiSt+AAAEu6AoYZKCvoD9U6j8ngAABLugKWEAAACBhBLWAAoLC/X000+f8uvGjBmjwsJCHyQCAAD+jhLWAI5Xwqqqqk74uk8//VRJSUm+igUAAPxYUNwd6dovfvELbdmyRX369FFERISio6PVtGlTrV+/Xhs3btRll12mXbt2qby8XPfff7/Gjx8v6f8/gqm0tFQXXHCBhg0bpoULFyotLU0ffPCBYmJiHP9mAADAV4KuhP33R2v13Z7iBn3P7q0S9NuLzzru8Ycfflhr1qzRypUrNXv2bF144YVas2bNv8ZIPP/880pOTlZZWZkGDBigK6+8UikpKd97j02bNmn69OmaNm2axo4dq3fffVfjxo1r0N8DAAD4j6ArYf5g4MCB35vjNWnSJL333nuSpF27dmnTpk3/VsI6dOigPn36SJL69++v7du3N1peAADQ+IKuhJ3oilVjiYuL+9fns2fP1owZM7Ro0SLFxsZq1KhRx5zzFRUV9a/Pw8LCVFZW1ihZAQCAG2zMbwDx8fEqKSk55rGioiI1bdpUsbGxWr9+vRYvXtzI6QAAgD8KuithLqSkpOjss89Wjx49FBMTo+bNm//r2OjRozVlyhR169ZN6enpGjx4sMOkAADAXxhrresMpyQzM9MuW7bse99bt26dunXr5ihR4wu13xcAgEBljFlurc081jGWIwEAAByghAEAADgQNCUs0JZVT1eo/J4AAAS7oChh0dHRKigoCPqCYq1VQUGBoqOjXUcBAABnKCjujmzdurVycnKUl5fnOorPRUdHq3Xr1q5jAACAMxQUJSwiIuJ7E+rPxJrdRfrx6yv0l7G91b9dcoO8JwAAwA8FxXJkQ2rTNFa7DpZp5vpc11EAAEAQo4T9QGJshPq3bapZ64N/aRMAALhDCTuGrAyvvttbrH1F//6MRwAAgIZACTuG7AyvJGnWBpYkAQCAb1DCjqFr8yZKS4phXxgAAPAZStgxGGOUlZGqBZvzVVFV7ToOAAAIQpSw48hK9+rwkWot3XbAdRQAABCEKGHHMbRTM0WFe1iSBAAAPkEJO46YyDAN6ZSiWZQwAADgA5SwE8jO8Gp7wWFtzSt1HQUAAAQZStgJZKXXjqpgSRIAADQ0StgJtEmOVWdvE83ewPR8AADQsChhJ5Gd4dWSbQUqrahyHQUAAAQRSthJZKV7VVltNX9TvusoAAAgiFDCTiKzfVPFR4dzlyQAAGhQlLCTiAjzaESXVM3akCtrres4AAAgSFDC6mFUeqpySyq0dk+x6ygAACBIUMLqYVTdqAqWJAEAQEOhhNVDanyUerdO1MwNlDAAANAwKGH1lJXh1cpdhSoorXAdBQAABAFKWD1lZ3hlrTRnI4NbAQDAmaOE1VOPVolq1iSKRxgBAIAGQQmrJ4/HaFR6quZuzFNVdY3rOAAAIMBRwk5BdoZXxeVVWrGz0HUUAAAQ4Chhp2BYl2YK9xiWJAEAwBmjhJ2ChOgIDWifzLwwAABwxihhpyg7w6sN+0u0u7DMdRQAABDAKGGnKCsjVRLT8wEAwJnxWQkzxjxvjMk1xqw5zvFEY8xHxphVxpi1xphbfZWlIXVKbaI2yTGUMAAAcEZ8eSXsRUmjT3D8HknfWWt7Sxol6S/GmEgf5mkQxhhlp3u1YEu+yiurXccBAAABymclzFo7V9KBE50iKd4YYyQ1qTu3yld5GlJWhlfllTVatLXAdRQAABCgXO4Je1JSN0l7JH0r6X5r7TGnoBpjxhtjlhljluXluX9s0OCOKYqO8LAkCQAATpvLEvYjSSsltZLUR9KTxpiEY51orZ1qrc201mampqY2ZsZjio4I09mdmmnm+lxZa13HAQAAAchlCbtV0t9trc2StknKcJjnlGRleJVzsExb8kpdRwEAAAHIZQnbKekcSTLGNJeULmmrwzynJCvDK0lMzwcAAKfFlyMqpktaJCndGJNjjLndGDPRGDOx7pT/lTTUGPOtpK8k/Ye1Nt9XeRpaWlKMMlrEU8IAAMBpCffVG1trrzvJ8T2SzvfVz28MWRleTZu7VcXllUqIjnAdBwAABBAm5p+BrHSvqmqs5m8KmAt4AADAT1DCzkC/tklKjIlgSRIAAJwyStgZCA/zaETXVM3ekKuaGkZVAACA+qOEnaHsjFTllx7Rt7uLXEcBAAABhBJ2hkZ29coYRlUAAIBTQwk7Q8lxkerTJkmzN1DCAABA/VHCGkB2ulercoqUV1LhOgoAAAgQlLAG8M/p+VwNAwAA9UUJawBntUpQ84QozaKEAQCAeqKENQBjjLLSvZq3MV+V1TWu4wAAgABACWsgo9K9Kqmo0tfbD7iOAgAAAgAlrIEM69JMEWFGszfkuY4CAAACACWsgTSJCtegDinMCwMAAPVCCWtAWRlebc4t1a4Dh11HAQAAfo4S1oCy60ZVcDUMAACcDCWsAXVoFqcOzeIoYQAA4KQoYQ1sVHqqFm0tUNmRatdRAACAH6OENbDsDK+OVNVo4ZZ811EAAIAfo4Q1sIEdkhUbGcaSJAAAOCFKWAOLCg/TsM7NNGt9rqy1ruMAAAA/RQnzgewMr/YUlWvD/hLXUQAAgJ+ihPnAqPTaURWz1jM9HwAAHBslzAdaJEare8sEzWJfGAAAOA5KmI9kZ3i1fOdBFR2udB0FAAD4IUqYj2RleFVdYzVnE0uSAADg31HCfKRPmyQ1jY1gSRIAABwTJcxHwjxGI7umas7GPFXXMKoCAAB8HyXMh7IyvDpw6IhW5RS6jgIAAPwMJcyHRnZNlceIJUkAAPBvKGE+lBQbqf7tmvIIIwAA8G8oYT6WleHV2j3F2l9c7joKAADwI5QwH8uqm54/ewNXwwAAwP9HCfOxjBbxapkYzZIkAAD4HkqYjxljlJXh1fxN+aqoqnYdBwAA+AlKWCPITvfq0JFqfb3toOsoAADAT1DCGsHQzimKDPewJAkAAP6FEtYIYiPDNaRjimaxOR8AANShhDWSrPRUbcs/pG35h1xHAQAAfoAS1kiyM5pLYno+AACoRQlrJG1TYtUpNY4lSQAAIIkS1qiyM7xasvWADlVUuY4CAAAco4Q1oqwMr45U12j+5nzXUQAAgGOUsEaU2S5ZTaLCeYQRAACghDWmyHCPhndpplnr82StdR0HAAA4RAlrZFkZXu0rLtd3e4tdRwEAAA5RwhrZqPRUSYyqAAAg1FHCGpk3Plq9WifyCCMAAEIcJcyBUelefbOrUAcOHXEdBQAAOEIJcyA7wytrpbkb81xHAQAAjlDCHOiVlqhmTSJZkgQAIIRRwhzweIxGdvVqzsY8VVXXuI4DAAAcoIQ5kp3hVVFZpb7ZVeg6CgAAcIAS5siwLs0U5jGMqgAAIERRwhxJjIlQZrum7AsDACBEUcIcys7wav2+Eu0pLHMdBQAANDJKmEPZGV5J0iwe6A0AQMihhDnU2dtErZvGsC8MAIAQRAlzyBijrHSvFmwuUHlltes4AACgEVHCHMvO8KqsslpLth1wHQUAADQiSphjQzqlKDrCw5IkAAAhhhLmWHREmIZ2aqaZ63NlrXUdBwAANBJKmB/IyvBq54HD2pJ3yHUUAADQSChhfiArPVWSNJtRFQAAhAxKmB9o3TRWXZs3YXo+AAAhhBLmJ7IyvFq67YBKyitdRwEAAI2AEuYnstO9qqqxmr8p33UUAADQCChhfqJ/u6ZKiA5nSRIAgBBBCfMT4WEejeiaqlkb8lRTw6gKAACCHSXMj2Sle5VfWqG1e4pdRwEAAD5GCfMjo9JTZYxYkgQAIARQwvxISpMo9W6dpJnMCwMAIOhRwvxMdoZXq3MKlV9a4ToKAADwIUqYn8nO8MpaafaGPNdRAACAD1HC/Ez3lglKjY/SLJYkAQAIapQwP+PxGGWlp2ruxjxVVte4jgMAAHyEEuaHsjO8Kimv0vIdB11HAQAAPkIJ80PDuqQqIsxoFqMqAAAIWpQwP9QkKlwDOyQzLwwAgCBGCfNTWelebcot1a4Dh11HAQAAPkAJ81NZGV5J0mzukgQAIChRwvxUx2ZxapcSy5IkAABBihLmp4wxykr3auGWApUdqXYdBwAANDBKmB/LzvCqoqpGi7bmu44CAAAamM9KmDHmeWNMrjFmzQnOGWWMWWmMWWuMmeOrLIFqYIdkxUSEadZ6HmEEAECw8eWVsBcljT7eQWNMkqSnJV1irT1L0tU+zBKQoiPCdHbnZpq5PlfWWtdxAABAA/JZCbPWzpV04ASnXC/p79banXXnswP9GLIzvNpdWKZNuaWuowAAgAbkck9YV0lNjTGzjTHLjTE3He9EY8x4Y8wyY8yyvLzQWprLykiVJO6SBAAgyLgsYeGS+ku6UNKPJP3aGNP1WCdaa6daazOttZmpqamNmdG5lokx6tYygRIGAECQcVnCciR9Ya09ZK3NlzRXUm+HefxWdkaqlu84qKKyStdRAABAA3FZwj6QNMwYE26MiZU0SNI6h3n8Vla6V9U1VvM2hdZSLAAAwcyXIyqmS1okKd0Yk2OMud0YM9EYM1GSrLXrJH0uabWkpZKetdYed5xFKOvbtqmSYiNYkgQAIIiE++qNrbXX1eOcRyU96qsMwSLMYzSya6rmbMhTTY2Vx2NcRwIAAGeIifkBIjvDq4JDR7Qqp9B1FAAA0AAoYQFiRJdUeYw0iyVJAACCAiUsQDSNi1Tftk01awOb8wEACAaUsACSneHVt7uLlFtc7joKAAA4Q5SwAJKV7pUkzeZqGAAAAY8SFkC6tYxXy8RoRlUAABAEKGEBxBijUelezd+cryNVNa7jAACAM0AJCzBZ6akqrajSsu0HXEcBAABngBIWYM7u3EyRYR6WJAEACHCUsAATFxWuQR2TNXMDJQwAgEBGCQtA2Rlebc07pB0Fh1xHAQAAp4kSFoD+OaqCJUkAAAIXJSwAtW8Wp47N4pieDwBAAKOEBaisDK8Wby3Q4SNVrqMAAIDTQAkLUNkZXh2pqtGCzQWuowAAgNNACQtQA9onq0lUOPvCAAAIUJSwABUZ7tGwzs00e0OurLWu4wAAgFNECQtg2Rle7S0q1/p9Ja6jAACAU0QJC2Cj0lMlMaoCAIBARAkLYN6EaPVIS9AsShgAAAGHEhbgstO9WrHzoA4eOuI6CgAAOAWUsACXleFVjZXmbmJwKwAAgYQSFuB6t05SSlwkS5IAAAQYSliA83iMRnZN1ZyNeaquYVQFAACBghIWBLIyvDp4uFIrdx10HQUAANQTJSwIjOiaqjCPYVQFAAABhBIWBBJjItS/XVPNXM/mfAAAAgUlLEhkZ3i1bm+x9haVuY4CAADqgRIWJLLSvZKk2Ru4GgYAQCCghAWJrs2bKC0phn1hAAAECEpYkDDGKCsjVQs256uiqtp1HAAAcBKUsCCSneHV4SPVWrL1gOsoAADgJChhQWRIx2aKCvewJAkAQACghAWRmMgwDemUotkbKGEAAPg7SliQyc7wanvBYW3NK3UdBQAAnAAlLMj8c1QFS5IAAPg3SliQaZMcqy7eJprFkiQAAH6NEhaEsjO8WrrtgEorqlxHAQAAx0EJC0Kj0r2qrLaavynfdRQAAHAclLAglNm+qeKjwzWLfWEAAPgtSlgQigjzaESXVM3akCtrres4AADgGChhQSorw6vckgqt3VPsOgoAADgGSliQGpWeKmMYVQEAgL+ihAWpZk2i1Kt1EqMqAADwU5SwIJaVnqqVuwpVUFrhOgoAAPgBSlgQy87wylppzsY811EAAMAPUMKCWI9WiWrWJIp9YQAA+CFKWBDzeIyy0lM1d2OeqqprXMcBAABHoYQFuewMr4rLq7RiZ6HrKAAA4CiUsCB3dpdmCvcYliQBAPAzlLAglxAdoQHtk3mEEQAAfoYSFgKyM7zasL9EuwvLXEcBAAB1KGEhICvDK4np+QAA+BNKWAjolBqntsmxLEkCAOBHKGEhwJjaURULt+SrvLLadRwAACBKWMjIyvCqvLJGi7YWuI4CAABECQsZgzumKCYijCVJAAD8BCUsRERHhOnszimauT5X1lrXcQAACHmUsBCSleFVzsEybc4tdR0FAICQRwkLIaPSa0dVzNrAkiQAAK5RwkJIWlKMMlrEMy8MAAA/QAkLMVkZXi3bflDF5ZWuowAAENIoYSEmO8OrqhqreRvzXUcBACCkUcJCTN82SUqMidC7K3K4SxIAAIcoYSEmPMyju0d10sz1uZo6d6vrOAAAhCxKWAgaP6KjLuzVUo98vl5zN+a5jgMAQEiihIUgY4wevaqXujaP173Tv9GOgkOuIwEAEHIoYSEqNjJcU2/MlCRNeGW5DlVUOU4EAEBooYSFsLYpsXry+r7auL9ED72zio36AAA0IkpYiBveJVW/uCBDn367T5PnbHEdBwCAkEEJg+4c3lGX9G6lR7/YwCONAABoJJQwyBijR67spW4tEnT/9G+0LZ+N+gAA+BolDJKkmMgwPXNjf4V5jMa/vEylbNQHAMCnKGH4lzbJsXrq+n7amn9ID761UjU1bNQHAMBXKGH4nqGdm+k/x3TTF2v366lZm13HAQAgaFHC8G9uO7u9Lu+bpsdmbNRX6/a7jgMAQFCihOHfGGP0xyt66qxWCfrJGyu1Ja/UdSQAAIIOJQzHFB0RpmduzFRkuEfjX16mkvJK15EAAAgqlDAcV1pSjJ68vp+2FxzWT99cxUZ9AAAaECUMJzSkU4p+fWE3zVi3X5NmbnIdBwCAoEEJw0ndPLS9ruzXWo/P2KQv1+5zHQcAgKBACcNJGWP0f5f3UO/WiXrgrVXanMtGfQAAzhQlDPUSHRGmKTf2V3RE7Ub9YjbqAwBwRnxWwowxzxtjco0xa05y3gBjTJUx5ipfZUHDaJkYo6dv6K+dBw7rp28wUR8AgDPhyythL0oafaITjDFhkh6R9KUPc6ABDeyQrN9e3F1frc/V4zM2uo4DAEDA8lkJs9bOlXTgJKfdK+ldSbm+yoGGN25wO43NbK1JMzfr8zV7XccBACAgOdsTZoxJk3S5pMn1OHe8MWaZMWZZXl6e78PhhIwx+p9Le6hPmyQ9+NYqbdxf4joSAAABx+XG/Mcl/Ye1tuZkJ1prp1prM621mampqY0QDScTHRGmKeP6KzYqXONfXqaiw2zUBwDgVLgsYZmS3jDGbJd0laSnjTGXOcyDU9QiMVqTb+in3YVluv/Nb1TNRn0AAOrNWQmz1naw1ra31raX9I6ku62177vKg9OT2T5Zv7vkLM3ekKe/fLnBdRwAAAJGuK/e2BgzXdIoSc2MMTmSfispQpKstVN89XPR+G4Y1E5rdhfr6dlbdFarRF3Yq6XrSAAA+D2flTBr7XWncO4tvsqBxvG7S7prw75i/eztVerkjVNGiwTXkQAA8GtMzEeDiAqv3agfHx2u8S8vV+HhI64jAQDg1yhhaDDehGhNubG/9hWV697pbNQHAOBEKGFoUP3aNtX/XHqW5m3K15++WO86DgAAfstne8IQuq4d2FZr9hTpmTlb1aNVoi7u3cp1JAAA/A5XwuATv7noLA1o31QPvbNK3+0pdh0HAAC/QwmDT0SGe/TUDf2UFBOp8a8s08FDbNQHAOBolDD4jDe+dqN+bkmFfjx9haqqT/qEKgAAQgYlDD7Vp02Sfn9ZDy3YXKBHPmejPgAA/8TGfPjc2Mw2Wru7SNPmbVOPtERd2ifNdSQAAJzjShgaxX9d1F0DOyTr5++s1prdRa7jAADgHCUMjSIizKOnb+inlLhITXhluQpKK1xHAgDAKUoYGk2zJlF65sZM5ZdW6Mevf6NKNuoDAEIYJQyNqmfrRP3xip5atLVAf/h0nes4AAA4w8Z8NLor+rXWmt3Fen7BNvVolagr+7d2HQkAgEbHlTA48Z9jMjSkY4p++d63Wp1T6DoOAACNjhIGJ8LDPHry+r5KbRKlCa8sV14JG/UBAKGFEgZnUppE6Zkb++vg4SO65/UVbNQHAIQUShic6pGWqEeu7KWl2w7o9x9/5zoOAACNho35cO7SPmlaUzdR/6y0RI3NbOM6EgAAPseVMPiF/xidobM7p+i/3lujlbvYqA8ACH71KmHGmPuNMQmm1nPGmBXGmPN9HQ6hIzzMoyev6ydvQpQmvrJcuSXlriMBAOBT9b0Sdpu1tljS+ZKaSrpR0sM+S4WQ1DQuUlNvzFRh2RHd/eoKHalioz4AIHjVt4SZuv8dI+kVa+3ao74HNJjurRL06FW9tWzHQf3Px2tdxwEAwGfqW8KWG2O+VG0J+8IYEy+JyxTwiYt7t9KEkR316uKdemPpTtdxAADwifreHXm7pD6StlprDxtjkiXd6rtYCHU//1GGvttTrN98sFZdmserf7umriMBANCg6nslbIikDdbaQmPMOEn/JanId7EQ6sI8Rn+7rq9aJEbrrleXa38xG/UBAMGlviVssqTDxpjekh6UtEXSyz5LBUhKio3U1Jv6q7SiSne9ulwVVdWuIwEA0GDqW8KqrLVW0qWSnrTWPiUp3nexgFoZLRL056t7a8XOQv3uQzbqAwCCR31LWIkx5peqHU3xiTHGIynCd7GA/29Mz5a6e1QnTV+6S68t2eE6DgAADaK+JewaSRWqnRe2T1JrSY/6LBXwAw+en65R6an63YdrtWz7AddxAAA4Y/UqYXXF6zVJicaYiySVW2vZE4ZGE+YxeuLavkpLitHEV1doXxEb9QEAga2+jy0aK2mppKsljZW0xBhzlS+DAT+UGBOhqTdlquxIlSa8ulzllWzUBwAErvouR/5K0gBr7c3W2pskDZT0a9/FAo6ta/N4/WVsH63aVajffLBGtfeLAAAQeOpbwjzW2tyjvi44hdcCDWp0jxa6L7uz3lqWo1cXs1EfABCY6jsx/3NjzBeSptd9fY2kT30TCTi5n5zbVWv3FOu/P/pOXZvHa1DHFNeRAAA4JfXdmP+QpKmSetV9TLXW/ocvgwEn4vEY/fXaPmqbHKt7Xl+hPYVlriMBAHBK6r2kaK1911r7QN3He74MBdRHQnSEpt7UX+WVNZrIRn0AQIA5YQkzxpQYY4qP8VFijClurJDA8XT2xuuv1/TR6pwi/eo9NuoDAALHCUuYtTbeWptwjI94a21CY4UETuS87s31k3O76N0VOXpx4XbXcQAAqBfucERQuC+7i87r3ly//2SdFm0pcB0HAICTooQhKHg8Ro+N7a32KbUb9XMOHnYdCQCAE6KEIWjER9dO1K+sqtH4l5er6HCl60gAABwXJQxBpVNqE026vq825Zbo2mmLlV9a4ToSAADHRAlD0MlK9+rZmwdoW36pxk5ZxAwxAIBfooQhKI3smqpXbh+kvJIKXT1lkbblH3IdCQCA76GEIWgNaJ+s6eMHq6yyWldPWaR1e5Ld4x8AACAASURBVBltBwDwH5QwBLUeaYl6a8JghXuMrnlmkb7ZedB1JAAAJFHCEAI6e+P19sQhahoXqRueXaKFW/JdRwIAgBKG0NAmOVZvTxii1k1jdMsLX2vGd/tdRwIAhDhKGEKGNyFab44foowW8Zrw6nJ9sHK360gAgBBGCUNIaRoXqdfuGKTMdk31kzdX6rUlO1xHAgCEKEoYQk58dIReum2gstK9+tV7azRlzhbXkQAAIYgShpAUHRGmKeP666JeLfXwZ+v15y82yFrrOhYAIISEuw4AuBIZ7tET1/ZVk6hwPTlrs0orqvSbi7rL4zGuowEAQgAlDCEtzGP0xyt6Kj46XNPmbVNJeZUeubKnwsO4SAwA8C1KGEKeMUb/Oaab4qMj9Ng/NupQRZWeuK6PosLDXEcDAAQx/twHVFvE7juni35zUXd9vnaf7nhpmQ4fqXIdCwAQxChhwFFuG9ZBf7qylxZsztdNzy1VUVml60gAgCBFCQN+YOyANvrbdf20KqdQ109brILSCteRAABBiBIGHMOFvVpq2k2Z2pJXqrHPLNLeojLXkQAAQYYSBhzHqHSvXr5tkPYXV+iqyYu0Pf+Q60gAgCBCCQNOYGCHZE2/c7AOH6nS1c8s0oZ9Ja4jAQCCBCUMOImerRP11oQhMpKumbpIK3cVuo4EAAgClDCgHro0j9c7E4cqPjpcN0xbrEVbClxHAgAEOEoYUE9tU2L19oShapUUo1teWKqZ6/e7jgQACGCUMOAUtEiM1psThqhr83iNf3m5Plq1x3UkAECAooQBpyg5LlKv3zlI/do11X1vfKPpS3e6jgQACECUMOA0xEdH6KVbB2pEl1T98u/fatrcra4jAQACDCUMOE0xkWGadlOmLuzZUv/36To99uUGWWtdxwIABIhw1wGAQBYZ7tGk6/oqLipMk2ZuVklFlX59YXd5PMZ1NACAn6OEAWcozGP08BW91CQqQs8v2KbS8io9fGUvhVHEAAAnQAkDGoDHY/Tri7opPjpcT3y1SYeOVOnxa/oqMpwVfwDAsVHCgAZijNFPz+uq+Ohw/f6TdTpUsUxTxvVXTGSY62gAAD/En+lAA7tjeEc9cmVPzd2Up5ufX6ri8krXkQAAfogSBvjANQPaatK1fbVi50FdP22xDhw64joSAMDPUMIAH7m4dytNvam/Nu0v1dhnFmlfUbnrSAAAP0IJA3woO6O5XrptoPYWlunqZxZqZ8Fh15EAAH6CEgb42OCOKXr9zsEqKa/SVVMWauP+EteRAAB+gBIGNILebZL05vghkqRrnlmk1TmFjhMBAFyjhAGNJL1FvN6eOERxUeG6ftoSLdla4DoSAMAhShjQiNqlxOmdiUPVPCFKNz2/VLM25LqOBABwhBIGNLIWidF6a8IQdfY20fiXl+mT1XtdRwIAOEAJAxxIaRKl6eMHq0+bJN07fYXe+nqX60gAgEbmsxJmjHneGJNrjFlznOM3GGNWG2O+NcYsNMb09lUWwB8lREfo5dsGaViXVP383dV6bv4215EAAI3Il1fCXpQ0+gTHt0kaaa3tKel/JU31YRbAL8VEhmnaTf11QY8W+t+Pv9PjMzbKWus6FgCgEfishFlr50o6cILjC621B+u+XCypta+yAP4sKjxMf7uur67q31qPz9ik33+yjiIGACEg3HWAOrdL+ux4B40x4yWNl6S2bds2Viag0YSHefSnK3upSVS4npu/TYcqqvR/l/dUmMe4jgYA8BHnJcwYk6XaEjbseOdYa6eqbrkyMzOTSwQISh6P0W8v7q6E6HBNmrlZpRVVemxsH0WGc/8MAAQjpyXMGNNL0rOSLrDWMrkSIc8YowfOT1eT6HD94dP1OlRRpcnj+is6Isx1NABAA3P2J7Yxpq2kv0u60Vq70VUOwB+NH9FJf7i8p2ZvzNPNzy9VSXml60gAgAbmyxEV0yUtkpRujMkxxtxujJlojJlYd8pvJKVIetoYs9IYs8xXWYBAdP2gtnr8mj5avuOgbnh2iQ4eOuI6EgCgAZlAuwsrMzPTLltGX0Po+Grdft312gq1S47Vq3cMUvOEaNeRAAD1ZIxZbq3NPNYxdvwCfu6cbs314q0DtKewTFdPWaRdBw67jgQAaACUMCAADO3UTK/dOVhFZZW6esoibc4tcR0JAHCGKGFAgOjTJklvThisqhqrsc8s1vIdx52FDAAIAJQwIIBktEjQOxOHKD46XGOfWawpc7aopiaw9nUCAGpRwoAA075ZnD66d5hGn9VCD3+2Xre99LUOcOckAAQcShgQgBKiI/Tk9X31v5f10MItBRrzxDwt3cbyJAAEEkoYEKCMMbpxcDu9d/dQxUSG6bppi/XUrM0sTwJAgKCEAQHurFaJ+vDHZ2tMz5Z69IsNuvmFpcovrXAdCwBwEpQwIAjER0do0rV99IfLe2rJtgMa88Q8LdrC41gBwJ9RwoAgYYzR9YPa6oN7zlaTqHDd8OxiPTFjk6pZngQAv0QJA4JMt5YJ+ujeYbq0T5r+OmOjbnp+iXJLyl3HAgD8ACUMCEJxUeF6bGxv/enKXlq+46DGPDFfCzbnu44FADgKJQwIUsYYjR3QRh/cM0xJsREa99wSPfaPjSxPAoCfoIQBQS69Rbw+/PHZuqJva036apNueHax9hezPAkArlHCgBAQGxmuv4ztrT9f3VurdhVpzBPzNHdjnutYABDSKGFACLmqf2t9+OOzldIkUje/sFSPfrFeVdU1rmMBQEiihAEhpkvzeH1wzzCN7d9GT83aouunLdG+IpYnAaCxUcKAEBQTGaZHruqlx6/pozV7ijRm0jzN3pDrOhYAhBRKGBDCLuubpo/uHSZvfJRueeFrPfzZelWyPAkAjYISBoS4TqlN9P49Z+v6QW01Zc4WXTt1sfYUlrmOBQBBjxIGQNERYfrD5T016bq+2rCvRGMmzdNX6/a7jgUAQY0SBuBfLundSh/dO0xpSTG6/aVl+r9PvmN5EgB8hBIG4Hs6NIvTu3cN1U1D2mnavG26esoi5Rw87DoWAAQdShiAfxMdEab/ubSHnr6hn7bklmrME/P05dp9rmMBQFChhAE4rjE9W+rj+4apXUqcxr+yXP/90VodqWJ5EgAaAiUMwAm1S4nTO3cN0S1D2+uFBdt19ZSF2nWA5UkAOFOUMAAnFRUept9dcpamjOuvbfmHNGbSPH2+Zq/rWAAQ0ChhAOptdI8W+uS+4eqY2kQTX12h336wRhVV1a5jAUBAooQBOCVtkmP19oQhumNYB720aIeunLxQ2/MPuY4FAAGHEgbglEWGe/RfF3XXszdlateBMl30t/n6ePUe17EAIKBQwgCctnO7N9en9w9Xl+ZN9OPXv9Gv3vtW5ZUsTwJAfVDCAJyRtKQYvTVhiCaM6KjXluzU5U8v1Na8UtexAMDvUcIAnLGIMI9+Oaabnr8lU/uKynTx3+brg5W7XccCAL9GCQPQYLIzapcnu7VM0P1vrNQv3l3N8iQAHAclDECDapkYozfGD9bdozrpja936dInF2hzLsuTAPBDlDAADS48zKOfj87QS7cNVF5phS7+23y9uzzHdSwA8CuUMAA+M7Jrqj69b7h6tk7Ug2+v0kNvr9LhI1WuYwGAX6CEAfCpFonRev2OQbo3u7PeWZGjS59coI37S1zHAgDnKGEAfC48zKMHz0/XK7cN0sHDR3TJk/P11rJdsta6jgYAzlDCADSaYV2a6dP7hqtvm6b6+Tur9eBbq3SoguVJAKGJEgagUXkTovXqHYP0k3O76L2Vu3XJk/O1fl+x61gA0OgoYQAaXZjH6CfndtVrdwxScXmVLn1ygaYv3cnyJICQQgkD4MzQTrXLkwM7JOuXf/9W97+xUqUsTwIIEZQwAE6lxkfppVsH6mfnd9XHq/fo4r/N19o9Ra5jAYDPUcIAOOfxGP04u4um3zlYh49U6fKnF+qFBdtUXcPyJIDgRQkD4DcGdUzRp/cN19BOKfrvj77TFZMXclUMQNCihAHwKylNovTCLQP0xLV9tPvgYV3y5AL9/uPvGGUBIOhQwgD4HWOMLu2Tpq8eGKVrBrTRs/O36bzH5ujLtftcRwOABkMJA+C3EmMj9IfLe+rdu4YqISZC419ZrjtfXqY9hWWuowHAGaOEAfB7/ds11Uf3DtMvL8jQ/E35OvexOXp23lZVVde4jgYAp40SBiAgRIR5NGFkJ3350xEa3DFFv/9knS55coFW7ip0HQ0ATgslDEBAaZMcq+duztTkG/qp4FCFLn96gX79/hoVl1e6jgYAp4QSBiDgGGN0Qc+WmvHASN0ytL1eW7JD5/xljj5atYdHHwEIGJQwAAErPjpCv734LH1wzzC1SIjWvdO/0c0vfK2dBYddRwOAk6KEAQh4PVsn6v17ztbvLu6uFTsO6ry/ztFTszbrSBUb9wH4L0oYgKAQ5jG65ewOmvHASJ3TzatHv9igCyfN09JtB1xHA4BjooQBCCotEqP19A399fwtmTp8pFpjn1mkn7+zSgcPHXEdDQC+hxIGIChlZzTXPx4YoQkjO+rvK3brnMfm6J3lOWzcB+A3KGEAglZsZLh+eUE3fXzfMHVoFqefvb1K101brM25pa6jAQAlDEDwy2iRoLcnDNEfr+ip7/YU64In5uqxLzeovLLadTQAIYwSBiAkeDxG1w1sq5k/G6WLerXSpJmbNfrxuZq/Kd91NAAhihIGIKQ0axKlv17TR6/ePkjGGI17bonuf+Mb5ZVUuI4GIMRQwgCEpGFdmumz+4frvnO66LNv9+mcv8zWa0t2qKaGjfsAGgclDEDIio4I0wPnddVnPxmu7q0S9Kv31uiqKQu1fl+x62gAQgAlDEDI65TaRNPvHKzHxvbW9oLDunDSfP3x03U6fKTKdTQAQYwSBgCqfSj4Ff1aa+aDI3V1/9Z6Zu5WnffYXH21br/raACCFCUMAI6SFBuph6/spbcnDlFcVJhuf2mZJr6yXHuLylxHAxBkKGEAcAwD2ifr43uH66EfpWvWhlyd+5c5en7+NlWzcR9AA6GEAcBxRIZ7dE9WZ/3jpyOV2T5Z//Pxd7r0qflanVPoOhqAIEAJA4CTaJsSqxdvHaAnr++r/cUVuuypBfrdh2tVUl7pOhqAAEYJA4B6MMbool6t9NWDI3Xj4HZ6adF2nfvYHH367V4eCg7gtFDCAOAUJERH6L8v7aH37z5bzZpE6e7XVui2F7/WrgOHXUcDEGAoYQBwGnq3SdIH95ytX1/UXUu2HdB5f52jybO3qLK6xnU0AAGCEgYApyk8zKPbh3XQjAdGakSXVD3y+XpdNGm+lu844DoagABACQOAM9QqKUZTb8rUtJsyVVJeqSsnL9Iv/75ahYePuI4GwI9RwgCggZzXvbn+8cBI3Tm8g95alqNz/jJH732Tw8Z9AMdECQOABhQXFa5fXdhdH/14mNokx+qnb67SuOeWaGteqetoAPwMJQwAfKB7qwT9/a6h+v1lPbQ6p0ijn5inJ2ZsUkVVtetoAPwEJQwAfMTjMRo3uJ2+enCkRp/VQn+dsVEXPD5PC7fku44GwA9QwgDAx7zx0Zp0XV+9fNtAVdVYXT9tiR54cyUPBQdCHCUMABrJiK6p+vKnI3Rvdmd9vHqvRj06W3/8bJ2KDvP4IyAUUcIAoBFFR4TpwfPT9dWDI3Vhz5aaOnerhv9ppp6Zs0XllewXA0KJCbRbpzMzM+2yZctcxwCABrFub7H+9Pl6zdqQp5aJ0frpuV11Rb80hYfxNzIQDIwxy621mcc65rP/lxtjnjfG5Bpj1hznuDHGTDLGbDbGrDbG9PNVFgDwV91aJuiFWwfqjfGD1TwhWj9/d7UueGKevly7j/liQJDz5Z9aL0oafYLjF0jqUvcxXtJkH2YBAL82uGOK3rt7qKaM66fqGqvxryzXVVMW6evtPAIJCFY+K2HW2rmSTvSvx6WSXra1FktKMsa09FUeAPB3xhiN7tFSX/50hP54RU/tOnBYV09ZpDte+lob95e4jgeggbncdJAmaddRX+fUfe/fGGPGG2OWGWOW5eXlNUo4AHAlPMyj6wa21ZyHsvTQj9K1ZNsBjX58rh56e5X2FDLWAggWAbHz01o71Vqbaa3NTE1NdR0HABpFTGSY7snqrLkPZen2YR30wco9GvXn2frjp+t4ODgQBFyWsN2S2hz1deu67wEAjtI0LlK/urC7Zj00Spf0bqWp87ZqxJ9mafJsxloAgcxlCftQ0k11d0kOllRkrd3rMA8A+LW0pBj9+ere+uz+4RrQPlmPfL5eox6drTeW7lRVdY3reABOkc/mhBljpksaJamZpP2SfispQpKstVOMMUbSk6q9g/KwpFuttScdAMacMACotXTbAT382Tqt2FmoTqlx+vnoDJ3fvblq/3kF4A9ONCeMYa0AEMCstfryu/360+frtSXvkPq1TdIvLuimgR2SXUcDIEfDWgEAvmeM0Y/OaqEvfjJCj1zZU3sKyzX2mUW6/cWvtWEfYy0Af8aVMAAIImVHqvXiwu16evZmlVZU6Yq+rfXA+V2VlhTjOhoQkliOBIAQU3j4iCbP3qIXFm6XJN08pJ3uHtVZTeMi3QYDQgwlDABC1J7CMv31Hxv17oocxUWGa+KoTrrt7A6KiQxzHQ0ICZQwAAhxG/eX6E+fb9CMdfvljY/ST87tqrGZrRUextZgwJfYmA8AIa5r83g9e3Om3p44RG2SY/Wf732r8x+fq8/X7FWg/TEOBAtKGACEkAHtk/XOxCGadlOmwozRxFdX6PKnF2rx1gLX0YCQQwkDgBBjjNF53Zvrs/uH609X9tL+4nJdO3Wxbn1hqdbtLXYdDwgZ7AkDgBBXXlmtlxZu11OzNqukokqX90nTT8/rqjbJsa6jAQGPjfkAgJMqOlypyXO26IUF22StNG5wO/04u7OSGWsBnDZKGACg3vYWlenxf2zS28t3KS4yXBNGdtRtwzooNjLcdTQg4FDCAACnbHNu7ViLL7/br9T4KN1/ThddM6CNIhhrAdQbIyoAAKesszdeU2/K1Lt3DVH7lFj91/trdP5f5+qT1Yy1ABoCJQwAcEL92yXrrQlD9NzNmYoIM7rn9RW67KkFWrgl33U0IKBRwgAAJ2WM0Tndmuuz+0foz1f3Vl5Jha6ftkQ3Pb9Ua/cUuY4HBCRKGACg3sI8Rlf1b62ZPxulX43pptU5hbpw0nz95I1vtOvAYdfxgIDCxnwAwGkrKqvUM3O26PkF21RdY3XDoHa6N7uzUppEuY4G+AXujgQA+NS+onI98dVGvfn1LsVGhuvO4R11+/AOahLFWAuENkoYAKBRbM4t1Z+/2KDP1+5TQnS4bh7aXrcMbc+VMYQsShgAoFGt2lWop2dv1hdr9ys6wqNrB7TVnSM6Ki0pxnU0oFFRwgAATmzOLdGUOVv1/je7JUmX9Gmlu0Z2Upfm8Y6TAY2DEgYAcGpPYZmmzduqN5buUllltc7r3lx3j+qkvm2buo4G+BQlDADgFw4cOqIXF27XSwu3q6isUkM6puiuUZ00vEszGWNcxwMaHCUMAOBXSiuq9MbSnZo2b6v2F1eoR1qC7hrZWaN7tFCYhzKG4EEJAwD4pYqqar3/zW5NmbNV2/IPqUOzOE0Y0VGX90tTVHiY63jAGaOEAQD8WnWN1Rdr9+np2Zu1ZnexmidE6Y5hHXXdoLbMGkNAo4QBAAKCtVbzN+dr8uwtWrilQIkxEbp5SDvdzKwxBChKGAAg4Hyz86Amz96iL79j1hgCFyUMABCwNueWaPLsrfpgZe2ssUv7pOmuUR3V2cusMfg/ShgAIODtLizTtLlb9cbXO1VeWaPzuzfX3Vmd1adNkutowHFRwgAAQePAoSN6ccE2vbRox79mjd2d1UnDOjNrDP6HEgYACDqlFVWavmSnnp1fO2usZ1qi7hrVST86i1lj8B+UMABA0KqoqtZ7K3brmbnMGoP/oYQBAIJedY3V52tqZ42t3VM7a+zO4R113cC2imPWGByhhAEAQoa1VvM21c4aW7S1btbY0Pa6ZWh7JcdFuo6HEEMJAwCEpBU7D2pK3ayxmIgwXTuwje4YzqwxNB5KGAAgpG3aX6LJc7bow5V7JEmX9U3TxJHMGoPvUcIAAJCUc/Cwnp23TW98vVMVVbWzxu4axawx+A4lDACAoxSUVujFhdv10sLtKi6v0tBOKbprFLPG0PAoYQAAHENpRZVeX7JDz87bptwSZo2h4VHCAAA4gX/OGpsyZ4u2FxxWx2ZxmjCyoy7v21qR4R7X8RDAKGEAANRDdY3VZ2v2avLsLVq7p1gtEqJ1x/AOzBrDaaOEAQBwCqy1mrspX5Nnb9birQeYNYbTRgkDAOA0rdh5UJNnb9E/jpo1dufwjmrFrDHUAyUMAIAztHF/iabM2aIPVu6RkXRJn1a6aUh79W6dyB2VOC5KGAAADeSfs8beXrZLh45Uq0dagm4c3E6X9E5TTCQPDMf3UcIAAGhgJeWVev+b3Xpl8Q5t3F+q+OhwXdW/tcYNbqdOqU1cx4OfoIQBAOAj1lp9vf2gXlm8Q5+v2avKaquhnVJ04+B2Ord7c0WEMeIilFHCAABoBHklFXpr2S69vmSndheWyRsfpesGttV1A9uqRWK063hwgBIGAEAjqq6xmrU+V68s3qG5m/LkMUbndWuuG4e009BOKWzkDyEnKmFMngMAoIGFeYzO7d5c53Zvrh0Fh/T6kp16c9kufb52nzqmxumGQe10Vb/WSoyNcB0VDnElDACARlBeWa1Pv92rVxbv0Dc7CxUd4dElvVvpxsHt1bN1out48BGWIwEA8CNrdhfptSU79P43e1RWWa3ebZI0blBbXdy7laIjGHMRTChhAAD4oaKySr23IkevLN6hLXmHlBgTobGZrXXDoHZq3yzOdTw0AEoYAAB+zFqrxVsP6NXFO/TF2n2qqrEa3qWZxg1up3MyvApnzEXAooQBABAgcovL9cbXtWMu9hWXq2VitK4f2FbXDGwjbzxjLgINJQwAgABTVV2jGety9dqSHZq3KV/hHqMf9WihGwe306AOyYy5CBCMqAAAIMCEh3k0ukcLje7RQlvzSvXakp16Z3mOPlm9V128TTRucDtd3i9NCdGMuQhUXAkDACBAlB2p1ker9+i1xTu0KqdIsZFhurRPmsYNbquzWjHmwh+xHAkAQJBZtatQry7eoQ9X7VFFVY36tU3SjUPa6YIeLRlz4UcoYQAABKnCw0f0zvIcvbZkp7blH1LT2AiNHdBGNwxsp7Ypsa7jhTxKGAAAQa6mxmrhlgK9sni7ZqzLVY21Gtk1VTcObqdR6V6FedjI7wIlDACAELK3qEzTl+7S9KU7lVdSobSkGF0/qK2uGdBGzZpEuY4XUihhAACEoMrqGv3ju/16ZdEOLdpaoIgwowt6tNSNQ9ops11Txlw0AkZUAAAQgiLCPBrTs6XG9GypzbklenXxTr27IkcfrtqjjBbxumFwO13eN01NoqgDLnAlDACAEPL/2rvXIKvr+47j7+/ucr+KsAF2QSQaEVFkYajW0YmStDI1moBJm0SnzdTmScytmWmTTjudyaM+6C0PMmlSk9aq02TCrg5DjdqYlEk6apQFUS4xDCmwu1y9cBWWZb99cI6ZxQmXgyw/zvJ+PTrnf/78z2fnN3v47P//P7/fkd4+Vq7r4ZHnt7Gh5wBjhjfysbYW7rvpCuZMHV863pDj5UhJknSSzGTdjrd45PltrFq/k96+fhbPmsSnb5rJ0nnTGN7kepXngyVMkiSd0puHe/nhmh08+vx2tr9xhMljh/OJRTP4+KIZXDl5TOl4dc0SJkmSzqi/P/nZln088tw2frJ5N/0JbTMnsnxhK3ddP50Jo10iqVaWMEmSVJNd+4/yxLpu2td08as9hxje1MCHr30fy9pauO0DUxjW6OXKs2EJkyRJ5yQzebX7wG++VfnG4V4mjx3O3fNbWL6wxTUrz8ASJkmS3rPevn5Wv7aX9jVdPLt5N8dPJHOmjmN5Wyv33Did5vEjS0e86FjCJEnSefXm4V5Wre+hvbObdTveoiHgtg9MYVlbK783930uIl5lCZMkSYNmy55DdHR28fjabnbuP8q4kU3cdcM0lrW1XvIz81vCJEnSoOvvT57b+jrtnV089eoujvSeYOak0Sxra2HZglZmXj66dMQLzhImSZIuqMPH+njq1V20d3bx3NbXyYTFsyaxfGELS6+fxviRl8Z0F5YwSZJUTPdbb/PE2sp0F1v3HWZEUwO/f91UlrW1cOvVU2hsGLqXKy1hkiSpuHeWSuro7Gblyz3sf/s4zeNG8NEFLSxva+WaqeNKRzzvLGGSJOmicqzvBD/dvIcVa7r5n1/uoa8/uW76eJa3tXL3jdOZPHZE6YjnhSVMkiRdtF4/dIyVL/fQ0dnNK937aWoIPnhNZbqLJdc2M6Kpfqe7sIRJkqS68Nrug7R3dvHE2m52HzjGhFHD+Mj8ynQXC2ZMrLvpLixhkiSprpzoT/53yz7aO7t4esMujh7vZ/bkMSxra+Fjba20TBxVOuJZKVbCIuJO4BtAI/BQZv7du16fCTwMTKzu89XMfPJ0x7SESZJ0aTl49Dg/eqUy3cULv34DgJtnX87yha0snTeVMSOaCic8tSIlLCIagdeADwNdwIvAJzNz44B9vgOszcxvRcRc4MnMnHW641rCJEm6dO144wiPr+2mvbOLba8fYdSwRpbOm8qytlZufv/lF910F6crYYNZHRcDWzJzazXE94F7gI0D9klgfPXxBKBnEPNIkqQ6N2PSaL6w5Go+f8dVdG5/kxVrulm1voeOtd1MmzDyN9NdXNU8tnTUMxrMM2H3Andm5gPV5/cDv5OZDw7YZxrwDHAZMAb4UGau+S3H+izwWYCZM2cu3LZt26BkliRJ9efo8RP8eNNuOjq75u/a0wAABpVJREFUWf3aXk70J/NbJ7B8YSsfuWE6l40ZXixbqcuRZ1PC/rya4R8i4mbgu8C8zOw/1XG9HClJkk5lz8GjrFzXQ3tnN5t2HmBYY3DHnGaWtbVy+zXNDG9quKB5Sl2O7AZmDHjeWt020J8CdwJk5nMRMRKYDOwZxFySJGmIah43kgdunc0Dt85mY88BOjq7eGJdD09v2M1lo4dx9/zpLF/YyvUtE4pPdzGYZ8KaqNyYv4RK+XoR+FRmbhiwz4+AH2Tmv0fEtcCzQEueJpRnwiRJUi36TvTzs1/tY0VnF/+9cTe9ff1c3TyWP7ttNp9YNOPMB3gPipwJy8y+iHgQeJrK9BPfy8wNEfF14KXMXAl8BfjXiPgylZv0/+R0BUySJKlWTY0N3D6nmdvnNLP/7eP81/qddHR2sXv/0aK5nKxVkiRdkvr7k4ZBntLidGfCLuzdaZIkSReJwS5gZ3z/ou8uSZJ0ibKESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgGWMEmSpAIsYZIkSQVYwiRJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZGZpTPUJCL2AtsuwFtNBvZdgPfR4HD86p9jWP8cw/rm+J0fV2TmlN/2Qt2VsAslIl7KzEWlc+jcOH71zzGsf45hfXP8Bp+XIyVJkgqwhEmSJBVgCTu175QOoPfE8at/jmH9cwzrm+M3yLwnTJIkqQDPhEmSJBVgCZMkSSrAEvYuEXFnRPwyIrZExFdL51FtImJGRPw0IjZGxIaI+GLpTKpdRDRGxNqIWFU6i2oXERMjYkVEbI6ITRFxc+lMqk1EfLn6GfpqRPxnRIwsnWkosoQNEBGNwDeBpcBc4JMRMbdsKtWoD/hKZs4FbgI+5xjWpS8Cm0qH0Dn7BvBUZs4B5uNY1pWIaAG+ACzKzHlAI/BHZVMNTZawky0GtmTm1szsBb4P3FM4k2qQmTszs7P6+CCVD/+WsqlUi4hoBf4AeKh0FtUuIiYAtwHfBcjM3sx8q2wqnYMmYFRENAGjgZ7CeYYkS9jJWoAdA5534X/gdSsiZgELgBfKJlGN/hn4C6C/dBCdkyuBvcC/VS8pPxQRY0qH0tnLzG7g74HtwE5gf2Y+UzbV0GQJ05AUEWOBduBLmXmgdB6dnYi4C9iTmWtKZ9E5awLagG9l5gLgMOD9tXUkIi6jchXoSmA6MCYi7iubamiyhJ2sG5gx4HlrdZvqSEQMo1LAHsvMjtJ5VJNbgLsj4v+o3A5wR0Q8WjaSatQFdGXmO2egV1ApZaofHwJ+nZl7M/M40AH8buFMQ5Il7GQvAldHxJURMZzKjYgrC2dSDSIiqNyLsikz/7F0HtUmM7+Wma2ZOYvK799PMtO/wOtIZu4CdkTENdVNS4CNBSOpdtuBmyJidPUzdQl+uWJQNJUOcDHJzL6IeBB4msq3Qb6XmRsKx1JtbgHuB16JiHXVbX+VmU8WzCRdaj4PPFb9Y3Yr8JnCeVSDzHwhIlYAnVS+cb4WlzAaFC5bJEmSVICXIyVJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gknaWI+GBErCqdQ9LQYAmTJEkqwBImaciJiPsi4hcRsS4ivh0RjRFxKCL+KSI2RMSzETGluu+NEfF8RKyPiMer6+YREVdFxI8j4uWI6IyI91cPPzYiVkTE5oh4rDqjuCTVzBImaUiJiGuBPwRuycwbgRPAp4ExwEuZeR2wGvjb6j/5D+AvM/MG4JUB2x8DvpmZ86msm7ezun0B8CVgLjCbyioNklQzly2SNNQsARYCL1ZPUo0C9gD9wA+q+zwKdETEBGBiZq6ubn8Y+GFEjANaMvNxgMw8ClA93i8ys6v6fB0wC/j54P9YkoYaS5ikoSaAhzPzaydtjPibd+13rmu2HRvw+AR+jko6R16OlDTUPAvcGxHNABExKSKuoPJ5d291n08BP8/M/cCbEXFrdfv9wOrMPAh0RcRHq8cYERGjL+hPIWnI8y84SUNKZm6MiL8GnomIBuA48DngMLC4+toeKveNAfwx8C/VkrUV+Ex1+/3AtyPi69VjfPwC/hiSLgGRea5n5CWpfkTEocwcWzqHJL3Dy5GSJEkFeCZMkiSpAM+ESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgH/D71NauZ7eJIJAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# dot_img_file = \"d_mod-\"+str(d_model)+\"-d_k-\"+str(d_k)+\"-d_v-\"+str(d_v)+\"-n_heads-\"+str(n_heads)+\"-enc_lays-\"+str(n_encoder_layers)+\"-dec_lays-\"+str(n_decoder_layers)+'-transformer.png'\n",
        "# tf.keras.utils.plot_model(transformer, to_file=dot_img_file, show_shapes=True)"
      ],
      "metadata": {
        "id": "RSK1eufbeQPL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#transformer = keras.models.load_model('./Models/')"
      ],
      "metadata": {
        "id": "Oly3kYeU3FnV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Testing"
      ],
      "metadata": {
        "id": "KJ_Gx7RVeNla"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def greedy_translate(model, source_sentence, target_sentence_start=sp_ben_bpe.bos_id()):\n",
        "  # Tokenizing and padding\n",
        "  source_seq = tokenize(source_sentence, sp_en_bpe)\n",
        "  source_seq = tf.keras.preprocessing.sequence.pad_sequences(source_seq, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "  predict_seq = [[target_sentence_start]]\n",
        "  \n",
        "  predict_sentence = [target_sentence_start] # Deep copy here to prevent updates on target_sentence_start\n",
        "  while predict_sentence[-1] != sp_ben_bpe.eos_id() and len(predict_sentence) < MAX_TOKENS:\n",
        "    predict_output = model([np.array(source_seq), np.array(predict_seq)], training=None)\n",
        "    predict_label = tf.argmax(predict_output, axis=-1) # Pick the label with highest softmax score\n",
        "    predict_seq = tf.concat([predict_seq, predict_label], axis=-1) # Updating the prediction sequence\n",
        "    predict_sentence.append(predict_label[0][0].numpy())\n",
        "  \n",
        "  if len(predict_sentence) == MAX_TOKENS:\n",
        "      return detokenize(predict_sentence, sp_ben_bpe, True, False)\n",
        "  return detokenize(predict_sentence, sp_ben_bpe)"
      ],
      "metadata": {
        "id": "zfg1AaKDnPQM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow_probability as tfp"
      ],
      "metadata": {
        "id": "pLrtBO__iPPl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I love you.\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", greedy_translate(transformer, sentence))"
      ],
      "metadata": {
        "id": "KtWeJl1JoapJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "99f6118b-d636-4513-fd98-9a8a8e663f2d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  I love you.\n",
            "Bengali Sentence:  আমি তোমাকে ভালোবাসি।\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I am tired.\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", greedy_translate(transformer, sentence))"
      ],
      "metadata": {
        "id": "K2axdVLjq6z1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a1796d1f-046e-4c07-bb48-0287111fd9fb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  I am tired.\n",
            "Bengali Sentence:  ক্লান্ত হয়ে উঠলাম।\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentences = list(test_val_data['src'][:250])\n",
        "refs = [list(test_val_data['tgt'][:250])]\n",
        "sys = [greedy_translate(transformer, sentence) for sentence in sentences]"
      ],
      "metadata": {
        "id": "9XNckV-wBgiN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "bleu = BLEU()\n",
        "bleu.corpus_score(sys, refs)"
      ],
      "metadata": {
        "id": "otRhPJC1VKyj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0f0c3ea0-3c0f-4c41-c1c6-79ab91025811"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "BLEU = 1.51 18.1/3.7/0.9/0.1 (BP = 0.943 ratio = 0.945 hyp_len = 3386 ref_len = 3583)"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "chrf = CHRF()\n",
        "chrf.corpus_score(sys, refs)"
      ],
      "metadata": {
        "id": "mlhToSrvYNqd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "563e1a0c-a721-48de-f84e-4694bf974dff"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "chrF2 = 25.52"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "ter = TER()\n",
        "ter.corpus_score(sys, refs)"
      ],
      "metadata": {
        "id": "DtIadLMJYe1w",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9b77a855-f11e-448e-e0eb-b2971d430bf3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TER = 100.29"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "val_perf = pd.DataFrame({\"src\":sentences, \"sys\":sys, \"refs\":refs[0]})\n",
        "val_perf"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "KLHa33EULz3b",
        "outputId": "508eb993-2188-4561-a83b-b7db4b851351"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                   src  \\\n",
              "0    Through her work, she spread the message of th...   \n",
              "1    Interacting with beneficiaries and store owner...   \n",
              "2                                       These include:   \n",
              "3    It is no surprise that today Japan is India’s ...   \n",
              "4           Record growth in last two and a half years   \n",
              "..                                                 ...   \n",
              "245  •India will emerge as a global leader in trans...   \n",
              "246        BHIM stands for Bharat Interface for Money.   \n",
              "247          Heartiest congratulations, Beti Shivangi.   \n",
              "248                But, we are now awaiting the rains.   \n",
              "249  As reported by the Swiss Bank Authorities, the...   \n",
              "\n",
              "                                                   sys  \\\n",
              "0    তার কাজ ও পরিচ্ছন্নতা বাড়াতে এবং মানবসম্পদ বি...   \n",
              "1    নমাল এবং গ্রিডের সঙ্গে যুক্ত দেশগুলির মধ্যে বি...   \n",
              "2                                   এদের মধ্যে রয়েছে:   \n",
              "3    এটি আজ শনিবার ভারত আজ জাপানে নতুন পেট্রোর চতুর...   \n",
              "4               গত দুই বছরে দুই বছর ধরে বৃদ্ধি পেয়েছে   \n",
              "..                                                 ...   \n",
              "245  @ভারত-এর মতো আন্তর্জাতিক শিল্প ও দক্ষ জনশক্তি ...   \n",
              "246                নতুন টাকা দিয়ে ভারতকে একটি পিস্তল।   \n",
              "247           হার্ট, ডিম, কাটা, কাটাগড়, ধাচালা, কচ্ছপ   \n",
              "248               কিন্তু, আমরা এখন শেষ পর্যন্ত বৃষ্টি।   \n",
              "249  সুইস ব্যাংকের টাকা কোম্পানি, ভারতীয় ব্যাংকের ...   \n",
              "\n",
              "                                                  refs  \n",
              "0    নিজের কর্মেরউদাহরণ স্থাপন করে তিনি মানুষকে সেব...  \n",
              "1    সারা দেশের ৫ হাজারেরও বেশি স্থান থেকে দোকান মা...  \n",
              "2                         এই প্রকল্পগুলির মধ্যে রয়েছে-  \n",
              "3    জাপান যে বর্তমানে ভারতের চতুর্থ বৃহত্তম প্রত্য...  \n",
              "4                    গত আড়াই বছরেরেকর্ড পরিমাণ অগ্রগতি  \n",
              "..                                                 ...  \n",
              "245  · সর্বাধুনিক প্রযুক্তি এবং দক্ষ শ্রমিকের ব্যবহ...  \n",
              "246                  BHIM মানে ভারত ইন্টারফেস ফর মানি।  \n",
              "247               কন্যা শিবাঙ্গীকে অনেক অনেক অভিনন্দন!  \n",
              "248                           বর্ষার প্রতীক্ষা চলছেই ।  \n",
              "249  স্যুইস ব্যাঙ্কবলেছে যে ভারতীয়দের দ্বারা জমা ধন...  \n",
              "\n",
              "[250 rows x 3 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-9b37b1e1-1e44-4a4b-b275-8239455f25b5\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>src</th>\n",
              "      <th>sys</th>\n",
              "      <th>refs</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Through her work, she spread the message of th...</td>\n",
              "      <td>তার কাজ ও পরিচ্ছন্নতা বাড়াতে এবং মানবসম্পদ বি...</td>\n",
              "      <td>নিজের কর্মেরউদাহরণ স্থাপন করে তিনি মানুষকে সেব...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Interacting with beneficiaries and store owner...</td>\n",
              "      <td>নমাল এবং গ্রিডের সঙ্গে যুক্ত দেশগুলির মধ্যে বি...</td>\n",
              "      <td>সারা দেশের ৫ হাজারেরও বেশি স্থান থেকে দোকান মা...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>These include:</td>\n",
              "      <td>এদের মধ্যে রয়েছে:</td>\n",
              "      <td>এই প্রকল্পগুলির মধ্যে রয়েছে-</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>It is no surprise that today Japan is India’s ...</td>\n",
              "      <td>এটি আজ শনিবার ভারত আজ জাপানে নতুন পেট্রোর চতুর...</td>\n",
              "      <td>জাপান যে বর্তমানে ভারতের চতুর্থ বৃহত্তম প্রত্য...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Record growth in last two and a half years</td>\n",
              "      <td>গত দুই বছরে দুই বছর ধরে বৃদ্ধি পেয়েছে</td>\n",
              "      <td>গত আড়াই বছরেরেকর্ড পরিমাণ অগ্রগতি</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>245</th>\n",
              "      <td>•India will emerge as a global leader in trans...</td>\n",
              "      <td>@ভারত-এর মতো আন্তর্জাতিক শিল্প ও দক্ষ জনশক্তি ...</td>\n",
              "      <td>· সর্বাধুনিক প্রযুক্তি এবং দক্ষ শ্রমিকের ব্যবহ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>246</th>\n",
              "      <td>BHIM stands for Bharat Interface for Money.</td>\n",
              "      <td>নতুন টাকা দিয়ে ভারতকে একটি পিস্তল।</td>\n",
              "      <td>BHIM মানে ভারত ইন্টারফেস ফর মানি।</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>247</th>\n",
              "      <td>Heartiest congratulations, Beti Shivangi.</td>\n",
              "      <td>হার্ট, ডিম, কাটা, কাটাগড়, ধাচালা, কচ্ছপ</td>\n",
              "      <td>কন্যা শিবাঙ্গীকে অনেক অনেক অভিনন্দন!</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>248</th>\n",
              "      <td>But, we are now awaiting the rains.</td>\n",
              "      <td>কিন্তু, আমরা এখন শেষ পর্যন্ত বৃষ্টি।</td>\n",
              "      <td>বর্ষার প্রতীক্ষা চলছেই ।</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>249</th>\n",
              "      <td>As reported by the Swiss Bank Authorities, the...</td>\n",
              "      <td>সুইস ব্যাংকের টাকা কোম্পানি, ভারতীয় ব্যাংকের ...</td>\n",
              "      <td>স্যুইস ব্যাঙ্কবলেছে যে ভারতীয়দের দ্বারা জমা ধন...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>250 rows × 3 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9b37b1e1-1e44-4a4b-b275-8239455f25b5')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-9b37b1e1-1e44-4a4b-b275-8239455f25b5 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-9b37b1e1-1e44-4a4b-b275-8239455f25b5');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def sampling_translate(model, source_sentence, temperature=0.6, target_sentence_start=sp_ben_bpe.bos_id()):\n",
        "  # Tokenizing and padding\n",
        "  source_seq = tokenize(source_sentence, sp_en_bpe)\n",
        "  source_seq = tf.keras.preprocessing.sequence.pad_sequences(source_seq, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "  predict_seq = [[target_sentence_start]]\n",
        "  logprob = 0\n",
        "  \n",
        "  predict_sentence = [target_sentence_start] # Deep copy here to prevent updates on target_sentence_start\n",
        "  while predict_sentence[-1] != sp_ben_bpe.eos_id() and len(predict_sentence) < MAX_TOKENS:\n",
        "    predict_output = model([np.array(source_seq), np.array(predict_seq)], training=None)\n",
        "    probs = predict_output[-1].numpy()[0]\n",
        "    log_probs = np.log(probs)\n",
        "    u = np.random.uniform(low=1e-6, high=1.0 - 1e-6, size=log_probs.shape)\n",
        "    g = -np.log(-np.log(u))\n",
        "    sample_idx = np.argmax(log_probs + g * temperature, axis=-1)\n",
        "    sample_prob = predict_output[-1].numpy()[0][sample_idx]\n",
        "    logprob += np.log(sample_prob)\n",
        "    predict_label = np.arange(predict_output.shape[-1])[sample_idx] # Sample the label softmax score\n",
        "    predict_seq = tf.concat([predict_seq, tf.cast([[predict_label]], dtype=tf.int32)], axis=-1) # Updating the prediction sequence\n",
        "    predict_sentence.append(predict_label)\n",
        "  \n",
        "  if len(predict_sentence) == MAX_TOKENS:\n",
        "      return detokenize(predict_sentence, sp_ben_bpe, True, False), logprob\n",
        "  return detokenize(predict_sentence, sp_ben_bpe), logprob"
      ],
      "metadata": {
        "id": "2dmVo_eQhE8v"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I love you.\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", sampling_translate(transformer, sentence))"
      ],
      "metadata": {
        "id": "CB5iHHlxh1cC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "50f97bd9-7493-4558-9a85-7534a259c480"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  I love you.\n",
            "Bengali Sentence:  ('আমি আপনাকে ভালোবাসি।', -3.823479017271893)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I am hungry.\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", sampling_translate(transformer, sentence))"
      ],
      "metadata": {
        "id": "IiZmzCY3q1S9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "885f3ea9-b524-4fd2-a61f-e2a7d6ebf59a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  I am hungry.\n",
            "Bengali Sentence:  ('আমি ক্ষুধার্ত মানুষ।', -4.342673036982887)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def generate_samples(sentence, n_samples, model=None, temperature=0.6):\n",
        "    samples, log_probs = [], []\n",
        "    for _ in range(n_samples):\n",
        "        sample, logp = sampling_translate(model, sentence, temperature)\n",
        "        samples.append(sample)\n",
        "        log_probs.append(logp)\n",
        "    return samples, log_probs"
      ],
      "metadata": {
        "id": "xIxJ9HqTtxHY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def rouge1_similarity(system, reference):\n",
        "    sys_counter = Counter(system)\n",
        "    ref_counter = Counter(reference)\n",
        "    overlap = 0\n",
        "    for token in sys_counter:\n",
        "        token_count_sys = sys_counter[token]\n",
        "        token_count_ref = ref_counter[token]\n",
        "        overlap += min(token_count_ref, token_count_sys)\n",
        "    precision = overlap / sum(sys_counter.values())\n",
        "    recall = overlap / sum(ref_counter.values())\n",
        "    if precision + recall != 0:\n",
        "        rouge1_score = 2 * ((precision * recall)/(precision + recall))\n",
        "    else:\n",
        "        rouge1_score = 0     \n",
        "    return rouge1_score"
      ],
      "metadata": {
        "id": "uhIlVoVwt7YC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def weighted_avg_overlap(samples, log_probs):\n",
        "    scores = {}\n",
        "    for index_candidate, candidate in enumerate(samples):    \n",
        "        overlap, weight_sum = 0.0, 0.0\n",
        "        for index_sample, (sample, logp) in enumerate(zip(samples, log_probs)):           \n",
        "            if index_candidate == index_sample:\n",
        "                continue\n",
        "            sample_p = float(np.exp(logp))\n",
        "            weight_sum += sample_p\n",
        "            sample_overlap = rouge1_similarity(candidate, sample)\n",
        "            overlap += sample_p * sample_overlap\n",
        "        score = overlap / weight_sum\n",
        "        scores[index_candidate] = score\n",
        "    return scores"
      ],
      "metadata": {
        "id": "7KxiaRENt-C_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def mbr_decode(sentence, n_samples, model=None, temperature=0.6):\n",
        "    samples, log_probs = generate_samples(sentence, n_samples, model, temperature)\n",
        "    scores = weighted_avg_overlap(samples, log_probs)\n",
        "    max_score_key = max(scores, key=scores.get)\n",
        "    return Counter({sample:score for sample, score in zip(samples, scores.values())})"
      ],
      "metadata": {
        "id": "ny5EYLFWt_5u"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def mbr_decode(sentence, n_samples, model=None, temperature=0.6):\n",
        "    samples, log_probs = generate_samples(sentence, n_samples, model, temperature)\n",
        "    scores = weighted_avg_overlap(samples, log_probs)\n",
        "    max_score_key = max(scores, key=scores.get)\n",
        "    return samples[max_score_key]"
      ],
      "metadata": {
        "id": "9LifgZ0pbrRr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "rand_idx = np.random.randint(0, test_val_data.shape[0], 10)\n",
        "sentences = list(test_val_data['src'][rand_idx])\n",
        "refs = [list(test_val_data['tgt'][rand_idx])]\n",
        "sys = [mbr_decode(sentence, 5, transformer) for sentence in sentences]\n",
        "val_perf = pd.DataFrame({'src':sentences, 'sys':sys, 'ref':refs[0]})\n",
        "val_perf.values"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "I5V2He8Fb2s9",
        "outputId": "44d0c817-294c-4e87-e37b-8a946cb74b4a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([['One of our largest states, Maharashtra, has already adopted a plan of its own in this direction.',\n",
              "        'প্রসঙ্গত, মহারাষ্ট্র, মহারাষ্ট্র, মহারাষ্ট্র এবং এই নির্দেশনা ইতোমধ্যেই একটি পরিকল্পনা রয়েছে।',\n",
              "        'মহারাষ্ট্রের মতোআমাদের একটি বৃহত্তম রাজ্য ইতিমধ্যেই এই লক্ষ্যে একটি পরিকল্পনা গ্রহণ করেছে।'],\n",
              "       ['As there are no middlemen, the goods are available at very reasonable rates.',\n",
              "        'কারখানাটিরও অস্বাভাবিক নিয়ম নেই, তবে খুব কম মজুরি পাওয়া যায় না।',\n",
              "        'মাঝে দালাল না থাকায় অনেক সস্তায় পাওয়া যায়।'],\n",
              "       ['Kochi ranks first in the total number of international and domestic tourist arrivals in Kerala.',\n",
              "        'থাইল্যান্ডের প্রথম পর্যটকের সংখ্যা এবং পর্যটন পর্যটকদের মধ্যে পর্যটকদের অংশগ্রহণ।',\n",
              "        'কেরলে আগত দেশ-বিদেশের পর্যটকদের সংখ্যার নিরিখেকোচির স্থান রয়েছে প্রথমেই।'],\n",
              "       ['My personal engagement with the leadership, Government, Industry and people of Japan is now a decade old.',\n",
              "        'আমার ব্যক্তিগত সম্পর্ক আমার ব্যক্তিগত সম্পর্ক আমার ব্যক্তিগত যোগাযোগ এবং জাপান, শিল্পবিদ, শিল্প ও নাগরিক।',\n",
              "        'জাপানের নেতৃত্ব, সরকার, শিল্প গোষ্ঠী এবং জনসাধারণের সঙ্গেআমার ব্যক্তিগত সম্পর্ক এক দশকেরও বেশি।'],\n",
              "       ['All this in turn will give a major boost to socio-economic activity in the state, opening up employment opportunities for the local people.',\n",
              "        'এই প্রসঙ্গে সকল উন্নয়নমূলক প্রকল্পের জন্য সাংস্কৃতিক বিকাশে সামাজিক দক্ষতা উন্নয়ন প্রকল্পের সূচনা করবে।',\n",
              "        'এর ফলে রাজ্যের আর্থ-সামাজিক উন্নয়ন ত্বরান্বিত হবে এবং স্হানীয় মানুষের জন্য কর্মসংস্হানের সুযোগ তৈরি হবে।'],\n",
              "       ['And, Cyprus too has always reciprocated with its warmth and friendship.',\n",
              "        'আর, সাইপ্রাস আর সাইপ্রাস ও বন্ধুত্ব রয়েছে।',\n",
              "        'একইভাবে, সাইপ্রাসও তারবিনিময়ে আন্তরিকতা ও মৈত্রীর হাত বাড়িয়ে দিয়েছে আমাদের দিকে।'],\n",
              "       ['The Prime Minister asserted that more people are paying taxes because they have faith that their money is being used properly and for the welfare of people.',\n",
              "        'মন্ত্রী বলেন, ‘দেশি মানুষের জন্য যারা বিশ্বাস করে তাদের অর্থ হচ্ছে তাদেরকে অর্থেন্দ্রিয়ই-সংযত করে।',\n",
              "        'প্রধানমন্ত্রী আরও বলেন, বহু সংখ্যক মানুষ কর প্রদান করছেন, কারন তাদের মনে বিশ্বাস রয়েছে যে করের টাকা যথাথয ব্যবহার হচ্ছে এবং মানুষের কল্যাণে কাজে লাগানো হচ্ছে।'],\n",
              "       ['Now, about three thousand small and medium merchants have this privilege through the new Special Notified Zone.',\n",
              "        'এখন, তিনটি ছোট এবং মাঝারি ও মাঝারি ব্যবসায়ী শ্রেণীর পর্যটন কেন্দ্রগুলিতে নানা উদ্যোগ নিয়েছে।',\n",
              "        'কিন্তু নতুন বিশেষবিজ্ঞাপিত অঞ্চলের আওতায় এই সুযোগ বর্তমানে লাভ করছে প্রায় ৩ হাজার ক্ষুদ্র ওমাঝারি বাণিজ্যিক সংস্থা।'],\n",
              "       ['India has emerged as a bright spot in the global economy which is driving global growth as well.',\n",
              "        'ভারতে বিশ্ব অর্থনীতি সারা বিশ্বে একটি উজ্জ্বল শিল্প সঞ্চার করে এবং বিশ্বের অর্থনীতি বিশ্বব্যাপী বিনিয়োগ করছে।',\n",
              "        'বিশ্ব অর্থনীতিতে একটি উজ্জ্বল বিন্দু রূপে ভারতের আবির্ভাব বিশ্বের অগ্রগতিকেও সঠিক পথে চালিত করেছে।'],\n",
              "       ['“Remembering Shyamji Krishna Varma on his birth anniversary.',\n",
              "        '@সামবীর মন্দিরটি স্মরণাভ্যাসের জন্মদিনে স্মরণ করিয়ে দেন কৃষ্ণনগর।',\n",
              "        '“শ্যামজি কৃষ্ণ ভার্মার জন্মবার্ষিকীতে তাঁকে স্মরণ করিআমি।']],\n",
              "      dtype=object)"
            ]
          },
          "metadata": {},
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "rand_idx = np.random.randint(0, train_dev_data.shape[0], 10)\n",
        "sentences = list(train_dev_data['src'][rand_idx])\n",
        "refs = [list(train_dev_data['tgt'][rand_idx])]\n",
        "sys = [mbr_decode(sentence, 5, transformer) for sentence in sentences]\n",
        "val_perf = pd.DataFrame({'src':sentences, 'sys':sys, 'ref':refs[0]})\n",
        "val_perf.values"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "81C9DxwTcJP7",
        "outputId": "33845891-bdaf-47ec-b5d8-0fefdcd3dd0e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([['What kind?', 'কোন ধরনের?', 'কি রকম ঘা?'],\n",
              "       ['The palace is an extended part of a huge complex.',\n",
              "        'প্রাসাদটি একটি জটিল জটিল একটি জটিল স্তর ধারণ করে।',\n",
              "        'প্রাসাদ একটি বিশাল জটিল একটি বর্ধিত অংশ।'],\n",
              "       ['Then he abruptly disappeared.', 'তারপর হঠাৎ হারিয়ে গেলো সে।',\n",
              "        'পরে হঠাৎ বেপাত্তা হয়ে যান।'],\n",
              "       ['How is that experience?', 'কীভাবে অভিজ্ঞতা?',\n",
              "        'এমন অভিজ্ঞতা কার না হয়েছে?'],\n",
              "       ['\"In India, faith and Nature have had a deep link since ancient times.\"',\n",
              "        '\"ভারতে, বিশ্বাস এবং প্রকৃতি এবং প্রকৃতি যে প্রাচীন কালে ছিল \"ভারত।\"',\n",
              "        'সুপ্রাচীনকাল থেকেই বিশ্বাস ও প্রকৃতির মধ্যে এক গভীর সংযোগ রয়েছে বলে আমরা মনে করি।'],\n",
              "       ['He had a rope tied around his waist.',\n",
              "        'তার কোমর ঘিরে ফেলে তিনি।', 'তাঁর কোমড়ে দড়ি বেঁধে দেওয়া হয়।'],\n",
              "       ['There have been numerous ideas and attempts to reduce the amount of carbon emissions.',\n",
              "        'অনেকগুলি অণুযায়ী অনেক কিছু সফ্টভ্যতার জন্য প্রয়োজনীয় উপাদান সরবরাহ করা হয়েছে।',\n",
              "        'যানবাহনের কার্বন নির্গমন কমাতে কিছু উপায় নিয়ে যথেষ্ট গবেষণা করা হয়েছে।'],\n",
              "       ['\"Do you think it is possible for mere humans to come to know our almighty Creator, as stated here in the Bible?\"',\n",
              "        '\"আপনার কি মনে হয়, তুমি কি জানো আর আমাদের সৃষ্টিকর্তা সম্বন্ধে জানতে পারবে না?',\n",
              "        '\"বাইবেলের এখানে যেমন বর্ণনা করা হয়েছে, আপনি কি মনে করেন, সামান্য মানুষের পক্ষে আমাদের সর্বশক্তিমান সৃষ্টিকর্তাকে জানা সম্ভব?\"'],\n",
              "       ['Are we leaving for good?', 'আমরা ভাল যাব?',\n",
              "        'আমরা কি ভালকিছুর জন্যই যাচ্ছি?'],\n",
              "       ['Investigation was taken away.',\n",
              "        'তদন্ত করে খুঁজতে তদন্ত করা হয়েছে।',\n",
              "        'কেড়ে নেওয়া হয়েছে তদন্তভারও।']], dtype=object)"
            ]
          },
          "metadata": {},
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I love you.\"\n",
        "translated_sentence = mbr_decode(sentence, 2, transformer, 1)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "id": "lzrxKRMAaOgr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0efc2f76-5c61-4035-e506-1a68ab036e9d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  I love you.\n",
            "Bengali:  Counter({'তোমায় ভালোবাসি।?': 0.8333333333333333, 'আমি তোমায় ভালবাসি।': 0.8333333333333333})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"How are you.\"\n",
        "translated_sentence = mbr_decode(sentence, 2, transformer, 1)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "id": "3ACvLvfouZz0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "526b4ffe-eec1-4300-ffbd-beefd91b5ab5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  How are you.\n",
            "Bengali:  Counter({'তুই কেমন করিয়াছিলে ভালোবাসে।': 0.5909090909090909, 'তুমি কেমন করবে.': 0.5909090909090909})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"How are you.\"\n",
        "translated_sentence = mbr_decode(sentence, 10, transformer)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "id": "hFnTZVktqKjJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dd8498cd-480a-4cb0-fb06-dfb7bc0dc191"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  How are you.\n",
            "Bengali:  Counter({'তুমি কেমন আছ।': 0.9305736909790155, 'তুমি কেমন আছ?': 0.8933531279880936, 'তুমি কেমন আছ.': 0.8921497437190088, 'তুমি কেমন করে আছো।': 0.8074346643835761, 'কী ভাবে তুমি কেমন আছো।': 0.7336428687765982, 'তুমি কীভাবে জানো।': 0.6476839734061707, 'কেমন করে মানুষ।': 0.6221847705590963, 'তুই কীভাবে তুমি কেমন লেগেছে।': 0.5777984527752899, 'তুমি জানো তো?': 0.5546939915414959})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I am hungry.\"\n",
        "translated_sentence = mbr_decode(sentence, 10, transformer)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "id": "QlRYyoz_qXcQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3aa19560-3e28-4cc6-a07e-2f83bb71ed78"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  I am hungry.\n",
            "Bengali:  Counter({'আমি ক্ষুধার্ত মানুষ।': 0.8034417484271178, 'আমি ক্ষুধার্ত।': 0.7408959724716873, 'আমি ক্ষুধার্ত.': 0.7293499618315494, 'আমি ক্ষুধার্ত হয়েছি।': 0.715670845734493, 'আমি ক্ষুধার্ত বোধ করি।': 0.6852335741659037, 'আমি ক্ষুধা ঘুরে দেখেছি।': 0.6616903228114911, 'আমার ক্ষুধা লেগেছে।': 0.6409023735781312, 'আমি খাঁচার শিকার।': 0.5248980649524358})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I want ice cream\"\n",
        "translated_sentence = mbr_decode(sentence, 10, transformer)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "id": "EqW3eaYJqpyB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f98d0de2-97f5-4f8f-809b-acedb77e1e70"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  I want ice cream\n",
            "Bengali:  Counter({'আমি মদ খেতে চাই।': 0.8353513048171028, 'আমি মদ খেতে চাই': 0.7455274875315632, 'আমি চাই আমি মদ': 0.7143903816328921, 'আমি চাইলাম': 0.611768273794336, 'আমি মরিচের গুঁড়া': 0.5968469905260151, 'আমি চাই আমার পোশাক': 0.5905388336710808, 'আমি চাই আমি ক্রিম্ট': 0.5783493182353484, 'আমি চাই আমি ক্রিমিয়া': 0.5610072465035275, 'আমি মদ বলি,': 0.5310762306100834})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I am a boy.\"\n",
        "translated_sentence = mbr_decode(sentence, 10, transformer)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uuS4_iszLgkd",
        "outputId": "3a6e5efb-7b79-45cd-d855-bbe45f87cc7f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  I am a boy.\n",
            "Bengali:  Counter({'আমি ছেলে।': 0.9847441688712745, 'আমি তো ছেলে।': 0.8522429584701589, 'আমি ছেলে ছেলে।': 0.7828932372825279, 'ছেলেটা আমি ছেলে।': 0.7307984336757547, 'আমি ছেলেটাকে আমি।': 0.6934141731470815})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "fAp6nH_vLrT4"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}