Neural_Machine_Translation_tf_3L_4H.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Neural_Machine_Translation_tf-3L-4H.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "gpuClass": "standard",
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "#### Mounting Google Drive"
      ],
      "metadata": {
        "id": "0dqNGaFLKKAJ"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "bVenoImvJZs9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e4eea3cf-7401-4110-f02c-6007cbceb9e0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "cd \"/content/drive/MyDrive/IASNLP\""
      ],
      "metadata": {
        "id": "iNpRDc1-KMj8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c7d4a25f-a61c-4f3d-d3a7-a59d943be715"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/drive/MyDrive/IASNLP\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Importing Necessary Libraries"
      ],
      "metadata": {
        "id": "OlMuBUdIApMX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install sentencepiece\n",
        "!pip install sacrebleu\n",
        "!pip install tensorflow-gpu # ---> For GPU"
      ],
      "metadata": {
        "id": "Iwi6TDKsB0St"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers\n",
        "from tensorflow.keras import Model\n",
        "from tensorflow.keras import Sequential\n",
        "from tensorflow.keras.layers import Dense, Lambda, Layer, Embedding, LayerNormalization\n",
        "import sentencepiece as spm\n",
        "from sacrebleu.metrics import BLEU, CHRF, TER\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from collections import Counter"
      ],
      "metadata": {
        "id": "DXSYYy2b-jMM"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Loading Data"
      ],
      "metadata": {
        "id": "ToLMSNAHBnvJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We load the `train_data`, `train_dev_data`, `test_val_data` and `test_data` as well as the byte-pair encoder tokenizer for English and Bengali i.e. `sp_en_bpe` and `s_ben_bpe`."
      ],
      "metadata": {
        "id": "zXIf-FL7IG7f"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "train_data, train_dev_data = pd.read_csv(\"train_data.csv\")[['src', 'tgt']], pd.read_csv(\"train_dev.csv\")[['src', 'tgt']] \n",
        "test_val_data, test_data = pd.read_csv(\"test_val.csv\")[['src', 'tgt']], pd.read_csv(\"test_data.csv\")[['src', 'tgt']]\n",
        "sp_en_bpe, sp_ben_bpe = spm.SentencePieceProcessor(), spm.SentencePieceProcessor()\n",
        "sp_en_bpe.load('eng_bpe.model'); sp_ben_bpe.load('ben_bpe.model');"
      ],
      "metadata": {
        "id": "IXsCg4mBBeoo"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Data Preprocessing"
      ],
      "metadata": {
        "id": "auf4TlPFhNtn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We decide on some formats of the data and batches. We take the maximum tokens we can have per tokenized English and Bengali sentence to be `MAX_TOKENS = 60`. We use the vocabulary size of `32000`(i.e. `ENCODER_VOCAB = DECODER_VOCAB = 32000`). A batch size of `256` was choosen for our Mini-Batch Gradient Descent which is used to train our Model."
      ],
      "metadata": {
        "id": "3z7BUnn_G3WL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MAX_TOKENS = 60\n",
        "VOCAB_SIZE = 32000\n",
        "BATCH_SIZE = 256\n",
        "BUFFER_SIZE = BATCH_SIZE*4"
      ],
      "metadata": {
        "id": "7jTuQSqOAybO"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The following two functions help in tokenizing an detokenizing the data based on our sentencepiece byte-pair encoding model."
      ],
      "metadata": {
        "id": "NlEW7GndbCfw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def tokenize(sentence, sp_model, add_bos = True, add_eos = True):\n",
        "    bos = [sp_model.bos_id()] if add_bos else []\n",
        "    eos = [sp_model.eos_id()] if add_eos else []\n",
        "    inputs = bos + sp_model.encode_as_ids(sentence) + eos\n",
        "    return np.reshape(np.array(inputs), [1, -1])\n",
        "def detokenize(tokenized, sp_model, is_bos = True, is_eos = True):\n",
        "    integers = np.squeeze(tokenized).tolist()\n",
        "    if is_eos:\n",
        "        return sp_model.DecodeIdsWithCheck(integers[int(is_bos):integers.index(sp_model.eos_id())])\n",
        "    else:\n",
        "        if sp_model.pad_id() in tokenized:\n",
        "            return sp_model.DecodeIdsWithCheck(integers[int(is_bos):integers.index(sp_model.pad_id())])\n",
        "        else:\n",
        "            return sp_model.DecodeIdsWithCheck(integers[int(is_bos):])"
      ],
      "metadata": {
        "id": "VC-6hav2BrJJ"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We have tokenized the data."
      ],
      "metadata": {
        "id": "5-76mwBnbuzp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Tokenization\n",
        "src_train_data_enc = [np.squeeze(tokenize(train_data['src'].iloc[i], sp_en_bpe)) for i in range(train_data.shape[0])]\n",
        "tgt_train_data_enc = [np.squeeze(tokenize(train_data['tgt'].iloc[i], sp_ben_bpe)) for i in range(train_data.shape[0])]\n",
        "src_train_dev_data_enc = [np.squeeze(tokenize(train_dev_data['src'].iloc[i], sp_en_bpe)) for i in range(train_dev_data.shape[0])]\n",
        "tgt_train_dev_data_enc = [np.squeeze(tokenize(train_dev_data['tgt'].iloc[i], sp_ben_bpe)) for i in range(train_dev_data.shape[0])]\n",
        "src_test_val_data_enc = [np.squeeze(tokenize(test_val_data['src'].iloc[i], sp_en_bpe)) for i in range(test_val_data.shape[0])]\n",
        "tgt_test_val_data_enc = [np.squeeze(tokenize(test_val_data['tgt'].iloc[i], sp_ben_bpe)) for i in range(test_val_data.shape[0])]"
      ],
      "metadata": {
        "id": "nT0QpepuDoFF"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We have padded the data with post-truncation and post padding upto MAX_TOKENS number of tokens."
      ],
      "metadata": {
        "id": "wdyEjbm5hWXg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Padding\n",
        "# The reason why we add MAX_TOKENS+1 after-pad length for Bengali is that when we form the tf.Dataset later, we will shift the target sentence once and will truncate the last token once\n",
        "# Hence, it will get adjusted\n",
        "train_src = tf.keras.preprocessing.sequence.pad_sequences(src_train_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "train_tgt = tf.keras.preprocessing.sequence.pad_sequences(tgt_train_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "train_dev_src = tf.keras.preprocessing.sequence.pad_sequences(src_train_dev_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "train_dev_tgt = tf.keras.preprocessing.sequence.pad_sequences(tgt_train_dev_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "test_val_src = tf.keras.preprocessing.sequence.pad_sequences(src_test_val_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "test_val_tgt = tf.keras.preprocessing.sequence.pad_sequences(tgt_test_val_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')"
      ],
      "metadata": {
        "id": "NT63vC8PcK-2"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below, we discard all those pairs where the tokenized source or target sentences are larger than MAX_TOKENS number of tokens."
      ],
      "metadata": {
        "id": "FvxtnfKniMRA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Removing all pairs whose either the target sentence or source setence exceeds MAX_TOKENS number of tokens\n",
        "same_idx = list(set([i for i in range(train_src.shape[0]) if train_src[i][-1] == 1 or train_src[i][-1] == 0]).intersection(set([i for i in range(train_tgt.shape[0]) if train_tgt[i][-1] == 1 or train_tgt[i][-1] == 0])))\n",
        "train_src = train_src[same_idx]\n",
        "train_tgt = train_tgt[same_idx]\n",
        "same_idx = list(set([i for i in range(train_dev_src.shape[0]) if train_dev_src[i][-1] == 1 or train_dev_src[i][-1] == 0]).intersection(set([i for i in range(train_dev_tgt.shape[0]) if train_dev_tgt[i][-1] == 1 or train_dev_tgt[i][-1] == 0])))\n",
        "train_dev_src = train_dev_src[same_idx]\n",
        "train_dev_tgt = train_dev_tgt[same_idx]\n",
        "same_idx = list(set([i for i in range(test_val_src.shape[0]) if test_val_src[i][-1] == 1 or test_val_src[i][-1] == 0]).intersection(set([i for i in range(test_val_tgt.shape[0]) if test_val_tgt[i][-1] == 1 or test_val_tgt[i][-1] == 0])))\n",
        "test_val_src = test_val_src[same_idx]\n",
        "test_val_tgt = test_val_tgt[same_idx]"
      ],
      "metadata": {
        "id": "L6-cV6zpcO5e"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_labels = np.zeros(train_tgt.shape)\n",
        "train_labels[:,0:train_tgt.shape[1] -1] = train_tgt[:,1:]\n",
        "train_dev_labels = np.zeros(train_dev_tgt.shape)\n",
        "train_dev_labels[:,0:train_dev_tgt.shape[1] -1] = train_dev_tgt[:,1:]\n",
        "test_val_labels = np.zeros(test_val_tgt.shape)\n",
        "test_val_labels[:,0:test_val_tgt.shape[1] -1] = test_val_tgt[:,1:]"
      ],
      "metadata": {
        "id": "JGMvPvIxOx-0"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"*** Preprocessed Data ***\")\n",
        "print(\"Train Data Shape(Source): \", train_src.shape)\n",
        "print(\"Train Data Shape(Target): \", train_tgt.shape)\n",
        "print(\"Train Labels Shape(Target): \", train_labels.shape)\n",
        "print(\"Train Dev Data Shape(Source): \", train_dev_src.shape)\n",
        "print(\"Train Dev Data Shape(Target): \", train_dev_tgt.shape)\n",
        "print(\"Train Dev Labels Shape(Target): \", train_dev_labels.shape)\n",
        "print(\"Test Val Data Shape(Source): \", test_val_src.shape)\n",
        "print(\"Test Val Data Shape(Target): \", test_val_tgt.shape)\n",
        "print(\"Test Labels Shape(Target): \", test_val_labels.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rO5HD33XOGxf",
        "outputId": "a393adca-d4e1-455f-9ffc-9338d0d0fd6e"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "*** Preprocessed Data ***\n",
            "Train Data Shape(Source):  (154836, 60)\n",
            "Train Data Shape(Target):  (154836, 60)\n",
            "Train Labels Shape(Target):  (154836, 60)\n",
            "Train Dev Data Shape(Source):  (3649, 60)\n",
            "Train Dev Data Shape(Target):  (3649, 60)\n",
            "Train Dev Labels Shape(Target):  (3649, 60)\n",
            "Test Val Data Shape(Source):  (1194, 60)\n",
            "Test Val Data Shape(Target):  (1194, 60)\n",
            "Test Labels Shape(Target):  (1194, 60)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below we prepare the data in the form ready to feed to the Model."
      ],
      "metadata": {
        "id": "xtsMxbAYzePz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "train_data_prep = tf.data.Dataset.from_tensor_slices((train_src, train_tgt, train_labels))\n",
        "train_dev_data_prep = tf.data.Dataset.from_tensor_slices((train_dev_src, train_dev_tgt, train_dev_labels))\n",
        "test_val_data_prep = tf.data.Dataset.from_tensor_slices((test_val_src, test_val_tgt, test_val_labels))"
      ],
      "metadata": {
        "id": "_OkR5tbtyBQj"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below, we can see how our data looks"
      ],
      "metadata": {
        "id": "jSGBKDhc2sc-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for encoder_inputs, decoder_inputs, labels in train_data_prep.take(1):\n",
        "    print(f'encoder_inputs.shape: {encoder_inputs.shape}')\n",
        "    print(f'decoder_inputs.shape: {decoder_inputs.shape}')\n",
        "    print(f\"labels.shape: {labels.shape}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ao1TX6GHzWn7",
        "outputId": "3fb5265b-bf85-4b5e-8a73-65a66afd1aba"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "encoder_inputs.shape: (60,)\n",
            "decoder_inputs.shape: (60,)\n",
            "labels.shape: (60,)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Model"
      ],
      "metadata": {
        "id": "zjXk6pt-28kL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will use the Transformer Architechture with the following hyperparameters."
      ],
      "metadata": {
        "id": "a9UFHuFO3dHe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Transformer parameters\n",
        "d_model = 256 # 512 in the original paper\n",
        "d_k = 32 # 64 in the original paper\n",
        "d_v = 32 # 64 in the original paper\n",
        "n_heads = 4 # 8 in the original paper\n",
        "n_encoder_layers = 3 # 6 in the original paper\n",
        "n_decoder_layers = 3 # 6 in the original paper\n",
        "\n",
        "max_token_length = MAX_TOKENS # 512 in the original paper"
      ],
      "metadata": {
        "id": "v9ULcpgbJ9WP"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class SingleHeadAttention(Layer):\n",
        "  def __init__(self, input_shape=(3, -1, d_model), dropout=.0, masked=None):\n",
        "    super(SingleHeadAttention, self).__init__()\n",
        "    self.q = Dense(d_k, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.normalize_q = Lambda(lambda x: x / np.sqrt(d_k))\n",
        "    self.k = Dense(d_k, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.v = Dense(d_v, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.dropout = dropout\n",
        "    self.masked = masked\n",
        "  \n",
        "  # Inputs: [query, key, value]\n",
        "  def call(self, inputs, training=None):\n",
        "    assert len(inputs) == 3\n",
        "    # We use a lambda layer to divide vector q by sqrt(d_k) according to the equation\n",
        "    q = self.normalize_q(self.q(inputs[0]))\n",
        "    k = self.k(inputs[1])\n",
        "    # The dimensionality of q is (batch_size, query_length, d_k) and that of k is (batch_size, key_length, d_k)\n",
        "    # So we will do a matrix multication by batch after transposing last 2 dimensions of k\n",
        "    # tf.shape(attn_weights) = (batch_size, query_length, key_length)\n",
        "    attn_weights = tf.matmul(q, tf.transpose(k, perm=[0,2,1]))\n",
        "    if self.masked: # Prevent future attentions in decoding self-attention\n",
        "      # Create a matrix where the strict upper triangle (not including main diagonal) is filled with -inf and 0 elsewhere\n",
        "      length = tf.shape(attn_weights)[-1]\n",
        "      #attn_mask = np.triu(tf.fill((length, length), -np.inf), k=1) # We need to use tensorflow functions instead of numpy\n",
        "      attn_mask = tf.fill((length, length), -np.inf)\n",
        "      attn_mask = tf.linalg.band_part(attn_mask, 0, -1) # Get upper triangle\n",
        "      attn_mask = tf.linalg.set_diag(attn_mask, tf.zeros((length))) # Set diagonal to zeros to avoid operations with infinity\n",
        "      # This matrix is added to the attention weights so all future attention will have -inf logits (0 after softmax)\n",
        "      attn_weights += attn_mask\n",
        "    # Softmax along the last dimension\n",
        "    attn_weights = tf.nn.softmax(attn_weights, axis=-1)\n",
        "    if training: # Attention dropout included in the original paper. This is possibly to encourage multihead diversity.\n",
        "      attn_weights = tf.nn.dropout(attn_weights, rate=self.dropout)\n",
        "    v = self.v(inputs[2])\n",
        "    return tf.matmul(attn_weights, v)\n",
        "\n",
        "class MultiHeadAttention_my(Layer):\n",
        "  def __init__(self, dropout=.0, masked=None):\n",
        "    super(MultiHeadAttention_my, self).__init__()\n",
        "    self.attn_heads = list()\n",
        "    for i in range(n_heads): \n",
        "      self.attn_heads.append(SingleHeadAttention(dropout=dropout, masked=masked))\n",
        "    self.linear = Dense(d_model, input_shape=(-1, n_heads * d_v), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    \n",
        "  def call(self, x, training=None):\n",
        "    attentions = [self.attn_heads[i](x, training=training) for i in range(n_heads)]\n",
        "    concatenated_attentions = tf.concat(attentions, axis=-1)\n",
        "    return self.linear(concatenated_attentions)\n",
        "\n",
        "class TransformerEncoder(Layer):\n",
        "  def __init__(self, dropout=.1, attention_dropout=.0, **kwargs):\n",
        "    super(TransformerEncoder, self).__init__(**kwargs)\n",
        "    self.dropout_rate = dropout\n",
        "    self.attention_dropout_rate = attention_dropout\n",
        "  def build(self, input_shape):\n",
        "    self.multihead_attention = MultiHeadAttention_my(dropout=self.attention_dropout_rate)\n",
        "    self.dropout1 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization1 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.linear1 = Dense(input_shape[-1] * 4, input_shape=input_shape, activation='relu',\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.linear2 = Dense(input_shape[-1], input_shape=self.linear1.compute_output_shape(input_shape),\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.dropout2 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization2 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    super(TransformerEncoder, self).build(input_shape)\n",
        "  def call(self, x, training=None):\n",
        "    sublayer1 = self.multihead_attention((x, x, x), training=training)\n",
        "    sublayer1 = self.dropout1(sublayer1, training=training)\n",
        "    layernorm1 = self.layer_normalization1(x + sublayer1)\n",
        "    \n",
        "    sublayer2 = self.linear2(self.linear1(layernorm1))\n",
        "    sublayer1 = self.dropout2(sublayer2, training=training)\n",
        "    layernorm2 = self.layer_normalization2(layernorm1 + sublayer2)\n",
        "    return layernorm2\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape\n",
        "\n",
        "class TransformerDecoder(Layer):\n",
        "  def __init__(self, dropout=.0, attention_dropout=.0, **kwargs):\n",
        "    super(TransformerDecoder, self).__init__(**kwargs)\n",
        "    self.dropout_rate = dropout\n",
        "    self.attention_dropout_rate = attention_dropout\n",
        "  def build(self, input_shape):\n",
        "    self.multihead_self_attention = MultiHeadAttention_my(dropout=self.attention_dropout_rate, masked=True)\n",
        "    self.dropout1 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization1 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.multihead_encoder_attention = MultiHeadAttention_my(dropout=self.attention_dropout_rate)\n",
        "    self.dropout2 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization2 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.linear1 = Dense(input_shape[-1] * 4, input_shape=input_shape, activation='relu',\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.linear2 = Dense(input_shape[-1], input_shape=self.linear1.compute_output_shape(input_shape),\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.dropout3 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization3 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    super(TransformerDecoder, self).build(input_shape)\n",
        "  def call(self, x, hidden, training=None):\n",
        "    sublayer1 = self.multihead_self_attention((x, x, x))\n",
        "    sublayer1 = self.dropout1(sublayer1, training=training)\n",
        "    layernorm1 = self.layer_normalization1(x + sublayer1)\n",
        "    \n",
        "    sublayer2 = self.multihead_encoder_attention((x, hidden, hidden))\n",
        "    sublayer2 = self.dropout2(sublayer2, training=training)\n",
        "    layernorm2 = self.layer_normalization2(layernorm1 + sublayer2)\n",
        "    \n",
        "    sublayer3 = self.linear2(self.linear1(layernorm1))\n",
        "    sublayer3 = self.dropout3(sublayer3, training=training)\n",
        "    layernorm3 = self.layer_normalization2(layernorm2 + sublayer3)\n",
        "    return layernorm3\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape\n",
        "\n",
        "class SinusoidalPositionalEncoding(Layer): # This is a TensorFlow implementation of https://github.com/graykode/nlp-tutorial/blob/master/5-1.Transformer/Transformer_Torch.ipynb\n",
        "  def __init__(self):\n",
        "    super(SinusoidalPositionalEncoding, self).__init__()\n",
        "    self.sinusoidal_encoding = np.array([self.get_positional_angle(pos) for pos in range(max_token_length)], dtype=np.float32)\n",
        "    self.sinusoidal_encoding[:, 0::2] = np.sin(self.sinusoidal_encoding[:, 0::2])\n",
        "    self.sinusoidal_encoding[:, 1::2] = np.cos(self.sinusoidal_encoding[:, 1::2])\n",
        "    self.sinusoidal_encoding = tf.cast(self.sinusoidal_encoding, dtype=tf.float32) # Casting the array to Tensor for slicing\n",
        "  def call(self, x):\n",
        "    return x + self.sinusoidal_encoding[:tf.shape(x)[1]]\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape\n",
        "  def get_angle(self, pos, dim):\n",
        "    return pos / np.power(10000, 2 * (dim // 2) / d_model)\n",
        "  def get_positional_angle(self, pos):\n",
        "    return [self.get_angle(pos, dim) for dim in range(d_model)]\n",
        "\n",
        "class Transformer(Model):\n",
        "  def __init__(self, dropout=.1, attention_dropout=.0, **kwargs):\n",
        "    super(Transformer, self).__init__(**kwargs)\n",
        "    self.encoding_embedding = Embedding(VOCAB_SIZE, d_model)\n",
        "    self.decoding_embedding = Embedding(VOCAB_SIZE, d_model)\n",
        "    self.pos_encoding = SinusoidalPositionalEncoding()\n",
        "    self.encoder = [TransformerEncoder(dropout=dropout, attention_dropout=attention_dropout) for i in range(n_encoder_layers)]\n",
        "    self.decoder = [TransformerDecoder(dropout=dropout, attention_dropout=attention_dropout) for i in range(n_decoder_layers)]\n",
        "    self.decoder_final = Dense(VOCAB_SIZE, input_shape=(None, d_model))\n",
        "  def call(self, inputs, training=None): # Source_sentence and decoder_input\n",
        "    source_sentence, decoder_input = inputs\n",
        "    embedded_source = self.encoding_embedding(source_sentence)\n",
        "    encoder_output = self.pos_encoding(embedded_source)\n",
        "    for encoder_unit in self.encoder:\n",
        "      encoder_output = encoder_unit(encoder_output, training=training)\n",
        "    \n",
        "    embedded_target = self.decoding_embedding(decoder_input)\n",
        "    decoder_output = self.pos_encoding(embedded_target)\n",
        "    for decoder_unit in self.decoder:\n",
        "      decoder_output = decoder_unit(decoder_output, encoder_output, training=training)\n",
        "    if training:\n",
        "      decoder_output = self.decoder_final(decoder_output)\n",
        "      decoder_output = tf.nn.softmax(decoder_output, axis=-1)\n",
        "    else:\n",
        "      decoder_output = self.decoder_final(decoder_output[:, -1:, :])\n",
        "      decoder_output = tf.nn.softmax(decoder_output, axis=-1)\n",
        "    return decoder_output"
      ],
      "metadata": {
        "id": "KvsIOzPnzt4F"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "transformer = Transformer(dropout=0.3) # Instantiating a new transformer model\n",
        "train = [tf.cast(train_src, dtype=tf.float32), tf.cast(train_tgt, dtype=tf.float32)] # Cast the tuples to tensors\n",
        "validation = [tf.cast(train_dev_src, dtype=tf.float32), tf.cast(train_dev_tgt, dtype=tf.float32)]"
      ],
      "metadata": {
        "id": "eRsGRYTR6XQW"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "transformer.build([train_src.shape, train_tgt.shape])\n",
        "transformer.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IHXUpv-0eh-b",
        "outputId": "268e6425-25a1-4633-91cb-607745f23121"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"transformer\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " embedding (Embedding)       multiple                  8192000   \n",
            "                                                                 \n",
            " embedding_1 (Embedding)     multiple                  8192000   \n",
            "                                                                 \n",
            " sinusoidal_positional_encod  multiple                 0         \n",
            " ing (SinusoidalPositionalEn                                     \n",
            " coding)                                                         \n",
            "                                                                 \n",
            " transformer_encoder (Transf  multiple                 658304    \n",
            " ormerEncoder)                                                   \n",
            "                                                                 \n",
            " transformer_encoder_1 (Tran  multiple                 658304    \n",
            " sformerEncoder)                                                 \n",
            "                                                                 \n",
            " transformer_encoder_2 (Tran  multiple                 658304    \n",
            " sformerEncoder)                                                 \n",
            "                                                                 \n",
            " transformer_decoder (Transf  multiple                 790016    \n",
            " ormerDecoder)                                                   \n",
            "                                                                 \n",
            " transformer_decoder_1 (Tran  multiple                 790016    \n",
            " sformerDecoder)                                                 \n",
            "                                                                 \n",
            " transformer_decoder_2 (Tran  multiple                 790016    \n",
            " sformerDecoder)                                                 \n",
            "                                                                 \n",
            " dense (Dense)               multiple                  8224000   \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 28,952,960\n",
            "Trainable params: 28,952,960\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training"
      ],
      "metadata": {
        "id": "0JkChx5Ndp1s"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "transformer.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])"
      ],
      "metadata": {
        "id": "InXkWf-LawJi"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "EPOCHS = 10\n",
        "checkpoint_filepath = '/Models/checkpoint'\n",
        "model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n",
        "    filepath=checkpoint_filepath,\n",
        "    save_weights_only=True,\n",
        "    monitor = \"accuracy\",\n",
        "    mode='max',\n",
        "    save_best_only=True)"
      ],
      "metadata": {
        "id": "LMttctIwaNxG"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "with tf.device('/device:GPU:0'):\n",
        "    history = transformer.fit(train, tf.cast(train_labels, dtype=tf.float32), batch_size=256, epochs=EPOCHS, callbacks=[model_checkpoint_callback])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bzdsT8kmBEMo",
        "outputId": "c0c6c28b-e007-40e4-c97a-b8f184577da4"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.8873 - accuracy: 0.7958WARNING:tensorflow:Can save best model only with val_accuracy available, skipping.\n",
            "605/605 [==============================] - 421s 664ms/step - loss: 1.8873 - accuracy: 0.7958\n",
            "Epoch 2/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.7724 - accuracy: 0.7914WARNING:tensorflow:Can save best model only with val_accuracy available, skipping.\n",
            "605/605 [==============================] - 405s 669ms/step - loss: 1.7724 - accuracy: 0.7914\n",
            "Epoch 3/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.7127 - accuracy: 0.7959WARNING:tensorflow:Can save best model only with val_accuracy available, skipping.\n",
            "605/605 [==============================] - 405s 670ms/step - loss: 1.7127 - accuracy: 0.7959\n",
            "Epoch 4/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.6009 - accuracy: 0.8025WARNING:tensorflow:Can save best model only with val_accuracy available, skipping.\n",
            "605/605 [==============================] - 406s 670ms/step - loss: 1.6009 - accuracy: 0.8025\n",
            "Epoch 5/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.5697 - accuracy: 0.8038WARNING:tensorflow:Can save best model only with val_accuracy available, skipping.\n",
            "605/605 [==============================] - 405s 670ms/step - loss: 1.5697 - accuracy: 0.8038\n",
            "Epoch 6/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.5460 - accuracy: 0.8047WARNING:tensorflow:Can save best model only with val_accuracy available, skipping.\n",
            "605/605 [==============================] - 405s 670ms/step - loss: 1.5460 - accuracy: 0.8047\n",
            "Epoch 7/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.5196 - accuracy: 0.8057WARNING:tensorflow:Can save best model only with val_accuracy available, skipping.\n",
            "605/605 [==============================] - 405s 670ms/step - loss: 1.5196 - accuracy: 0.8057\n",
            "Epoch 8/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.4943 - accuracy: 0.8067WARNING:tensorflow:Can save best model only with val_accuracy available, skipping.\n",
            "605/605 [==============================] - 405s 670ms/step - loss: 1.4943 - accuracy: 0.8067\n",
            "Epoch 9/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.4733 - accuracy: 0.8074WARNING:tensorflow:Can save best model only with val_accuracy available, skipping.\n",
            "605/605 [==============================] - 405s 670ms/step - loss: 1.4733 - accuracy: 0.8074\n",
            "Epoch 10/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.4559 - accuracy: 0.8080WARNING:tensorflow:Can save best model only with val_accuracy available, skipping.\n",
            "605/605 [==============================] - 405s 670ms/step - loss: 1.4559 - accuracy: 0.8080\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "transformer.save_weights(\"model_weights\")\n",
        "# transformer.save(\"./Models/\")"
      ],
      "metadata": {
        "id": "6tmeafiFdZ_U"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(10, 10))\n",
        "plt.plot(history.history['accuracy'])\n",
        "plt.title('model accuracy')\n",
        "plt.ylabel('accuracy')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train'], loc='upper left')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "JbSCoj0cft6O",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 621
        },
        "outputId": "eccdb6b1-1ca7-4966-b7bc-3ea43500c808"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 720x720 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnQAAAJcCAYAAABwljmUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd5hedZn/8fedySST3nshCem0AEPokAgogoiKUlzrKmzDRRd3F113l9Xd/VlR3LUBVlyaKIKKlGAGEIIhgVBCMull0mZSJ22SKd/fH/NEhxiSCckzZ+aZ9+u65vI8pz33CTL58D3n3N9IKSFJkqS2q0PWBUiSJOnIGOgkSZLaOAOdJElSG2egkyRJauMMdJIkSW2cgU6SJKmNM9BJanci4kcR8Z/N3HdFRFyY75ok6UgY6CRJkto4A50ktVER0THrGiS1DgY6Sa1S7lbnP0bEyxGxMyK+HxGDIuK3EbE9ImZERJ8m+78zIuZHxNaIKIuISU22nRwRL+SOuxco2e+73hER83LHPhsRJzazxksj4sWIqI6I1RFx837bz8mdb2tu+0dy67tExNciYmVEbIuI3+fWTYuIigP8OVyYW745Iu6PiJ9GRDXwkYiYGhGzct+xLiL+NyI6NTn+uIh4PCI2R8SGiPhsRAyOiF0R0a/JfqdERFVEFDfn2iW1LgY6Sa3ZFcBFwHjgMuC3wGeBATT+/vp7gIgYD9wNfDK37WHgVxHRKRdufgncCfQFfpY7L7ljTwZ+APwV0A/4HvBQRHRuRn07gQ8BvYFLgb+JiHflzntMrt7/ydU0BZiXO+6rwKnAWbma/gloaOafyeXA/bnv/D+gHvgU0B84E7gA+NtcDT2AGcAjwFBgLPBESmk9UAZc2eS8HwTuSSnVNrMOSa2IgU5Sa/Y/KaUNKaU1wNPAH1JKL6aUaoAHgJNz+10F/Cal9HgukHwV6EJjYDoDKAa+kVKqTSndDzzf5DuuA76XUvpDSqk+pfRjYE/uuINKKZWllF5JKTWklF6mMVSen9v8fmBGSunu3PduSinNi4gOwF8CN6SU1uS+89mU0p5m/pnMSin9Mvedu1NKc1NKz6WU6lJKK2gMpPtqeAewPqX0tZRSTUppe0rpD7ltPwY+ABARRcA1NIZeSW2QgU5Sa7ahyfLuA3zunlseCqzctyGl1ACsBobltq1JKaUmx65ssnwMcGPuluXWiNgKjMgdd1ARcXpEzMzdqtwG/DWNI2XkzrH0AIf1p/GW74G2Ncfq/WoYHxG/joj1uduw/92MGgAeBCZHxGgaR0G3pZRmv8maJGXMQCepEKylMZgBEBFBY5hZA6wDhuXW7TOyyfJq4L9SSr2b/HRNKd3djO+9C3gIGJFS6gV8F9j3PauBYw9wzEag5g227QS6NrmOIhpv1zaV9vv8HWAhMC6l1JPGW9JNaxhzoMJzo5z30ThK90EcnZPaNAOdpEJwH3BpRFyQe6j/Rhpvmz4LzALqgL+PiOKIeA8wtcmxtwN/nRtti4jolnvZoUczvrcHsDmlVBMRU2m8zbrP/wEXRsSVEdExIvpFxJTc6OEPgFsiYmhEFEXEmbln9hYBJbnvLwY+BxzqWb4eQDWwIyImAn/TZNuvgSER8cmI6BwRPSLi9CbbfwJ8BHgnBjqpTTPQSWrzUkrlNI40/Q+NI2CXAZellPamlPYC76ExuGym8Xm7XzQ5dg5wLfC/wBZgSW7f5vhb4PMRsR34NxqD5b7zrgIuoTFcbqbxhYiTcps/DbxC47N8m4EvAR1SStty57yDxtHFncDr3no9gE/TGCS30xhO721Sw3Yab6deBqwHFgPTm2x/hsaXMV5IKTW9DS2pjYnXP1YiSWpPIuJ3wF0ppTuyrkXSm2egk6R2KiJOAx6n8RnA7VnXI+nN85arJLVDEfFjGnvUfdIwJ7V9jtBJkiS1cY7QSZIktXHtemLn/v37p1GjRmVdhiRJ0iHNnTt3Y0pp/96UQDsPdKNGjWLOnDlZlyFJknRIEfGG7YW85SpJktTGGegkSZLaOAOdJElSG9eun6E7kNraWioqKqipqcm6lLwrKSlh+PDhFBcXZ12KJEk6Aga6/VRUVNCjRw9GjRpFRGRdTt6klNi0aRMVFRWMHj0663IkSdIR8JbrfmpqaujXr19BhzmAiKBfv37tYiRSkqRCZ6A7gEIPc/u0l+uUJKnQGegkSZLaOANdK7N161a+/e1vH/Zxl1xyCVu3bs1DRZIkqbUz0LUybxTo6urqDnrcww8/TO/evfNVliRJasV8y7WVuemmm1i6dClTpkyhuLiYkpIS+vTpw8KFC1m0aBHvete7WL16NTU1Ndxwww1cd911wJ+mMduxYwdvf/vbOeecc3j22WcZNmwYDz74IF26dMn4yiRJUr4Y6A7iP341n9fWVh/Vc04e2pN/v+y4N9z+xS9+kVdffZV58+ZRVlbGpZdeyquvvvrH1iI/+MEP6Nu3L7t37+a0007jiiuuoF+/fq87x+LFi7n77ru5/fbbufLKK/n5z3/OBz7wgaN6HZIkqfUw0LVyU6dOfV2fuG9+85s88MADAKxevZrFixf/WaAbPXo0U6ZMAeDUU09lxYoVLVavJElqeQa6gzjYSFpL6dat2x+Xy8rKmDFjBrNmzaJr165MmzbtgH3kOnfu/MfloqIidu/e3SK1SpKkbPhSRCvTo0cPtm/ffsBt27Zto0+fPnTt2pWFCxfy3HPPtXB1kiSpNXKErpXp168fZ599NscffzxdunRh0KBBf9x28cUX893vfpdJkyYxYcIEzjjjjAwrlSRJrUWklLKuITOlpaVpzpw5r1u3YMECJk2alFFFLa+9Xa8kSW1VRMxNKZUeaJu3XCVJkto4A50kSVIbZ6A7gPZyG7q9XKckSYXOQLefkpISNm3aVPBhJ6XEpk2bKCkpyboUSZJ0hHzLdT/Dhw+noqKCqqqqrEvJu5KSEoYPH551GZIktXkpJSIis+830O2nuLj4dTMzSJIk7S+lxOLKHcxYsIEZr21g2oSB/P0F4zKrx0AnSZLUDLX1DTy/YjMzXqtkxoINrNq8C4ATh/dicM9sH2Ey0EmSJL2B6ppaniyvYsaCDcxcWEl1TR2dOnbg7GP78Vfnj+GCiYMY3Cv759ENdJIkSU2s3ryLJxZsYMaCSp5btom6hkS/bp1423GDuWDSIM4d159unVtXhGpd1UiSJLWwhobEK2u2MWPBBh5/bQML1zfOqT52YHc+fu4YLpo8kCkj+lDUIbuXHg7FQCdJktqdmtp6nl26kcdfq+SJBRuo3L6HDgGnjerL5y6dxAWTBjG6f7esy2w2A50kSWoXqrbvYebCSh5fsIHfL97I7tp6unUqYtqEgVw4eSDTxg+kT7dOWZf5phjoJElSQUopsaRyB4/nWou8uHorKcHQXiW8r3Q4F04axOlj+tK5Y1HWpR4xA50kSSoYdfUNPL9iS2N/uAUbWLmpsbXICcN68ckLxnPh5IFMHtIz0ybA+WCgkyRJbVp1TS1PLapixmsbmFlexbbdtXQq6sBZY/tx7bljuGDSQIb06pJ1mXlloJMkSW1OxZZdPLGgscHvc8s2UVuf6NO1mAsnDeKiyQM5d9yAVtdaJJ/az5VKkqQ2q6Eh8erabcx4bQOPL6hkwbpqAMYM6MZfnj2aCycP4pSRrbu1SD4Z6CRJUqu0r7XIjAWNrUU2VDe2Fikd1ZfPXjKRCyYN4tgB3bMus1XIa6CLiIuBW4Ei4I6U0hf32z4S+DHQO7fPTSmlh3PbPgN8DKgH/j6l9GhETADubXKKMcC/pZS+ERE3A9cCVbltn913LkmS1DZs3LGH3y2sZMZrG3i6SWuR8ycM4MJJg5g+oe22FsmnvAW6iCgCvgVcBFQAz0fEQyml15rs9jngvpTSdyJiMvAwMCq3fDVwHDAUmBER41NK5cCUJudfAzzQ5HxfTyl9NV/XJEmSjq6UEkurdvB4bsL7F1ZtISUY0quE9546nAsnD+KMAmktkk/5HKGbCixJKS0DiIh7gMuBpoEuAT1zy72Atbnly4F7Ukp7gOURsSR3vllNjr0AWJpSWpm/S5AkSUdbXX0Dc1ZuYcZrja1FVuRaixw/rCc3XDCOCycN4rihhddaJJ/yGeiGAaubfK4ATt9vn5uBxyLiE0A34MImxz6337HD9jv2auDu/dZdHxEfAuYAN6aUtuxfVERcB1wHMHLkyOZeiyRJOgLba2p5atFGZizYwO8WVv6xtciZx/bjY+eO4YKJAxnau7Bbi+RT1i9FXAP8KKX0tYg4E7gzIo4/1EER0Ql4J/CZJqu/A3yBxlG/LwBfA/5y/2NTSrcBtwGUlpamI74CSZJ0QDW19Tw6fz33z614XWuRCyYN5KJJgzh3/AC6t6PWIvmUzz/FNcCIJp+H59Y19THgYoCU0qyIKAH6N+PYtwMvpJQ27FvRdDkibgd+fRSuQZIkHabFG7Zz9+zV/OLFCrbuqmVE3y589OzRXDhpEKeM7E3Hog5Zl1hw8hnongfGRcRoGsPY1cD799tnFY3Pwv0oIiYBJTS+pfoQcFdE3ELjSxHjgNlNjruG/W63RsSQlNK63Md3A68e3cuRJElvpKa2nodfWcfds1fx/IotFBcFb508mGumjuSsY/vRoZ32h2speQt0KaW6iLgeeJTGliQ/SCnNj4jPA3NSSg8BNwK3R8SnaLxV+pGUUgLmR8R9NL5AUQf8XUqpHiAiutH45uxf7feVX46IKbnzrDjAdkmSdJSVr9/O3bNX8YsXKqiuqWNUv6585u0TueLU4fTv3jnr8tqNaMxP7VNpaWmaM2dO1mVIktSm7N5bz69fXss9z69m7sotdCrqwNuOH8w1U0dwxmhH4/IlIuamlEoPtM0nESVJUrMsWFfN3bNX8cCLa9heU8eY/t34l0sm8Z5ThtHP0bhMGegkSdIb2rW3jl+/tI67Zq9i3uqtdOrYgUuOb3w2burovvaKayUMdJIk6c+8umYb9zy/il++uJYde+oYO7A7//qOybzn5GFOvdUKGegkSRIAO/bU8auX1nL37FW8XLGNzh07cOmJQ7hm6khKj+njaFwrZqCTJKmde6ViG3fNXsVD89awc289Ewb14ObLJvPuk4fTq2tx1uWpGQx0kiS1Q9tranlwXuNo3Py11ZQUd+AdJw7lmqkjOWVkb0fj2hgDnSRJ7URKiZcqtnH3H1bx0Etr2V1bz8TBPfjC5cfxzinD6NXF0bi2ykAnSVKBq66p5cEX13DX7NUsWFdNl+Ii3nnSUK45fSQnDe/laFwBMNBJklSAUkq8sGord89exa9fXktNbQPHDe3Jf737eN550lB6lDgaV0gMdJIkFZBtu2p54MUK7p69mvIN2+nWqYh3nzyc908dyQnDe2VdnvLEQCdJUhuXUmLuyi3cNXsVv3l5HXvqGjhxeC/+33tO4LKThtK9s3/dFzr/CUuS1EZt3bWXn7+whrtnr2JJ5Q66d+7I+0qHc/VpIzl+mKNx7YmBTpKkNiSlxOzlm7l79ioefnU9e+samDKiN1++4kQuPXEI3RyNa5f8py5JUhuweedefvFCBXfNXsWyqp306NyRq08bwdWnjWTy0J5Zl6eMGegkSWqlUkrMWraJu2ev5tFX17O3voFTj+nDV983lktPGEKXTkVZl6hWwkAnSVIrs3HHHn4+t4J7nl/N8o076VnSkfefPpJrpo5kwuAeWZenVshAJ0lSK9DQ0Dgad9fsVTw2fz219Ympo/ryibeM5ZIThlBS7Gic3piBTpKkDG3asYd756zmntmrWbV5F727FvOhM0dxzdQRjB3oaJyax0AnSVIGtu2u5fanlvGDZ5aza289p4/uy41vHc/bjhvsaJwOm4FOkqQWtGtvHT98ZgXfe3Ip1TV1XHriED55wTjGDXI0Tm+egU6SpBawp66eu/6wim/NXMLGHXu5YOJA/uGt4zluqA2AdeQMdJIk5VFdfQP3z63gm08sZu22Gs4Y05fvfXAipx7TJ+vSVEAMdJIk5UFDQ+JXL6/lGzMWs3zjTk4a0Zsvv/ckzh7bj4jIujwVGAOdJElHUUqJGQsq+dpj5Sxcv52Jg3tw+4dKuXDSQIOc8sZAJ0nSUfLMko185dFy5q3eyuj+3bj16ilcduJQOnQwyCm/DHSSJB2huSu38NVHy5m1bBNDe5XwpStO4IpThtOxqEPWpamdMNBJkvQmzV+7jVseW8QTCyvp370T/37ZZK6ZOtI+cmpxBjpJkg7T0qod3PL4In7z8jp6lnTkH982gY+ePYqunfxrVdnw/3mSJDVTxZZd3DpjMT9/oYKS4iKunz6Wa88bQ68uxVmXpnbOQCdJ0iFUbq/hW79bwl2zVxERfPTs0fzNtGPp371z1qVJgIFOkqQ3tHXXXr775DJ+9OxyausTV5aO4O8vGMuQXl2yLk16HQOdJEn72bGnju8/vZw7nl7Gjr11XH7SUD554XhG9e+WdWnSARnoJEnKqamt585ZK/nOk0vZvHMvbztuEP9w0QQmDO6RdWnSQRnoJEnt3t66Bu6bs5r/+d1iNlTv4dxx/fn0Wydw0ojeWZcmNYuBTpLUbtU3JH754hq+8cQiVm/eTekxfbj16pM5Y0y/rEuTDouBTpLU7qSUeOTV9dzy+CIWV+7guKE9+eFHj2fa+AHOt6o2yUAnSWo3Uko8uaiKrz22iFfWbOPYAd349l+cwsXHDXa+VbVpBjpJUrswe/lmvvLoQp5fsYXhfbrw1fedxLtPHkaRQU4FwEAnSSpor1Rs4yuPlfPUoioG9ujMFy4/jqtOG0mnjh2yLk06agx0kqSCtGjDdm55bBGPzF9P767FfPaSiXzwjFF06VSUdWnSUWegkyQVlFWbdvGNGYt4YN4aunXqyCcvHMfHzhlNjxLnW1XhMtBJkgrC+m01fPN3i7nv+dV0LAquO3cMf33+sfTp1inr0qS8M9BJktq0TTv28J2ypfzkuZWklHj/6SO5fvpYBvYsybo0qcUY6CRJbVJ1TS13PLWM7/9+Obtr63nPKcO54YJxjOjbNevSpBZnoJMktSm79tbxo2dX8L0nl7Ftdy2XnjCET100nrEDu2ddmpQZA50kqU3YU1fP3X9Yxf/OXMrGHXuYPmEAN751AscP65V1aVLmDHSSpFatrr6BX7ywhlufWMyarbs5fXRfvvuBUygd1Tfr0qRWw0AnSWqVGhoSD7+6jlseW8SyjTs5aXgvvnjFCZwztr/zrUr7MdBJklqVlBJli6r46qPlzF9bzfhB3fneB0/lrZMHGeSkN2CgkyS1GnNWbObLj5Qze8VmRvTtwtevOol3nuR8q9KhGOgkSZmbv3YbX320nJnlVQzo0ZkvvOt4riod4XyrUjMZ6CRJmVm+cSe3PL6IX720ll5divnniyfykbOcb1U6XAY6SVKLW7dtN998YjH3zamgU1EHrp8+lmvPG0OvLs63Kr0ZBjpJUovZvHMv35655I/TdH3wjGP4u+ljGdCjc9alSW2agU6SlHc79tRxx9PLuOPp5ezaW+c0XdJRZqCTJOVNTW09P31uJd8uW8rmnXu5+LjB3PjW8Ywb1CPr0qSCYqCTJB11dfUN3D+3glufWMy6bTWcO64/n37rBE4a0Tvr0qSCZKCTJB01DQ2J37yyjlseX8TyjTs5eWRvvnblSZx1bP+sS5MKmoFOknTEUkqUlVfxlUfLeW1dNRMG9eD2D5Vy4aSBzu4gtQADnSTpiDy/YjNffmQhz6/Ywsi+XfnGVVO47KShzu4gtSADnSTpTXl1zTa++lg5ZeVVDHR2BylTBjpJ0mFZVrWDWx5fxK9fXkevLsXc9PaJfPhMZ3eQsmSgkyQ1y9qtjbM7/GxuBZ07duATbxnLx891dgepNTDQSZIOatOOPXy7bCl3PrcSEs7uILVCBjpJ0gFtr6nljqeXc8fTy9hdW88VpwznhgvHMbyPsztIrY2BTpL0OjW19dw5ayXfLlvCll21vP34xtkdxg50dgeptcproIuIi4FbgSLgjpTSF/fbPhL4MdA7t89NKaWHc9s+A3wMqAf+PqX0aG79CmB7bn1dSqk0t74vcC8wClgBXJlS2pLP65OkQlJb38DP5lTwzScWs766cXaHf3zbBE4c7uwOUmuXt0AXEUXAt4CLgArg+Yh4KKX0WpPdPgfcl1L6TkRMBh4GRuWWrwaOA4YCMyJifEqpPnfc9JTSxv2+8ibgiZTSFyPiptznf87X9UlSoWhoSPz6lXV8vcnsDl+/agpnHtsv69IkNVM+R+imAktSSssAIuIe4HKgaaBLQM/cci9gbW75cuCelNIeYHlELMmdb9ZBvu9yYFpu+cdAGQY6SXpDKSVmllfylUcXsWBdNRMH9+COD5VygbM7SG1OPgPdMGB1k88VwOn77XMz8FhEfALoBlzY5Njn9jt2WG455Y5JwPdSSrfl1g9KKa3LLa8HBh2oqIi4DrgOYOTIkYd5SZJUGGYvb5zdYc7KLRzTryu3Xj2Fy04cSgdnd5DapKxfirgG+FFK6WsRcSZwZ0Qcf4hjzkkprYmIgcDjEbEwpfRU0x1SSikX+P5MLgDeBlBaWnrAfSSpUL26ZhtfebScJxc1zu7wn+86nqtOG0FxkbM7SG1ZPgPdGmBEk8/Dc+ua+hhwMUBKaVZElAD9D3ZsSmnf/1ZGxAM03op9CtgQEUNSSusiYghQefQvSZLapqW52R1+8/I6enct5jNvn8iHnN1BKhj5DHTPA+MiYjSNYexq4P377bMKuAD4UURMAkqAKuAh4K6IuIXGlyLGAbMjohvQIaW0Pbf8VuDzuXM9BHwY+GLufx/M47VJUpuwdutubp2xmPtf+NPsDteeN4aeJc7uIBWSvAW6lFJdRFwPPEpjS5IfpJTmR8TngTkppYeAG4HbI+JTND4b95GUUgLmR8R9NL5AUQf8XUqpPiIGAQ/kHtbtCNyVUnok95VfBO6LiI8BK4Er83VtktTabdqxh2/NXMpPn1sJwIfObJzdoX93Z3eQClE05qf2qbS0NM2ZMyfrMiTpqKnOze7w/dzsDu89dTg3XDieYb27ZF2apCMUEXP39d/dX9YvRUiSjoKa2np+MmsF3y5bytZdtVxywmD+4aIJjB3YPevSJLUAA50ktWF76xr42dzVfPOJxWyo3sN54wfwj2+dwAnDe2VdmqQWZKCTpDZo2+5a7vrDKn707HI2VO/hlJG9ufXqkzljjLM7SO2RgU6S2pDVm3fxw2dWcO/zq9i5t56zx/bjS1ecyPnjBzi7g9SOGegkqQ14uWIrtz21jN++up4ALjtpKB8/dzTHDfXWqiQDnSS1Wg0Nid8trOT2p5fxh+Wb6dG5Ix8/ZzQfOXsUQ3r51qqkPzHQSVIrU1NbzwMvruH2p5exrGonQ3uV8LlLJ3HVaSPoYUNgSQdgoJOkVmLzzr3cOWslP5m1gk0793L8sJ7cevUULjlhiHOtSjooA50kZWxZ1Q6+//vl/PyFCmpqG3jLxIFce+4YzhjT1xcdJDWLgU6SMpBSYu7KLdz21DIeX7CB4g4dePfJw/j4uaMZN6hH1uVJamMMdJLUguobEo/OX89tTy1j3uqt9O5azPXTx/LBM49hYI+SrMuT1EYZ6CSpBezcU8fP5qzm+88sZ/Xm3RzTrytfuPw4rjh1OF07+atY0pHxt4gk5VFldQ0/nrWCnz63im27azn1mD78yyWTuGjyYIo6+HycpKPDQCdJebBow3Zuf2oZD85bS21DA2+bPJhrzxvNqcf0zbo0SQXIQCdJR0lKiWeXbuL2p5dRVl5FSXEHrp46gr88ezSj+nfLujxJBcxAJ0lHqLa+gd+8vI7bnlrGa+uq6d+9MzdeNJ4PnHEMfbp1yro8Se2AgU6S3qTqmlrumb2KHz6zgnXbahg7sDtfuuIELp8yjJLioqzLk9SOGOgk6TCt2bqbH/5+Ofc8v5ode+o4c0w//vvdJ3D++AF08EUHSRkw0ElSM726Zhu3P72MX7+8DoB3nDiEa88dw/HDemVcmaT2zkAnSQfR0JB4clEVtz21jFnLNtG9c0c+etYoPnrOaIb17pJ1eZIEGOgk6YBqaut5cN4abn96OUsqdzC4ZwmfvWQiV08dSc+S4qzLk6TXMdBJUhNbdu7lp8+t5MezVrJxxx4mD+nJ1686iUtPGEqnjh2yLk+SDshAJ0nAyk07+f7vl3PfnNXU1DZw/vgBXHfeGM46th8RvuggqXUz0Elq1+au3MLtTy3j0dfW07FD8K4pw/j4uWOYMLhH1qVJUrMZ6CS1O/UNicdfW8/tTy9n7sot9OpSzN9OO5YPnzmKgT1Lsi5Pkg6bgU5Su7F7bz33z13NHb9fzspNuxjRtws3XzaZ95WOoFtnfx1Karv8DSap4FVt38NPZq3gzudWsnVXLSeN6M0/XzyRtx03mCIbAUsqAAY6SQWrpraerzxazp3PraS2voGLJg3i2vPGUHpMH190kFRQDHSSCtKra7Zxwz0vsrRqJ1eVjuCvzh/DmAHdsy5LkvLCQCepoNQ3JL775FK+/vgi+nXvxJ0fm8q54wZkXZYk5ZWBTlLBWLVpF5+6bx5zV27hHScO4T/fdTy9u3bKuixJyjsDnaQ2L6XEz+ZU8B+/mk+HDsGtV0/hnScN9Tk5Se2GgU5Sm7Zpxx4+84tXeOy1DZwxpi9fu3IKw3p3ybosSWpRBjpJbdbvFm7gn+5/herdtfzLJZP42Dmj6WAbEkntkIFOUpuza28d//WbBfzfH1YxcXAPfvrxqUwc3DPrsiQpMwY6SW3KvNVb+dS981ixaSfXnTeGG986ns4di7IuS5IyZaCT1CbU1TfwvzOX8D+/W8LgniXc9fEzOPPYflmXJUmtgoFOUqu3fONOPnnvPF5avZV3nzyM/7j8OHqWFGddliS1GgY6Sa1WSom7Zq/iP3+9gE4dO/C/7z+Zd5w4NOuyJKnVMdBJapUqt9dw089f4XcLKzl3XH++8t6TGNyrJOuyJKlVMtBJanUem7+em37xCjv31HHzZZP50JmjbEciSQdhoJPUauzYU8cXfvUa985ZzXFDe/KNq6YwblCPrMuSpIQ1P2EAACAASURBVFbPQCepVZi7cjOfuvclKrbs4m+nHcsnLxxPp44dsi5LktoEA52kTO2ta+DWJxbxnbKlDOvThXv/6kxOG9U367IkqU0x0EnKzJLK7Xzy3nm8uqaaK0uH86/vmEwP25FI0mEz0ElqcSklfjJrJf/98AK6de7Idz9wKhcfPzjrsiSpzTLQSWpRG6pr+PTPXuLpxRuZNmEAX37viQzsYTsSSToSBjpJLebhV9bx2Qdeoaa2nv981/H8xekjibAdiSQdKQOdpLyrrqnl5gfn84sX13DS8F58/aopjBnQPeuyJKlgGOgk5dUflm3iH+57ifXVNdxwwTiuf8tYiotsRyJJR5OBTlJe7Kmr55bHFnHb08s4pm9X7v/rMzl5ZJ+sy5KkgmSgk3TUla9vbEeyYF017z99JJ+7dBJdO/nrRpLyxd+wko6ahobED55ZzpcfKadnl458/8OlXDBpUNZlSVLBM9BJOirWbt3Np3/2Es8u3cSFkwbxxStOoH/3zlmXJUntgoFO0hF7cN4a/vWXr1LXkPjSFSdwZekI25FIUgsy0El607btquVzD77Kr15ayykje/P1q6ZwTL9uWZclSe2OgU7Sm/LMko3ceN9LbNyxh0+/dTx/ff6xdLQdiSRlwkAn6bDU1Nbz5UfK+cEzyxkzoBu/+NBZnDi8d9ZlSVK7ZqCT1Gzz127jk/fMY3HlDj585jHc9PZJdOlUlHVZktTuGegkHVJ9Q+K2p5Zxy+Pl9OnaiR//5VTOHz8g67IkSTkGOkkHtXrzLm687yVmr9jMJScM5r/edQJ9unXKuixJUhMGOkkHlFLi5y+s4eaH5gPwtfedxHtOGWY7EklqhQx0kv7Mlp17+ewDr/DbV9czdVRfvnblSYzo2zXrsiRJb8BAJ+l1ysor+af7X2bLrr3c9PaJXHvuGIo6OConSa2ZgU4SALv31vP/fruAn8xayfhB3fnRR6cyeWjPrMuSJDVDXruARsTFEVEeEUsi4qYDbB8ZETMj4sWIeDkiLmmy7TO548oj4m25dSNy+78WEfMj4oYm+98cEWsiYl7u55L9v0/Sgb1csZVL/+dpfjJrJR87ZzQPXX+OYU6S2pC8jdBFRBHwLeAioAJ4PiIeSim91mS3zwH3pZS+ExGTgYeBUbnlq4HjgKHAjIgYD9QBN6aUXoiIHsDciHi8yTm/nlL6ar6uSSpE3ylbytceK2dAj87838dP5+yx/bMuSZJ0mPI5QjcVWJJSWpZS2gvcA1y+3z4J2DcM0AtYm1u+HLgnpbQnpbQcWAJMTSmtSym9AJBS2g4sAIbl8RqkgrZwfTVfemQhF04axCM3nGeYk6Q2Kp+BbhiwusnnCv48fN0MfCAiKmgcnftEc4+NiFHAycAfmqy+Pnfr9gcR0edARUXEdRExJyLmVFVVHdYFSYVm5sLGfwf+4/Lj6NW1OONqJElvVtYzaV8D/CilNBy4BLgzIg5ZU0R0B34OfDKlVJ1b/R3gWGAKsA742oGOTSndllIqTSmVDhhgp3u1bzPLK5k8pCeDepZkXYok6QjkM9CtAUY0+Tw8t66pjwH3AaSUZgElQP+DHRsRxTSGuf9LKf1i3w4ppQ0ppfqUUgNwO423fCW9geqaWuau3ML0if6HjSS1dfkMdM8D4yJidER0ovElh4f222cVcAFAREyiMdBV5fa7OiI6R8RoYBwwOxpb1H8fWJBSuqXpiSJiSJOP7wZezcM1SQXj94s3Ut+QmDZhYNalSJKOUN7eck0p1UXE9cCjQBHwg5TS/Ij4PDAnpfQQcCNwe0R8isYXJD6SUkrA/Ii4D3iNxjdb/y6lVB8R5wAfBF6JiHm5r/psSulh4MsRMSV3nhXAX+Xr2qRCUFZeSc+Sjpw8onfWpUiSjlA05qf2qbS0NM2ZMyfrMqQWl1Li9P9+gtNG9+Vb7z8l63IkSc0QEXNTSqUH2pb1SxGSMvDaumoqt+9hurdbJakgGOikdqisvLFdyfnjfSFCkgqBgU5qh8rKKzlhWC8G9OicdSmSpKPAQCe1M9t2NbYrmTbB0TlJKhQGOqmdeXpJFQ0J25VIUgEx0EntzMyFVfTuWswU25VIUsEw0EntSEND4slFVZw3bgBFHSLrciRJR4mBTmpH5q+tZuOOPT4/J0kFxkAntSMzyyuJgPNsVyJJBcVAJ7UjZeWVnDisF/27265EkgqJgU5qJ7bs3MuLq7f6dqskFSADndROPLW4ipTw+TlJKkAGOqmdKCuvom+3Tpw43HYlklRoDHRSO7CvXcn5421XIkmFyEAntQMvr9nG5p17vd0qSQXKQCe1A2X72pWMM9BJUiEy0EntwMzyKqaM6E2fbp2yLkWSlAcGOqnAbdqxh5crtjLddiWSVLAMdFKBs12JJBU+A51U4MrKq+jfvRPHD+2VdSmSpDwx0EkFrD7XruS88QPoYLsSSSpYBjqpgL1UsZWtu2p9fk6SCpyBTipgZQsr6RBw7rj+WZciScojA51UwMoWVXHKyD707mq7EkkqZAY6qUBVbd/DyxXbfLtVktoBA51UoJ5aVAXANJ+fk6SCZ6CTCtTM8koG9OjMcUN7Zl2KJCnPDHRSAaqrb+CpRVVMGz+ACNuVSFKhM9BJBWje6q1U19R5u1WS2gkDnVSAZpZXUtQhOMd2JZLULhjopAJUVl7FqSP70KtLcdalSJJagIFOKjCV1TXMX1vNtIm2K5Gk9sJAJxWYsn3tSsb7/JwktRcGOqnAlJVXMqhnZyYN6ZF1KZKkFmKgkwpIbX0DTy/eyPQJA21XIkntiIFOKiAvrNzC9po6p/uSpHbGQCcVkLJFVXTsEJw91nYlktSeGOikAjJzYSWlo/rQo8R2JZLUnhjopAKxflsNC9dvZ7qzQ0hSu2OgkwpEWXklgNN9SVI7ZKCTCkRZeRVDe5UwflD3rEuRJLUwA51UAPbWNfD7JRs533YlktQuGeikAjB35RZ27Kljuu1KJKldMtBJBaCsvJLiouAs25VIUrtkoJMKQFl5FVNH96V7545ZlyJJyoCBTmrj1mzdTfmG7Uwb79utktReGeikNm5fu5LpE31+TpLaKwOd1MaVlVcxrHcXjh1guxJJaq8MdFIbtqeunmeWbGT6xAG2K5GkdsxAJ7Vhc1ZsYdfeep+fk6R2zkAntWEzF1bSqagDZ43tl3UpkqQMGeikNqxsURWnj+lL1062K5Gk9sxAJ7VRqzfvYknlDqZN8HarJLV3BjqpjSpbVAXANKf7kqR2z0AntVFlCysZ2bcrY/p3y7oUSVLGDHRSG1RTW8+zSzcxfYLtSiRJBjqpTZq9fDO7a+t9fk6SBBjopDaprLyKzh07cMYY25VIkpoZ6CLiFxFxaUQYAKVWoKy8kjPG9KNLp6KsS5EktQLNDWjfBt4PLI6IL0bEhDzWJOkgVm7aybKNO5nu262SpJxmBbqU0oyU0l8ApwArgBkR8WxEfDQiivNZoKTXKyvf167E5+ckSY2afQs1IvoBHwE+DrwI3EpjwHs8L5VJOqCy8kpG9+/GKNuVSJJymjVfUEQ8AEwA7gQuSymty226NyLm5Ks4Sa+3r13JNVNHZl2KJKkVae4EkN9MKc080IaUUulRrEfSQcxatok9dQ1Mn+jtVknSnzT3luvkiOi970NE9ImIv81TTZLewJPlVZQUd+D00X2zLkWS1Io0N9Bdm1Lauu9DSmkLcG1+SpL0RmaWV3LWsf0pKbZdiSTpT5ob6IqiyfxCEVEEdDrUQRFxcUSUR8SSiLjpANtHRsTMiHgxIl6OiEuabPtM7rjyiHjboc4ZEaMj4g+59fdGxCHrk9qS5Rt3snLTLqbZrkSStJ/mBrpHaHwB4oKIuAC4O7fuDeVC37eAtwOTgWsiYvJ+u30OuC+ldDJwNY397sjtdzVwHHAx8O2IKDrEOb8EfD2lNBbYAnysmdcmtQkzF1YCMG28z89Jkl6vuYHun4GZwN/kfp4A/ukQx0wFlqSUlqWU9gL3AJfvt08CeuaWewFrc8uXA/eklPaklJYDS3LnO+A5c6OHbwHuzx3/Y+Bdzbw2qU0oW1TFmAHdGNmva9alSJJamWa95ZpSagC+k/tprmHA6iafK4DT99vnZuCxiPgE0A24sMmxz+137LDc8oHO2Q/YmlKqO8D+rxMR1wHXAYwcaesHtQ2799bz3LJNfPCMY7IuRZLUCjV3LtdxEXF/RLwWEcv2/RyF778G+FFKaThwCXBnvueLTSndllIqTSmVDhjgs0hqG2Yt28jeugafn5MkHVBzw9MPaRydqwOmAz8BfnqIY9YAI5p8Hp5b19THgPsAUkqzgBKg/0GOfaP1m4DeEdFxv/VSQZi5sIouxUVMtV2JJOkAmhvouqSUngAipbQypXQzcOkhjnkeGJd7+7QTjS85PLTfPquACwAiYhKNga4qt9/VEdE5IkYD44DZb3TOlFKi8Rm/9+bO+2HgwWZem9SqpZSYWV7J2WP70bmj7UokSX+uuYFuT+5W6OKIuD4i3g10P9gBuefZrgceBRbQ+Dbr/Ij4fES8M7fbjcC1EfESjW/OfiQ1mk/jyN1rNL5N+3cppfo3OmfuXP8M/ENELKHxmbrvN/PapFZtadVOKrbsZtoE326VJB1YNA5uHWKniNNoDFC9gS/Q+GbqV1JKzx30wFautLQ0zZnjVLRq3e54ehn/+ZsF/P6fpzO8j2+4SlJ7FRFz32jK1UO+5Zrr/XZVSunTwA7go0e5PkkHUVZexbiB3Q1zkqQ3dMhbrimleuCcFqhF0n527qlj9vLNTJ/o7VZJ0htrVh864MWIeAj4GbBz38qU0i/yUpUkAJ5duom99Q1MG2+7EknSG2tuoCuhsTXIW5qsS4CBTsqjsvJKunUqonSU7UokSW+suTNF+Nyc1MJSSpSVV3H22P506pjXftuSpDauWYEuIn5I44jc66SU/vKoVyQJgCWVO1izdTfXv2Vs1qVIklq55t5y/XWT5RLg3cDao1+OpH1mllcCON2XJOmQmnvL9edNP0fE3cDv81KRJKBxuq+Jg3swpFeXrEuRJLVyb/bBnHGAfRSkPNleU8uclZs539E5SVIzNPcZuu28/hm69TROtSUpD55Zsona+sR0p/uSJDVDc2+59sh3IZL+5MlFlfTo3JFTj+mTdSmSpDagWbdcI+LdEdGryefeEfGu/JUltV8pJWYurOKccf0pLrJdiSTp0Jr7t8W/p5S27fuQUtoK/Ht+SpLat/IN21lfXePbrZKkZmtuoDvQfs1teSLpMMxcWAXANJ+fkyQ1U3MD3ZyIuCUijs393ALMzWdhUntVVl7JpCE9GdSzJOtSJEltRHMD3SeAvcC9wD1ADfB3+SpKaq+qa2qZs3IL073dKkk6DM19y3UncFOea5HavWcWb6S+IXm7VZJ0WJr7luvjEdG7yec+EfFo/sqS2qeZ5ZX0KOnIKSN7H3pnSZJymnvLtX/uzVYAUkpbcKYI6ahKKVFWXsV54wbQ0XYlkqTD0Ny/NRoiYuS+DxExitfPHCHpCL22rprK7XtsVyJJOmzNbT3yL8DvI+JJIIBzgevyVpXUDpWVN7Yrcf5WSdLhau5LEY9ERCmNIe5F4JfA7nwWJrU3ZeWVHD+sJwN72K5EknR4mhXoIuLjwA3AcGAecAYwC3hL/kqT2o9tu2p5YdVW/nbasVmXIklqg5r7DN0NwGnAypTSdOBkYOvBD5HUXE8vqcq1K/F2qyTp8DU30NWklGoAIqJzSmkhMCF/ZUntS1l5Fb27FjNlRJ+sS5EktUHNfSmiIteH7pfA4xGxBViZv7Kk9qOhobFdybnjBlDUIbIuR5LUBjX3pYh35xZvjoiZQC/gkbxVJbUj89dWs3HHHqf7kiS9ac0dofujlNKT+ShEaq/KyisBOG+8gU6S9ObYjl7K2MzySk4a3ov+3TtnXYokqY0y0EkZ2rJzL/NWb+X8Cc6kJ0l68wx0UoaeWlxFQ8Ln5yRJR8RAJ2XoyfIq+nQt5sThvbMuRZLUhhnopIw0NCSeXFTF+eNtVyJJOjIGOikjr6zZxqade5nm83OSpCNkoJMyMrO8kgjblUiSjpyBTspIWXkVJw3vTd9unbIuRZLUxhnopAxs2rGHlyq2Mt3brZKko8BAJ2Xg6cUbSQmm2a5EknQUGOikDMwsr6Rft06cMKxX1qVIkgqAgU5qYfUNiacWVXH+hAF0sF2JJOkoMNBJLeyliq1s2VVruxJJ0lFjoJNaWFl5FR0CzhvXP+tSJEkFwkAntbCy8kpOHtmH3l1tVyJJOjoMdFILqtq+h5crtjHdt1slSUeRgU5qQU8tqgLw+TlJ0lFloJNa0MzySgb06MzkIT2zLkWSVEAMdFILqatv4OnFGzl/vO1KJElHl4FOaiHzVm9l2+5ap/uSJB11BjqphZSVV1HUITjHdiWSpKPMQCe1kJnllZw6sg+9uhRnXYokqcAY6KQWUFldw/y11ZxvuxJJUh4Y6KQWUJZrV+Lzc5KkfDDQSS3gyfIqBvXszKQhPbIuRZJUgAx0Up7V1Tfw1OIqpo0fSITtSiRJR5+BTsqzF1ZtZXtNHdN8fk6SlCcGOinPZpZX0rFDcLbtSiRJeWKgk/KsrLyKU4/pQ88S25VIkvLDQCfl0fptNSxYV830ib7dKknKHwOdlEdPLqoE8Pk5SVJeGeikPJq5sIohvUqYMMh2JZKk/DHQSXlSW9/A75dsZNoE25VIkvLLQCflyZwVW9ixx3YlkqT8M9BJeVJWXklxUXD2WNuVSJLyy0An5UlZeRWnjepL984dsy5FklTgDHRSHqzdupvyDduZPsF2JZKk/MtroIuIiyOiPCKWRMRNB9j+9YiYl/tZFBFbm2z7UkS8mvu5qsn6p5scszYifplbPy0itjXZ9m/5vDbpYMrKqwDblUiSWkbe7gVFRBHwLeAioAJ4PiIeSim9tm+flNKnmuz/CeDk3PKlwCnAFKAzUBYRv00pVaeUzm1yzM+BB5t87dMppXfk65qk5ppZXsmw3l0YO7B71qVIktqBfI7QTQWWpJSWpZT2AvcAlx9k/2uAu3PLk4GnUkp1KaWdwMvAxU13joiewFuAXx71yqUjsKeunmeXbGTahAG2K5EktYh8BrphwOomnyty6/5MRBwDjAZ+l1v1EnBxRHSNiP7AdGDEfoe9C3gipVTdZN2ZEfFSRPw2Io57g++6LiLmRMScqqqqw78q6RDmrNjCzr31Pj8nSWoxreX1u6uB+1NK9QAppcci4jTgWaAKmAXU73fMNcAdTT6/AByTUtoREZfQOHI3bv8vSindBtwGUFpamo72hUhl5ZV0KurAWWP7ZV2KJKmdyOcI3RpeP6o2PLfuQK7mT7dbAUgp/VdKaUpK6SIggEX7tuVG7aYCv2myf3VKaUdu+WGgOLef1KJmlldx+pi+dO3UWv57SZJU6PIZ6J4HxkXE6IjoRGNoe2j/nSJiItCHxlG4feuKIqJfbvlE4ETgsSaHvRf4dUqppskxgyP3wFJETKXx2jYd9auSDmL15l0sqdzB+eN9u1WS1HLyNoSQUqqLiOuBR4Ei4AcppfkR8XlgTkppX7i7GrgnpdT09mcx8HQun1UDH0gp1TXZfjXwxf2+8r3A30REHbAbuHq/c0p5V7ao8bnM6RN9fk6S1HKiPWee0tLSNGfOnKzLUAH5+I+fp3zDdp76x+m+4SpJOqoiYm5KqfRA25wpQjpKamrreWbJJqZPGGiYkyS1KAOddJQ8v2Izu2vrnR1CktTiDHTSUTJzYRWdOnbgzDG+XC1JalkGOukoKVtUyRlj+tGlU1HWpUiS2hkDnXQUrNq0i2VVO5nu7VZJUgYMdNJRULaoEsDpviRJmTDQSUfBzIWVjOrXlVH9u2VdiiSpHTLQSUeopraeZ5duYpqjc5KkjBjopCP03LJN7KlrsF2JJCkzBjrpCJWVV1FS3IEzxvTLuhRJUjtloJOOUFl5JWeO6UdJse1KJEnZMNBJR2D5xp2s2LSL6RN9fk6SlB0DnXQEysob25VMG2+gkyRlx0AnHYGZ5VWMGdCNkf26Zl2KJKkdM9BJb9LuvfU8t2yTo3OSpMwZ6KQ3adayjeyta2D6RNuVSJKyZaCT3qSy8iq6FBcxdXTfrEuRJLVzBjrpTUgpMbO8krPH9qNzR9uVSJKyZaCT3oRlG3eyevNuzne6L0lSK2Cgk96EmQv3tSvx+TlJUvYMdNKb8OSiKsYO7M6IvrYrkSRlz0AnHaade+r4w7LNTJ/g6JwkqXUw0EmHadbSTeytb2Caz89JkloJA510mGaWV9KtUxGlo/pkXYokSYCBTjosKSXKyqs4a2x/25VIkloNA510GJZU7mDN1t1M93arJKkVMdBJh2Fmea5diS9ESJJaEQOddBjKyquYMKgHQ3t3yboUSZL+yEAnNdOOPXU8v2Iz0yY6OidJal0MdFIzPbNkI7X1iWnjfX5OktS6GOikZiorr6R75462K5EktToGOqkZ9rUrOWdsf4qL/NdGktS6+DeT1AzlG7azblsN031+TpLUChnopGYoK68C4Hyfn5MktUIGOqkZZi6sZNKQngzuVZJ1KZIk/RkDnXQI1TW1zF25xWbCkqRWy0AnHcIzizdS15Cc7kuS1GoZ6KRDKCuvokdJR04Z2TvrUiRJOiADnXQQKSXKFlVy3rgBdLRdiSSplfJvKOkgFqzbzobqPZzv83OSpFbMQCcdxCPz1wMwbbyBTpLUehnopDdQW9/APbNXcf74AQzsabsSSVLrZaDLo8rtNXzpkYVs3LEn61L0Jjw6fz2V2/fwoTOPyboUSZIOykCXR9W7a/lO2VLumb0q61L0Jvxk1kqG9+nCNNuVSJJaOQNdHo0d2INzx/Xnp8+tora+IetydBgWrq9m9vLNfPCMYyjqEFmXI0nSQRno8uzDZ45ifXUNj+YerlfbcOeslXTu2IErS0dkXYokSYdkoMuz6RMHMrJvV3787IqsS1EzVdfU8sCLa7jspKH06dYp63IkSTokA12eFXUIPnTmMTy/YguvrtmWdTlqhp/PrWDX3npfhpAktRkGuhbwvtIRdCkucpSuDUgpcedzK5kyojcnDneqL0lS22CgawG9uhTznlOG8eBLa9m8c2/W5eggnlmyiWVVOx2dkyS1KQa6FvKRs0axt66Bu21h0qr9eNYK+nbrxCUnDMm6FEmSms1A10LGDerB2WP78dPnVlJnC5NWac3W3TyxYANXnzaCkuKirMuRJKnZDHQt6CNnjWbdthoee21D1qXoAP7vuZUA/MUZ3m6VJLUtBroW9JaJAxnepws/8uWIVqemtp57nl/NBZMGMaz3/2/vzoOjvO88j3++al1ICMQhEDobfGHAXAYhyZOsE9tjPHZsctgGzOWq1PyxyexM9ppkazYzldr8MVVbm93Zyc6Md3YCAmxsYzt2EnwlPmKb5j7MZTAGtQ4EiEMgJISu3/6hh4mMASOh7qef7verSkX3o6ef59PVAn14up/vM8zvOAAADAiFLo5CaablVWFtOXpG+4+d9zsO+tmwp0ln2jo5GQIAEEgUujh7ghEmCakmEtWkglzdc8tYv6MAADBgFLo4G5mToQWzivXLXY06ywiThPBxQ4t21bdoaWW50rhuKwAggCh0PlhRHdal7l6t21rvdxSo7+hcTmZI3767xO8oAAAMCoXOB3cU5qlqEiNMEsHZtk79avcxLZhVrBHZGX7HAQBgUCh0PllxT1iNLRf12wOMMPHTC9vqdam7l5MhAACBRqHzyf3eeAxGmPinp9dpzeaoKiaO1uTCEX7HAQBg0Ch0PgmlmZZVlWvTkTM60MQIEz+8d/Ck6s9c5OgcACDwKHQ+enJuqbIz0lQTqfU7SkqqiUQ1Li9LD04t9DsKAAA3hULno/ycTH1zVrFe2dmolnZGmMRT7ak2vX+oWYvnlSkjxF8DAECw8ZvMZ8urw+ro6tXzjDCJqzWbokpPMy2uKPM7CgAAN41C57PJhSNUOWm0aiJR9fQ6v+OkhIudPXphW70enFaocSOy/Y4DAMBNi2mhM7P5ZnbQzA6b2Q+v8v2fmdku7+uQmbX0+97fmtle7+vJfstXmtnRfo+b6S03M/s7b18fm9nsWD63obSimhEm8fTqrkad7+jW8qqw31EAABgS6bHasJmFJP1c0gOSGiRtNbPXnHP7L6/jnPtBv/X/TNIs7/bDkmZLmikpS9J7Zva6c+7y6aD/yTm3/opdPiTpNu9rnqR/8P5MePffOV5FI7O18qNaPqAfY8451USimlyYp7nhUX7HAQBgSMTyCF2FpMPOuSPOuU5J6yQ9dp31F0l6zrs9RdLvnXPdzrk2SR9Lmv8l+3tMUo3rs0lSvplNuLmnEB/poTQtrQorcuS0Dh5v9TtOUttRd1b7m85raVW5zLhuKwAgOcSy0BVL6v9J/wZv2ReYWbmkiZLe8RbtljTfzHLMbKykr0kq7feQn3pvq/7MzLIGsj8z+1Mz22Zm25qbmwfzvGJi4dxSZaWnMWg4xlZtjCovK10LZl71RxEAgEBKlJMiFkpa75zrkSTn3FuSNkjaqL6jdhFJPd66P5I0WdJcSaMl/eVAduSce8Y5N8c5N6egoGCI4t+8UbmZWjCzWL/c2ahz7V1+x0lKza2X9PreJn1nTolys2L2aQMAAOIuloWuUZ8/qlbiLbuahfrD262SJOfcT51zM51zD0gySYe85U3e26qXJP1CfW/tDnR/CWl5dVgXu/rOwMTQW7elTl09TksruTIEACC5xLLQbZV0m5lNNLNM9ZW2165cycwmSxqlvqNwl5eFzGyMd3u6pOmS3vLuT/D+NEkLJO31HvaapGXe2a6Vks4555pi9eRiYUrRCFVMHK1VkVpGmAyx7p5erd1cp6/cNlaTCob7HQcAgCEVs0LnnOuW9H1Jb0o6IOkFF1mXbAAAFblJREFU59w+M/uJmT3ab9WFktY55/o3mAxJH5jZfknPSFribU+S1prZHkl7JI2V9N+85RskHZF0WNL/lfRvY/TUYmpFdVgNZy/qnU9O+h0lqby9/4SOn+/QMkaVAACSUEw/SOSc26C+otV/2Y+vuP83V3lch/rOdL3aNr9+jeVO0vcGmzVR/PGU8ZowMlsrNx7VA1PG+x0nadREoirOH6avTx7ndxQAAIZcopwUAU96KE1LKsv10eHT+vQEI0yGwqcnWhU5clpPVZYplMaoEgBA8qHQJaBFFWXKZITJkFm9KarMUJqenFP65SsDABBAFLoENDo3U4/NKNLLOxp17iIjTG5Ga0eXXtreoEdmTNCY4Vlf/gAAAAKIQpegLo8weZERJjfllZ2Nauvs4WQIAEBSo9AlqGnFIzU3PEo1kSgjTAbp8nVbp5eM1MzSfL/jAAAQMxS6BLa8Oqy6M+16lxEmgxI5clqHT15gkDAAIOlR6BLYg1MLVTgiW6sitX5HCaSajVGNysnQN2YU+R0FAICYotAlsIxQmpZUlumDT0/p8ElGmAxE07mLevvACT0xt1TZGSG/4wAAEFMUugR3eYTJqo1Rv6MEyrOb69TrnJbM4+1WAEDyo9AluDHDs/SN6UV6aUeDzncwwuRGXOru0XNb6vT1O8apdHSO33EAAIg5Cl0ArKgOq72zRy9ua/A7SiC8sfe4Tl3o1LLqsN9RAACICwpdANxVMlJ3l49STaRWvYww+VI1kajCY3L0lVvH+h0FAIC4oNAFxIrqsKKn2/XeIUaYXM/exnPaHj2rJZXlSuO6rQCAFEGhC4j50wo1fkSWVnJyxHWtjkQ1LCOkx+/muq0AgNRBoQuIjFCanppXrt8fatZnzRf8jpOQzrV36dXdjVowq0gjczL8jgMAQNxQ6AJkUUWZMkNpqtlY63eUhPTi9np1dPVqaWXY7ygAAMQVhS5ACvKy9Mj0CVq/vUGtjDD5nN5ep9WboppTPkpTikb4HQcAgLii0AXM8uqw2jp7tH47I0z6e//TZkVPtzOqBACQkih0ATOjNF+zyvJVE4kywqSf1ZGoxg7P0vyphX5HAQAg7ih0AbSiOqyjp9r0/qfNfkdJCPVn2vXuwZNaXFGqzHR+pAEAqYfffgH00LQJKsjL0ipOjpAkrdkUVZqZFnPdVgBAiqLQBVBmepqWzCvXewebdSTFR5h0dPXo+W31enDqeBWOzPY7DgAAvqDQBdSieaXKCJlqIqk9aPi13cfU0t7FqBIAQEqj0AXUuLxsPTK9SOu3N+jCpW6/4/jCOafVkahuHz9clZNG+x0HAADfUOgCbHl1WBcudeulFB1hsqu+RXsaz2lpVVhmXLcVAJC6KHQBNrM0XzNK87VqY21KjjCpiUQ1PCtd35xV7HcUAAB8RaELuKerwzpyqk0fHD7ld5S4OnXhkn7zcZO+PbtYw7PS/Y4DAICvKHQB9yd3TdDY4Vla+dFRv6PE1fNb69XZ06ulVYwqAQCAQhdwmelpempemd492Kyjp9r8jhMX3T29enZzne65dYxuHZfndxwAAHxHoUsCT80rU3qaqSZS63eUuPjdJyfV2HKRUSUAAHgodElg3IhsPTx9gtZva1BbCowwWR2Jqmhktu6/c5zfUQAASAgUuiSxvDqs1kvdenlHco8w+az5gj48fEqL55UpPcSPLwAAEoUuacwqzdeMkpFaubFWziXvCJPVkagyQ2laWFHmdxQAABIGhS5JmJmWV4f1WXObPkzSESZt3hDlP7mrUGOHZ/kdBwCAhEGhSyIPT5+gscMztfKjWr+jxMQrOxvVeqlbS6vCfkcBACChUOiSSFZ6SIsryvTOwZOKnk6uESaXr9s6rXiEZpfl+x0HAICEQqFLMk9VlitkpppI1O8oQ2rL0TM6eKJVyyq5bisAAFei0CWZ8SOy9dBdE/TCtvqkGmFSE4lq5LAMfWNGkd9RAABIOBS6JLSiulytHd16eWej31GGxInzHXpz33E9MadEwzJDfscBACDhUOiS0OyyUbqreKRWJckIk2c316nHOS2p5LqtAABcDYUuCV0eYXL45AV9dPi033FuSmd3r57dUqd7by9Q+Zhcv+MAAJCQKHRJ6pHpEzQmN1MrN9b6HeWmvLnvuJpbL2kZo0oAALgmCl2Sys4IaVFFmX73yQnVn2n3O86grY5EVTY6R//m9gK/owAAkLAodElsSWW50sxUE6n1O8qgHGg6ry21Z7S0slxpaYwqAQDgWih0SaxwZLbmTyvU81vr1d4ZvBEmNZGostLT9PicEr+jAACQ0Ch0Se7p6rDOd3TrlYCNMDl3sUu/3Nmox2YWKT8n0+84AAAkNApdkru7fJSmFo0I3AiTl7Y36GJXDydDAABwAyh0Se7yCJNDJy4o8lkwRpj09jqt2RTV7LJ8TSse6XccAAASHoUuBTw6o0ijAzTC5MPDp3TkVBtH5wAAuEEUuhSQnRHSwrml+u2BYIwwqYlENSY3Uw/dVeh3FAAAAoFClyKWVJbLzLRmU9TvKNfVcLZd73xyQosqypSVznVbAQC4ERS6FFGUP0wPTh2vdVvrdbGzx+8417R2c50kafG8Mp+TAAAQHBS6FLKiemLfOJBdiTnCpKOrR+u21OmBKeNVlD/M7zgAAAQGhS6FzA2P0p0TRmjlR4k5wuQ3HzfpbHsXJ0MAADBAFLoUYmZ6ujqsgydatenIGb/jfEHNpqhuKchV9S1j/I4CAECgUOhSzKMzizQqJ0OrEmyEye76Fu2ub9GyqrDMuG4rAAADQaFLMdkZIS2sKNNb+4+r4WzijDCpiUSVmxnSt2YX+x0FAIDAodCloCWV5ZKkNZvqfE7S50xbp3718TF9c3ax8rIz/I4DAEDgUOhSUHH+MP3xlEKt21qnji7/R5i8sK1end29nAwBAMAgUehS1Ip7wmpp79KrPo8w6fGu21o5abRuH5/naxYAAIKKQpei5k0crcmFefqFzyNM3v3kpBrOXuToHAAAN4FCl6LMTCuqw/rkeKu2HPVvhEnNpqgKR2TrgSnjfcsAAEDQUehS2GMzizVyWIZW+jTC5OipNv3+ULMWzytTRogfRQAABovfoilsWGZICytK9db+E2psuRj3/a/ZFFVGyLSwojTu+wYAIJlQ6FLc0spyOdd3YkI8tXd264Vt9Zo/bYLG5WXHdd8AACSbmBY6M5tvZgfN7LCZ/fAq3/+Zme3yvg6ZWUu/7/2tme31vp7st3ytt829ZvYvZpbhLb/XzM71296PY/nckkXJqBw9MGW81m2J7wiTV3cdU2tHt5ZXlcdtnwAAJKuYFTozC0n6uaSHJE2RtMjMpvRfxzn3A+fcTOfcTEn/W9LL3mMfljRb0kxJ8yT9RzMb4T1sraTJku6SNEzSd/tt8oPL23PO/SRWzy3ZLK8O62x7l17bfSwu+3POqSYS1Z0TRuju8lFx2ScAAMkslkfoKiQdds4dcc51Slon6bHrrL9I0nPe7SmSfu+c63bOtUn6WNJ8SXLObXAeSVsklcTsGaSIqkljdMf4PK2M0wiT7dGzOtB0XsuqyrluKwAAQyCWha5YUn2/+w3esi8ws3JJEyW94y3aLWm+meWY2VhJX5NUesVjMiQtlfRGv8VVZrbbzF43s6nX2Nefmtk2M9vW3Nw8mOeVdMxMy6vD2t90XtuiZ2O+v1WRqPKy0/XYzKKY7wsAgFSQKCdFLJS03jnXI0nOubckbZC0UX1H7SKSrvyA1/9R31G8D7z7OySVO+dmqO/t219ebUfOuWecc3Occ3MKCgqG/pkE1IJZRX0jTD6qjel+TrZ26I29TXpiTqlyMtNjui8AAFJFLAtdoz5/VK3EW3Y1C/WHt1slSc65n3qfhXtAkkk6dPl7ZvbXkgok/ft+6593zl3wbm+QlOEd3cMNyMlM15NzS/XGvuNqOhe7ESbrttSrq8dpSSUnQwAAMFRiWei2SrrNzCaaWab6SttrV65kZpMljVLfUbjLy0JmNsa7PV3SdElvefe/K+lBSYucc739HlNo3geyzKxCfc/tdIyeW1JaWlmu3hiOMOnq6dXazVF99fYCTRybG5N9AACQimJW6Jxz3ZK+L+lNSQckveCc22dmPzGzR/utulDSOvf5T+NnSPrAzPZLekbSEm97kvSPksZLilwxnuQ7kvaa2W5JfydpoYvHJ/yTSOnoHN1/53g9t6U+JiNM3t5/QifOX2JUCQAAQyymH2Ly3vrccMWyH19x/2+u8rgO9Z3perVtXjWzc+7vJf39YLOiz4rqsN7ef0K/2n1Mj88Z2is41ERqVTJqmO69Y9yQbhcAgFSXKCdFIEFU3zJGt40brpUbh3aEyaETrdp05IyWVJYrlMaoEgAAhhKFDp9zeYTJvmPntX0IR5jURGqVmZ6mJ4b4qB8AAKDQ4Sq+NbtYednpWrmxdki219rRpVd2NOrRGUUanZs5JNsEAAB/QKHDF+RkpuvJOaV6fe9xHT/XcdPbe3lHo9o6e7SMkyEAAIgJCh2uallVWL3Oae3mmxth0nfd1lrNKM3X9JL8oQkHAAA+h0KHqyobk6P7Jo/Ts5vrdKl78CNMNn52Wp81t2kZg4QBAIgZCh2uaUX1RJ1u69SvdzcNehs1kVqNzs3Uw9MnDF0wAADwORQ6XNM9t47RrTcxwuRYy0W9vf+EnpxbquyMUAwSAgAAiUKH6zAzLa8q157Gc9pR1zLgxz+7uU6S9NS8sqGOBgAA+qHQ4bq+NbtEeVnpWjXAESaXunv03JY63XfneJWMyolNOAAAIIlChy+Rm5Wux+eUasOeJp04f+MjTF7fc1yn2zoZVQIAQBxQ6PClllWVq8c5rfXeQr0RNZFaTRqbq3tuGRu7YAAAQBKFDjcgPDZXX7tjnJ7dHL2hESZ7vc/cLaksVxrXbQUAIOYodLghK6rDOnWhUxv2fPkIk5pIrXIyQ/r23SWxDwYAACh0uDF/dOtYTSrI1cqPaq+7Xkt7p17ddUwLZhVr5LCM+IQDACDFUehwQ9LSTCuqw9rdcE47685ec70XtzXoUncvJ0MAABBHFDrcsG/NLtHw64ww6e11Wr0pqorwaE0uHBHfcAAApDAKHW7Y8Kx0PT6nRL/Z06STrV8cYfL+oWbVnWnXsmqOzgEAEE8UOgzIsqqwunrcv14For+aSK3G5WXpwamF8Q8GAEAKo9BhQCaOzdW9dxRo7eY6dXb3/uvyutPteu9QsxZVlCkjxI8VAADxxG9eDNiK6rCaWy/p9b1/GGGyZnNUITMt5rqtAADEHYUOA/bV2wo0cWyufuGNMLnY2aPnt9brwWmFGj8i299wAACkIAodBiwtzbS8qly76lu0q75Fv9p9TOcudmlZJSdDAADgBwodBuXbd5coNzOkVRtrVbOpVneMz1PFxNF+xwIAICVR6DAoedkZenxOqV7d1ai9jee1rLpcZly3FQAAP1DoMGjLqsrV66S8rHQtmFnsdxwAAFJWut8BEFyTCobru380UUX5w5SbxY8SAAB+4bcwbspfPTLF7wgAAKQ83nIFAAAIOAodAABAwFHoAAAAAo5CBwAAEHAUOgAAgICj0AEAAAQchQ4AACDgKHQAAAABR6EDAAAIOAodAABAwFHoAAAAAo5CBwAAEHAUOgAAgICj0AEAAAQchQ4AACDgKHQAAAABR6EDAAAIOAodAABAwFHoAAAAAo5CBwAAEHAUOgAAgICj0AEAAAQchQ4AACDgzDnndwbfmFmzpGgcdjVW0qk47AexwesXfLyGwcdrGGy8fkOj3DlXcLVvpHShixcz2+acm+N3DgwOr1/w8RoGH69hsPH6xR5vuQIAAAQchQ4AACDgKHTx8YzfAXBTeP2Cj9cw+HgNg43XL8b4DB0AAEDAcYQOAAAg4Ch0AAAAAUehiyEzm29mB83ssJn90O88GBgzKzWzd81sv5ntM7M/9zsTBs7MQma208x+7XcWDJyZ5ZvZejP7xMwOmFmV35kwMGb2A+/f0L1m9pyZZfudKRlR6GLEzEKSfi7pIUlTJC0ysyn+psIAdUv6D865KZIqJX2P1zCQ/lzSAb9DYND+l6Q3nHOTJc0Qr2WgmFmxpH8naY5zbpqkkKSF/qZKThS62KmQdNg5d8Q51ylpnaTHfM6EAXDONTnndni3W9X3i6TY31QYCDMrkfSwpH/2OwsGzsxGSvqqpP8nSc65Tudci7+pMAjpkoaZWbqkHEnHfM6TlCh0sVMsqb7f/QZRBgLLzMKSZkna7G8SDND/lPSfJfX6HQSDMlFSs6RfeG+b/7OZ5fodCjfOOdco6b9LqpPUJOmcc+4tf1MlJwod8CXMbLiklyT9hXPuvN95cGPM7BFJJ51z2/3OgkFLlzRb0j8452ZJapPE55EDxMxGqe/dqYmSiiTlmtkSf1MlJwpd7DRKKu13v8RbhgAxswz1lbm1zrmX/c6DAblH0qNmVqu+jzx83czW+BsJA9QgqcE5d/nI+Hr1FTwEx/2Sjjrnmp1zXZJellTtc6akRKGLna2SbjOziWaWqb4Pgb7mcyYMgJmZ+j67c8A59z/8zoOBcc79yDlX4pwLq+/v3zvOOY4MBIhz7rikejO7w1t0n6T9PkbCwNVJqjSzHO/f1PvEiS0xke53gGTlnOs2s+9LelN9Z/X8i3Nun8+xMDD3SFoqaY+Z7fKW/Rfn3AYfMwGp5s8krfX+Y3xE0tM+58EAOOc2m9l6STvUNzlgp7gMWExw6S8AAICA4y1XAACAgKPQAQAABByFDgAAIOAodAAAAAFHoQMAAAg4Ch0A+MDM7jWzX/udA0ByoNABAAAEHIUOAK7DzJaY2RYz22Vm/2RmITO7YGY/M7N9ZvY7Myvw1p1pZpvM7GMze8W7jqXM7FYz+62Z7TazHWZ2i7f54Wa23sw+MbO13iR9ABgwCh0AXIOZ3SnpSUn3OOdmSuqR9JSkXEnbnHNTJb0v6a+9h9RI+kvn3HRJe/otXyvp5865Geq7jmWTt3yWpL+QNEXSJPVdnQQABoxLfwHAtd0n6W5JW72DZ8MknZTUK+l5b501kl42s5GS8p1z73vLV0l60czyJBU7516RJOdchyR529vinGvw7u+SFJb0YeyfFoBkQ6EDgGszSauccz/63EKz/3rFeoO9huKlfrd7xL/JAAaJt1wB4Np+J+k7ZjZOksxstJmVq+/fzu946yyW9KFz7pyks2b2FW/5UknvO+daJTWY2QJvG1lmlhPXZwEg6fG/QQC4BufcfjP7K0lvmVmapC5J35PUJqnC+95J9X3OTpKWS/pHr7AdkfS0t3yppH8ys59423g8jk8DQAow5wb7TgEApCYzu+CcG+53DgC4jLdcAQAAAo4jdAAAAAHHEToAAICAo9ABAAAEHIUOAAAg4Ch0AAAAAUehAwAACLj/D9LFB45LZ7nKAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(10, 10))\n",
        "plt.plot(history.history['loss'])\n",
        "plt.title('model loss')\n",
        "plt.ylabel('loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train'], loc='upper left')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "9dIpZk6AK8Zl",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 621
        },
        "outputId": "e51ed0c8-63d1-470c-fc1d-e79b341491ca"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 720x720 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAJcCAYAAACxEXM4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeVhW94H+//vzsIoCCiqKKOC+xRUXkKhJzN7s++LeODZpmzSZbtPp9DvT30w77WRtkqZJNGoWszRr06wmUaO4odG4KyogLogoIMrO5/cHJDWNGlAePs/yfl0XV4nnAW5y5UrvnHOe+xhrrQAAANC6PK4DAAAABCNKGAAAgAOUMAAAAAcoYQAAAA5QwgAAAByghAEAADhACQMQFIwx84wx/18TX5trjJl0rt8HAM6EEgYAAOAAJQwAAMABShgAn9F4GfCnxpgvjTHHjTFzjDEJxpj3jTHHjDGLjDEdTnr91caYzcaYEmPMYmPMgJOODTfGrGv8ulckRf7Tz/qeMWZ949dmGWOGnGXmu4wxOcaYI8aYd4wxiY1/bowxDxtjDhljyowxG40xgxuPXWGM2dKYbZ8x5l/P6m8YAL9GCQPga26QdLGkvpKukvS+pH+T1EkN/876sSQZY/pKWijpvsZj70n6mzEm3BgTLuktSc9LipP0WuP3VePXDpc0V9K/SIqX9BdJ7xhjIpoT1BhzoaTfSbpZUldJeZJebjx8iaTxjb9HbONrihuPzZH0L9baaEmDJX3anJ8LIDBQwgD4mj9ZawuttfskfS5plbX2C2ttpaQ3JQ1vfN0tkv5urf3YWlsj6f8ktZGUIWmspDBJj1hra6y1f5W05qSfMUvSX6y1q6y1ddba+ZKqGr+uOe6QNNdau85aWyXpl5LSjTEpkmokRUvqL8lYa7daaw80fl2NpIHGmBhr7VFr7bpm/lwAAYASBsDXFJ70ecUp/rpd4+eJajjzJEmy1tZL2iupW+OxfdZae9LX5p30ebKkBxovRZYYY0okdW/8uub45wzlajjb1c1a+6mkxyU9IemQMeZpY0xM40tvkHSFpDxjzBJjTHozfy6AAEAJA+Cv9quhTElquAdLDUVqn6QDkro1/tlXepz0+V5J/22tbX/SR5S1duE5Zmirhsub+yTJWvuYtXakpIFquCz508Y/X2OtvUZSZzVcNn21mT8XQACghAHwV69KutIYc5ExJkzSA2q4pJglaYWkWkk/NsaEGWOulzT6pK99RtJsY8yYxhvo2xpjrjTGRDczw0JJ040xwxrvJ/sfNVw+zTXGjGr8/mGSjkuqlFTfeM/aHcaY2MbLqGWS6s/h7wMAP0UJA+CXrLXbJd0p6U+SDqvhJv6rrLXV1tpqSddLmibpiBruH3vjpK/NlnSXGi4XHpWU0/ja5mZYJOnXkl5Xw9m3XpJubTwco4ayd1QNlyyLJf2x8dhkSbnGmDJJs9VwbxmAIGO+ecsEAAAAWgNnwgAAAByghAEAADhACQMAAHCAEgYAAOBAqOsAzdWxY0ebkpLiOgYAAMB3Wrt27WFrbadTHfO7EpaSkqLs7GzXMQAAAL6TMSbvdMe4HAkAAOAAJQwAAMABShgAAIADfndP2KnU1NSooKBAlZWVrqN4XWRkpJKSkhQWFuY6CgAAOAcBUcIKCgoUHR2tlJQUGWNcx/Eaa62Ki4tVUFCg1NRU13EAAMA58NrlSGPMXGPMIWPMptMc72CMedMY86UxZrUxZvDZ/qzKykrFx8cHdAGTJGOM4uPjg+KMHwAAgc6b94TNk3TZGY7/m6T11tohkqZIevRcfligF7CvBMvvCQBAoPNaCbPWLpV05AwvGSjp08bXbpOUYoxJ8FYeAAAAX+Ly3ZEbJF0vScaY0ZKSJSWd6oXGmFnGmGxjTHZRUVErRmyakpISPfnkk83+uiuuuEIlJSVeSAQAAHydyxL2e0ntjTHrJf1I0heS6k71Qmvt09baNGttWqdOp1z+d+p0Jay2tvaMX/fee++pffv23ooFAAB8mLN3R1pryyRNlyTTcKPTHkm7XeU5F7/4xS+0a9cuDRs2TGFhYYqMjFSHDh20bds27dixQ9dee6327t2ryspK3XvvvZo1a5akfzyCqby8XJdffrkyMzOVlZWlbt266e2331abNm0c/2YAAMBbnJUwY0x7SSestdWSvi9paWMxOyf/+bfN2rL/nL/NNwxMjNFvrhp02uO///3vtWnTJq1fv16LFy/WlVdeqU2bNn09IzF37lzFxcWpoqJCo0aN0g033KD4+PhvfI+dO3dq4cKFeuaZZ3TzzTfr9ddf15133tmivwcAAPAdXithxpiFkiZK6miMKZD0G0lhkmStfUrSAEnzjTFW0mZJM72VpbWNHj36Gztejz32mN58801J0t69e7Vz585vlbDU1FQNGzZMkjRy5Ejl5ua2Wl4AAND6vFbCrLW3fcfxFZL6tvTPPdMZq9bStm3brz9fvHixFi1apBUrVigqKkoTJ0485c5XRETE15+HhISooqKiVbICAAA3eHZkC4iOjtaxY8dOeay0tFQdOnRQVFSUtm3bppUrV7ZyOgAA4IsC4rFFrsXHx2vcuHEaPHiw2rRpo4SEf8ydXXbZZXrqqac0YMAA9evXT2PHjnWYFAAA+ApjrXWdoVnS0tJsdnb2N/5s69atGjBggKNErS/Yfl8AAPyVMWattTbtVMe4HAkAAOAAJQwAAMCBgClh/nZZ9WwFy+8JAECgC4gSFhkZqeLi4oAvKNZaFRcXKzIy0nUUAABwjgLi3ZFJSUkqKChQSz3c21rJmBb5Vi0uMjJSSUmnfM45AADwIwFRwsLCwr6xUH8uNu0r1ewX1urJO0ZoSBIP1wYAAN4REJcjW1JyfJSOHq/WvKxc11EAAEAAo4T9k+jIMN04Mknvbjigw+VVruMAAIAARQk7hSkZKaquq9fCVfmuowAAgABFCTuFXp3aaXzfTnp+ZZ5q6updxwEAAAGIEnYa0zNSdOhYld7fdNB1FAAAEIAoYacxoW8npcRHaT436AMAAC+ghJ2Gx2M0JT1Fa/OO6suCEtdxAABAgKGEncGNaUlqGx7CXAUAAGhxlLAziIkM0w3MVQAAAC+ghH2HKenMVQAAgJZHCfsOvTu30/l9OuqFVcxVAACAlkMJa4Lp41JUWFalD5irAAAALYQS1gQT+3ZWcnwUN+gDAIAWQwlrgpPnKjYWlLqOAwAAAgAlrIluSktSFHMVAACghVDCmigmMkw3jEjS3zbsZ64CAACcM0pYM0zNSFZ1Xb1eXs1cBQAAODeUsGbo3Tla5/fpqOdXMlcBAADODSWsmaZlNMxVfLiZuQoAAHD2KGHNdEG/xrmK5bmuowAAAD9GCWsmj8do8thkZecd1aZ9zFUAAICzQwk7CzeldWeuAgAAnBNK2FmIbdMwV/HOhv0qZq4CAACcBUrYWZqakazq2nq9vGav6ygAAMAPUcLO0tdzFSuYqwAAAM1HCTsH0zJSdLCskrkKAADQbJSwczCxX2f1iIvSfG7QBwAAzUQJOwchHqMp6clak8tcBQAAaB5K2Dm6Ka272oSFcDYMAAA0CyXsHMW2CdMNI7vpbeYqAABAM1DCWsDU9BTmKgAAQLNQwlpAn4RoZfbuqBdWMlcBAACahhLWQqZlpOhAaaU+2lzoOgoAAPADlLAWckH/zuoe10bzsva4jgIAAPwAJayFhHiMpqanMFcBAACahBLWgpirAAAATUUJa0GxbcJ0/YiGuYojx6tdxwEAAD6MEtbCpmU0zFUsXJ3vOgoAAPBhlLAWdvJcRS1zFQAA4DQoYV4w9au5ii3MVQAAgFOjhHnBhV/NVSzPdR0FAAD4KEqYF4R4jKaMTdHq3CPavJ+5CgAA8G2UMC+5mbkKAABwBpQwL4mNapireGs9cxUAAODbKGFeNLVxruLlNcxVAACAb6KEeVHfhGiN6x2v51cwVwEAAL6JEuZlU9OZqwAAAN9GCfOyiwYkKKlDG83jBn0AAHASSpiXhXiMpqanaPWeI9qyv8x1HAAA4CMoYa2AuQoAAPDPKGGtIDYqTNeN6Ka31u9jrgIAAEiihLWaaRkpqmKuAgAANKKEtZK+CdHK6BWvF5irAAAAooS1qmkZKdpfWqmPmasAACDoUcJaEXMVAADgK5SwVhTiMZqSnqxVzFUAABD0KGGt7Ja0HsxVAAAASlhri40K07XDG+YqjjJXAQBA0KKEOfCPuYq9rqMAAABHKGEO9OsSrfSe8Xp+RS5zFQAABClKmCPTxjXMVSzaylwFAADBiBLmyKQBCerWvo2eW57rOgoAAHCAEubIyXMVWw8wVwEAQLChhDl0y6juigzzMFcBAEAQooQ51D4qXNcN76Y3v2CuAgCAYEMJc2xq41zFK9nMVQAAEEwoYY717xLTOFeRx1wFAABBhBLmA6ZmpGhfSYUWbT3kOgoAAGgllDAfMGlAZ3Vr30bzsva4jgIAAFoJJcwHhIZ4NCU9WSt3M1cBAECwoIT5iK/mKhasyHUdBQAAtAJKmI9grgIAgOBCCfMhUzNSVFnDXAUAAMGAEuZD+neJ0dieccxVAAAQBChhPmZaRipzFQAABAFKmI/5aq6C50kCABDYKGE+JjTEo8npyVqxu1jbDjJXAQBAoKKE+aBb0rorItTD2TAAAAIYJcwHdWj7j7mKkhPMVQAAEIgoYT7q67mKNcxVAAAQiChhPmpA1xiNSY3TghV5qqu3ruMAAIAWRgnzYdPHpTTOVRS6jgIAAFoYJcyHTRqQoG7t22je8lzXUQAAQAujhPmw0BCP7hzbMFex/eAx13EAAEALooT5uFtHNcxVzGOuAgCAgEIJ83Ed2obr2mHd9OYXBcxVAAAQQChhfuCruYpXs5mrAAAgUFDC/MDAROYqAAAINJQwPzEtI0UFRyv0CXMVAAAEBEqYn7h4YIISYyO5QR8AgABBCfMToSEe3ZmerKxdzFUAABAIKGF+5NZRPRQR6tH8FbmuowAAgHNECfMjcV/NVazbp9ITNa7jAACAc0AJ8zNTM1JUUVPHXAUAAH6OEuZnBibGaHRqnOavyGWuAgAAP0YJ80PTmasAAMDvUcL80FdzFdygDwCA//JaCTPGzDXGHDLGbDrN8VhjzN+MMRuMMZuNMdO9lSXQfDVXsTynWDsKmasAAMAfefNM2DxJl53h+D2Stlhrh0qaKOlBY0y4F/MElFtH9VB4qIfxVgAA/JTXSpi1dqmkI2d6iaRoY4yR1K7xtbXeyhNoGuYqEpmrAADAT7m8J+xxSQMk7Ze0UdK91tr6U73QGDPLGJNtjMkuKipqzYw+jbkKAAD8l8sSdqmk9ZISJQ2T9LgxJuZUL7TWPm2tTbPWpnXq1Kk1M/q0QYmxGp0SpwUrmasAAMDfuCxh0yW9YRvkSNojqb/DPH5p2rgU7T1SoU+3HXIdBQAANIPLEpYv6SJJMsYkSOonabfDPH7pkoEJ6hobqXlZe1xHAQAAzeDNiYqFklZI6meMKTDGzDTGzDbGzG58yW8lZRhjNkr6RNLPrbWHvZUnUIWGeHTn2Ia5ip3MVQAA4DdCvfWNrbW3fcfx/ZIu8dbPDya3je6hRz/ZqXlZufrv685zHQcAADQBi/kBIK5tuK4Zmqg31u1TaQVzFQAA+ANKWID4aq7iNeYqAADwC5SwADG4W8NcxfwVzFUAAOAPKGEBZGpGw1zFZ8xVAADg8yhhAeSSQV/NVeS6jgIAAL4DJSyAhDXOVSzLOcxcBQAAPo4SFmBuHdVd4aEezV+R6zoKAAA4A0pYgIlvF6Frhibq9bXMVQAA4MsoYQGIuQoAAHwfJSwADe4Wq1EpHbRgRR5zFQAA+ChKWICampGi/CMnmKsAAMBHUcIC1KWDuqhLTCQ36AMA4KMoYQEqLMSjyenJ+nznYeUcYq4CAABfQwkLYF/PVWTluY4CAAD+CSUsgMW3i9DVQxP1+roC5ioAAPAxlLAANy0jRSeqmasAAMDXUMIC3OBusUpLZq4CAABfQwkLAtPGNcxVLN7OXAUAAL6CEhYEvpqrmJeV6zoKAABoRAkLAmEhHt05tgdzFQAA+BBKWJC4bXQP5ioAAPAhlLAgEd8uQlcNaZirKKtkrgIAANcoYUHkH3MVBa6jAAAQ9ChhQeS8pFiNTO6gBStyVc9cBQAATlHCgsy0jBTlFZ/Q4h3MVQAA4BIlLMhcNrhhruIPH2zX8apa13EAAAhalLAgExbi0R9uHKIdhcd0/6vruSwJAIAjlLAgNL5vJ/3qyoH6cHOhHlm0w3UcAACCUqjrAHBjxrgUbT9Ypsc+zVHfLtH63pBE15EAAAgqnAkLUsYY/fbawUpL7qB/fW2DNhaUuo4EAEBQoYQFsYjQED01eaTi20borgXZOlRW6ToSAABBgxIW5Dq2i9DTU0aqtKJGs55fq8qaOteRAAAICpQwaFBirB6+ZajW7y3Rv72xUdbyjkkAALyNEgZJ0mWDu+onk/rqjS/26emlu13HAQAg4FHC8LUfX9RbV57XVb//YJs+3VboOg4AAAGNEoavGWP0fzcN1cCuMfrxwvXaWXjMdSQAAAIWJQzf0CY8RM9MSVNkWIi+vyBbR49Xu44EAEBAooThWxLbt9FfJo/UgZJK3fPSOtXU1buOBABAwKGE4ZRGJnfQ/1x/nrJ2Feu3725xHQcAgIDDY4twWjeOTNL2g2V65vM96tclWneMSXYdCQCAgMGZMJzRLy4foIn9Ouk3b2/Wil3FruMAABAwKGE4oxCP0WO3DVdyfJTufnGt8otPuI4EAEBAoIThO8VEhunZqaNUb6W7FmSrvKrWdSQAAPweJQxNktqxrZ64fYRyisp138vrVV/Po40AADgXlDA0WWafjvr1lQO0aGuh/u+j7a7jAADg13h3JJplakaKthce05OLd6lfl2hdM6yb60gAAPglzoShWYwx+s+rB2t0apx+9tcvtWFvietIAAD4JUoYmi081KM/3zFCnaIjdNeCbBWWVbqOBACA36GE4azEt4vQM1PSVF5Vq1kLslVZU+c6EgAAfoUShrM2oGuMHr5lmDYUlOrnr38pa3nHJAAATUUJwzm5dFAX/eslffX2+v3685JdruMAAOA3eHckztk9F/TW9sJy/fHD7erbOVqTBia4jgQAgM/jTBjOmTFGf7hhiAYnxurel7/Q9oPHXEcCAMDnUcLQItqEh+iZKWmKigjV9xes0ZHj1a4jAQDg0yhhaDFdYiP19OSRKiyr0t0vrlVNXb3rSAAA+CxKGFrU8B4d9L83nKeVu4/o/72z2XUcAAB8Fjfmo8VdNzxJ2w+W66klu9S/S7Qmp6e4jgQAgM/hTBi84qeX9tNF/Tvr//1ti7JyDruOAwCAz6GEwStCPEaP3DpMvTq11d0vrVNe8XHXkQAA8CmUMHhNdGSYnp0ySpI0c362jlXWOE4EAIDvoITBq3rER+nJO0Yo9/Bx3fvyetXV82gjAAAkShhaQUavjvrN1YP06bZD+sMH21zHAQDAJ/DuSLSKyWOTtf1gmf6ydLf6JkTrhpFJriMBAOAUZ8LQan5z1SCl94zXL9/YqHX5R13HAQDAKUoYWk1YiEdP3jFCXWIjNWvBWh0orXAdCQAAZyhhaFUd2obr2alpqqyp010LslVRXec6EgAATlDC0Or6JkTr0VuHafP+Mv30rxtkLe+YBAAEH0oYnLhoQIJ+dml/vfvlAT3+aY7rOAAAtDreHQlnZk/oqR2Fx/TgxzvUJyFalw3u4joSAACthjNhcMYYo99df56Gdm+v+19dr60HylxHAgCg1VDC4FRkWIiemTxS0ZGh+v78bB0ur3IdCQCAVkEJg3OdYyL1zJQ0HS6v0t0vrFN1bb3rSAAAeB0lDD5hSFJ7/eHGIVqde0T/8fYm3jEJAAh43JgPn3HNsG7aUXhMT3y2S/26RGv6uFTXkQAA8BrOhMGnPHBxP108MEG/fXeLPt9Z5DoOAABeQwmDT/F4jB6+ZZj6dI7WPS+u0+6icteRAADwCkoYfE67iFA9OzVNoSEefX9BtkoralxHAgCgxVHC4JO6x0XpyTtGKL/4hH688AvV1XOjPgAgsFDC4LPG9ozXf10zWEt2FOl37211HQcAgBbFuyPh024f00PbD5bp2WV71K9LtG5K6+46EgAALYIzYfB5v/7eQI3rHa9fvblJa/OOuI4DAECLoITB54WGePTE7SOU2D5S//L8Wu0rqXAdCQCAc0YJg19oHxWuZ6emqaqmXnfNz9aJ6lrXkQAAOCeUMPiN3p2j9dhtw7X1YJn+9bUNqucdkwAAP0YJg1+5oH9n/fLy/npv40E99ulO13EAADhrvDsSfueu83tq28FjemTRTvVLiNbl53V1HQkAgGbjTBj8jjFG/3PdeRrRo73uf3WDNu8vdR0JAIBmo4TBL0WGheipySPVPipMd83PVtGxKteRAABoFkoY/Fbn6Eg9MyVNR05Ua/YLa1VVW+c6EgAATUYJg18b3C1WD940TGvzjurf39wka3nHJADAP1DC4PeuHNJVP76oj15bW6A5y/a4jgMAQJNQwhAQ7ruojy4f3EX/895WLdlR5DoOAADfiRKGgODxGD1481D16xKjH760TjmHyl1HAgDgjChhCBhR4aF6ZspIhYd4NGtBNjfqAwB8GiUMASWpQ5T+cOMQ7T58XH//8oDrOAAAnBYlDAHnwv6d1btzO81Ztod3SwIAfBYlDAHHGKMZ41K1eX+ZVu054joOAACnRAlDQLp+RDd1iArTXCYrAAA+ihKGgBQZFqI7xiTr462Fyis+7joOAADfQglDwJqcnqxQj9Fzy3NdRwEA4FsoYQhYCTGR+t6QRL2WvVdllTWu4wAA8A2UMAS0GeNSdby6Tq+u2es6CgAA30AJQ0A7LylWo1Pi9NzyXNXW1buOAwDA1yhhCHgzMlO1r6RCH28pdB0FAICvUcIQ8C4emKDucW00h7kKAIAP8VoJM8bMNcYcMsZsOs3xnxpj1jd+bDLG1Blj4ryVB8ErxGM0LSNV2XlHtWFvies4AABI8u6ZsHmSLjvdQWvtH621w6y1wyT9UtISay3z5vCKm9OS1C4iVHOXczYMAOAbvFbCrLVLJTW1VN0maaG3sgDRkWG6ZVR3/f3LAzpYWuk6DgAA7u8JM8ZEqeGM2etneM0sY0y2MSa7qKio9cIhoEzLSFG9tZq/Itd1FAAA3JcwSVdJWn6mS5HW2qettWnW2rROnTq1YjQEku5xUbpkYBe9tCpfFdV1ruMAAIKcL5SwW8WlSLSSmeenqrSiRq+vK3AdBQAQ5JyWMGNMrKQJkt52mQPBIy25g87rFqu5y/eovt66jgMACGLenKhYKGmFpH7GmAJjzExjzGxjzOyTXnadpI+stce9lQM4mTFGMzNTtbvouJbs5P5CAIA7od76xtba25rwmnlqmLIAWs0V53XV797fqrnL9uiCfp1dxwEABClfuCcMaFXhoR5NSU/R5zsPa/vBY67jAACCFCUMQen20T0UGebRc4y3AgAcoYQhKHVoG67rRyTpjS/2qbi8ynUcAEAQooQhaM0Yl6Lq2nq9uCrfdRQAQBCihCFo9e4crQl9O+n5lXmqqmW8FQDQuihhCGozM1NVdKxK72444DoKACDIUMIQ1M7v01F9OrfTnGV7ZC3jrQCA1kMJQ1AzxmhGZqq2HCjTqj2nfXwpAAAtjhKGoHfd8G7qEBWmOcuYqwAAtB5KGIJeZFiI7hiTrEVbC5V7mCdoAQBaByUMkDQlPVmhHqN5WbmuowAAggQlDJDUOSZSVw1J1GvZe1VWWeM6DgAgCFDCgEYzMlN1vLpOr6ze6zoKACAIUMKARoO7xWp0apzmZeWqtq7edRwAQICjhAEnmZmZqn0lFfpoS6HrKACAAEcJA04yaUCCesRFMVcBAPA6ShhwkhCP0bSMFK3NO6r1e0tcxwEABDBKGPBPbh7VXdERoZrL2TAAgBdRwoB/0i4iVDeP6q73Nh7QgdIK13EAAAGKEgacwrSMFNVbqwUr8lxHAQAEKEoYcArd46J06aAuemlVvk5U17qOAwAIQE0qYcaYe40xMabBHGPMOmPMJd4OB7g0IzNVpRU1en3dPtdRAAABqKlnwmZYa8skXSKpg6TJkn7vtVSAD0hL7qAhSbF6bvke1ddb13EAAAGmqSXMNP7vFZKet9ZuPunPgIBkjNHMzFTtLjquJTuKXMcBAASYppawtcaYj9RQwj40xkRL4rkuCHiXD+6qhJgIxlsBAC2uqSVspqRfSBplrT0hKUzSdK+lAnxEeKhHU9JTtCznsLYfPOY6DgAggDS1hKVL2m6tLTHG3Cnp3yWVei8W4DtuH91DkWEexlsBAC2qqSXsz5JOGGOGSnpA0i5JC7yWCvAhHdqG6/oRSXpz/T4dLq9yHQcAECCaWsJqrbVW0jWSHrfWPiEp2nuxAN8yY1yqqmvr9dKqfNdRAAABoqkl7Jgx5pdqmKb4uzHGo4b7woCg0LtzO03s10kLVuSpqrbOdRwAQABoagm7RVKVGvbCDkpKkvRHr6UCfNCMcak6XF6lv2044DoKACAANKmENRavFyXFGmO+J6nSWss9YQgq5/fpqL4J7TR32R41XJ0HAODsNfWxRTdLWi3pJkk3S1pljLnRm8EAX2OM0YxxqdpyoEwrdx9xHQcA4OeaejnyV2rYCJtqrZ0iabSkX3svFuCbrh3eTXFtwxlvBQCcs6aWMI+19tBJf13cjK8FAkZkWIjuGNNDn2wrVO7h467jAAD8WFOL1AfGmA+NMdOMMdMk/V3Se96LBfiuyWOTFeoxmpeV6zoKAMCPNfXG/J9KelrSkMaPp621P/dmMMBXdY6J1FVDEvVq9l6VVtS4jgMA8FNNvqRorX3dWnt/48eb3gwF+LoZmak6UV2nV9fsdR0FAOCnzljCjDHHjDFlp/g4Zowpa62QgK8Z3C1WY1LjNC8rV7V19a7jAAD80BlLmLU22lobc4qPaGttTGuFBHzRjMxU7Sup0IebC11HAQD4Id7hCJylSQMS1CMuSnOXM1cBAGg+ShhwlkI8RtPHpWht3lGt31viOg4AwM9QwoBzcLqNxOoAACAASURBVFNad0VHhDLeCgBoNkoYcA7aRYTqllHd9d7GAzpQWuE6DgDAj1DCgHM0NSNF1lrNz8pzHQUA4EcoYcA56h4XpUsHddHC1fk6UV3rOg4AwE9QwoAWMDMzVaUVNXp93T7XUQAAfoISBrSAkckdNDQpVs8t26P6eus6DgDAD1DCgBZgjNGMzFTtPnxci3ccch0HAOAHKGFAC7nivK7qEhOpuctyXUcBAPgBShjQQsJCPJqSkaxlOYe17SCPVgUAnBklDGhBt4/uocgwj+Yy3goA+A6UMKAFtY8K1w0jkvTW+v06XF7lOg4AwIdRwoAWNn1cqqpr6/XiynzXUQAAPowSBrSw3p3baWK/Tnp+ZZ6qautcxwEA+ChKGOAFMzNTdbi8Sn/bcMB1FACAj6KEAV6Q2buj+ia005xle2Qt460AgG+jhAFeYIzRjHGp2nqgTCt2F7uOAwDwQZQwwEuuHd5NcW3DGW8FAJwSJQzwksiwEN05poc+2Vao3MPHXccBAPgYShjgRXeOTVaox+i55Yy3AgC+iRIGeFHnmEhdNTRRr60tUGlFjes4AAAfQgkDvGzGuFSdqK7TK2sYbwUA/AMlDPCywd1iNSY1TvOz8lRbV+86DgDAR1DCgFYwMzNV+0oq9OHmQtdRAAA+ghIGtIKLBiQoOT5Kc5btdh0FAOAjKGFAKwjxGE3LSNG6/BJ9kX/UdRwAgA+ghAGt5Ka07oqOCNXc5bmuowAAfAAlDGgl7SJCdevo7npv4wHtL6lwHQcA4BglDGhFU9JTZK3V/BW5rqMAAByjhAGtqHtclC4b3EULV+XrRHWt6zgAAIcoYUArm5mZqrLKWr2+tsB1FACAQ5QwoJWN6NFBQ5NiNXd5rurrres4AABHKGFAKzPGaEZmqvYcPq7FOw65jgMAcIQSBjhwxXld1SUmUnOW7XEdBQDgCCUMcCAsxKMpGclanlOsrQfKXMcBADhACQMcuX10D7UJC9FzyzkbBgDBiBIGONI+Klw3jOymt9bv1+HyKtdxAACtjBIGODR9XKqqa+v1wso811EAAK2MEgY41KtTO13Qr5NeWJmnqto613EAAK2IEgY4NjOzpw6XV+ud9ftdRwEAtCJKGODYuN7x6pcQrTnL9shaxlsBIFhQwgDHGsZbU7Tt4DGt2F3sOg4AoJVQwgAfcM2wboprG665jLcCQNCghAE+IDIsRHeO6aFPth3SnsPHXccBALQCShjgI+5MT1aYx6N5jLcCQFCghAE+onN0pK4amqjX1haotKLGdRwAgJdRwgAfMiMzRSeq6/Ty6nzXUQAAXkYJA3zIoMRYje0Zp/lZuaqtq3cdBwDgRZQwwMfMzOyp/aWV+mDzQddRAABeRAkDfMyF/TsrOT5Kc5irAICARgkDfEyIx2h6Roq+yC/RuvyjruMAALyEEgb4oJvSuis6MpTxVgAIYJQwwAe1jQjVraO66/1NB7WvpMJ1HACAF1DCAB81NSNF1lotWJHrOgoAwAsoYYCPSuoQpcsHd9XCVfk6XlXrOg4AoIVRwgAfNiMzRWWVtXp9XYHrKACAFkYJA3zYiB4dNLR7ez23PFf19dZ1HABAC6KEAT7MGKOZmanac/i4Ptt+yHUcAEALooQBPu7ywV3UNTaS8VYACDCUMMDHhYV4NCU9RVm7irX1QJnrOACAFkIJA/zAbaO7q01YCOOtABBAKGGAH2gfFa4bRnbT2+v3q+hYles4AIAWQAkD/MT0camqrqvXi6vyXEcBALQAShjgJ3p1aqcL+3fWCyvzVFlT5zoOAOAcUcIAPzJjXKoOl1frnQ37XUcBAJwjr5UwY8xcY8whY8ymM7xmojFmvTFmszFmibeyAIFiXO949e8SrbnL9shaxlsBwJ9580zYPEmXne6gMaa9pCclXW2tHSTpJi9mAQKCMUYzxqVq28FjemFVPkUMAPyY10qYtXappCNneMntkt6w1uY3vp45cKAJrh6WqNEpcfr1W5s0Ze5q5RefcB0JAHAWXN4T1ldSB2PMYmPMWmPMlNO90BgzyxiTbYzJLioqasWIgO+JDAvRy7PG6r+uGaQv8kt0ySNL9Jclu1RbV+86GgCgGVyWsFBJIyVdKelSSb82xvQ91QuttU9ba9OstWmdOnVqzYyAT/J4jKakp+jj+8fr/D6d9Lv3t+nqx5fry4IS19EAAE3ksoQVSPrQWnvcWntY0lJJQx3mAfxO19g2emZKmp66c4QOl1fp2ieW67fvbtHxqlrX0QAA38FlCXtbUqYxJtQYEyVpjKStDvMAfuuywV216IEJum10D81ZtkeXPLxUn23nNksA8GXenKhYKGmFpH7GmAJjzExjzGxjzGxJstZulfSBpC8lrZb0rLX2tHMWAM4sJjJM/33deXptdrrahIdo+nNr9KOFX/CYIwDwUcbf3uKelpZms7OzXccAfFpVbZ2eWrxbT3yWozbhIfrVFQN0U1qSjDGuowFAUDHGrLXWpp3qGIv5QACKCA3RvZP66L17M9U3oZ1+9vqXuu2Zldpz+LjraACARpQwIID17hytV2al63+uO0+b95fp0keW6onPclRdy5wFALhGCQMCnMdjdPuYHvrk/gmaNKCz/vjhdl31p2Val3/UdTQACGqUMCBIdI6J1JN3jNQzU9JUVlmjG/6cpd+8vUnlzFkAgBOUMCDIXDwwQR/fP0FT01O0YGWeLn5oiT7eUug6FgAEHUoYEITaRYTq/109SK//IEMxkWG6a0G27n5xrQ6VVbqOBgBBgxIGBLERPTro3R9n6qeX9tOirYd00UNL9NKqfNXX+9d0DQD4I0oYEOTCQjy654Le+vC+8RqcGKt/e3Ojbnl6hXIOHXMdDQACGiUMgCQptWNbvXTXGP3hxiHaUViuKx5dpkcW7VBVbZ3raAAQkChhAL5mjNHNad31yQMTdNngLnpk0U5d+dgyrck94joaAAQcShiAb+nYLkKP3TZcz00fpYrqOt301Ar96s2NKq2ocR0NAAIGJQzAaV3Qr7M++sl4fT8zVQtX5+vih5bo/Y0H5G/PnAUAX0QJA3BGbSNC9e/fG6i378lUp+gI/eDFdZr1/FodKK1wHQ0A/BolDECTnJcUq7fvGad/u6K/Pt9ZpIsfWqoFK3JVx5wFAJwVShiAJgsN8WjW+F766L4JGt6jvf7j7c268aksbT/InAUANBclDECz9YiP0oIZo/XwLUOVV3xCVz72uf7vw+2qrGHOAgCaihIG4KwYY3Td8CQtun+Crh6WqMc/y9Hlj36uFbuKXUcDAL9ACQNwTuLahuuhm4fphZljVFdvddszK/Xzv36pkhPVrqMBgE+jhAFoEZl9OurD+8Zr9oRe+uu6Ak16aIn+tmE/cxYAcBqUMAAtpk14iH5xeX+988NxSmzfRj9a+IVmzFujgqMnXEcDAJ9DCQPQ4gYlxurNu8fp198bqFV7juiSh5dqzrI9zFkAwEkoYQC8IsRjNDMzVR/9ZLzGpMbpt+9u0XVPLtfm/aWuowGAT6CEAfCqpA5RmjttlP5023DtL6nQ1Y8v1+/e36qKauYsAAQ3ShgArzPG6KqhiVp0/wTdOCJJf1myW5c+slTLdh52HQ0AnKGEAWg17aPC9b83DtHCu8Yq1GN055xVuv/V9TpynDkLAMGHEgag1aX3itd7956vH13YW++s369JDy3Rm18UMGcBIKhQwgA4ERkWogcu6ae///h8JcdH6SevbNCUuauVX8ycBYDgQAkD4FS/LtF6fXaG/uuaQfoiv0SXPLJEf1myS7V19a6jAYBXUcIAOOfxGE1JT9HH94/X+X066Xfvb9Nlj36uN78ooIwBCFiUMAA+o2tsGz09eaT+MnmkQj1GP3llgy58cIleWpWvqlomLQAEFuNvN8KmpaXZ7Oxs1zEAeFl9vdUn2w7p8c9ytGFvibrEROqu8T112+juigoPdR0PAJrEGLPWWpt2ymOUMAC+zFqr5TnFevyznVq5+4ji2oZrZmaqJqcnKyYyzHU8ADgjShiAgLA274ge/zRHn20vUnREqKZkJGvGuFTFt4twHQ0ATokSBiCgbNpXqicX5+j9TQcVGRqi20b30KzxPdUlNtJ1NAD4BkoYgICUc+iYnly8S2+v368QY3TDyCT9YEIv9YiPch0NACRRwgAEuL1HTuipJbv0WnaB6qzV1UMTdffEXuqTEO06GoAgRwkDEBQKyyr1zNLdenFVvipq6nTZoC6654LeOi8p1nU0AEGKEgYgqBw5Xq3nlu/RvKxcHaus1YS+nXTPBb01OjXOdTQAQYYSBiAolVXW6PkVeZq7bI+Kj1drdEqc7rmwt8b36ShjjOt4AIIAJQxAUKuortPLa/L19NLdOlBaqSFJsbp7Ym9dMjBBHg9lDID3UMIAQFJVbZ3eXLdPf16yS3nFJ9Q3oZ3unthb3xvSVaEhPMUNQMujhAHASWrr6vX3jQf0xGc52lFYruT4KM2e0EvXj+imiNAQ1/EABBBKGACcQn291cdbC/XEZzn6sqBUXWIiNWt8T902uofahFPGAJw7ShgAnIG1Vp/vPKzHP8vR6j1HFN82XDN4PiWAFkAJA4AmWpPb8HzKJTuKFB0ZqmkZKZo+LlVxbcNdRwPghyhhANBMGwtK9cRnOfpg80G1CQvRHWN66K7xPZUQw/MpATQdJQwAztLOwobnU76zoeH5lDelJWn2hF7qHsfzKQF8N0oYAJyj/OIT+vOSXXp9bcPzKa8Zlqi7J/ZW787tXEcD4MMoYQDQQg6WVurppbv10uo8VdXW6/LBXXT3xN4a3I3nUwL4NkoYALSw4vIqzV2+Rwuy8nSsqlYT+3XSDy/orbQUnk8J4B8oYQDgJaUVNXp+Ra7mLNujoydqNCY1Tj+8sLcye/N8SgCUMADwuhPVtXppVb6e+Xy3CsuqNLR7e90zsZcmDeD5lEAwo4QBQCupqq3TX9cW6Kklu7T3SIX6JUTr7gt66XtDEhVCGQOCDiUMAFpZbV293tmwX08u3qWcQ+VKiY/SDyb20nXDkxQeysPCgWBBCQMAR+rrrT7cfFCPf5ajzfvLlBjb8HzKW0f3UGQYz6cEAh0lDAAcs9Zq8Y4iPfFpjrLzjqpjuwj96MLeum10D86MAQGMEgYAPmTV7mI99PEOrdpzRD3iovTAJX111ZBEbuAHAtCZShj/+QUArWxMz3i9PGusnps+SlHhIbr35fW66vFlWrKjSP72H8YAzh4lDAAcMMbogn6d9d6Pz9fDtwxVaUWNps5drTueXaUNe0tcxwPQCihhAOCQx2N03fAkffLABP3mqoHadvCYrnliue5+ca12F5W7jgfAi7gnDAB8yLHKGj3z+R49+/luVdXW65ZR3XXfRX3UOSbSdTQAZ4Eb8wHAzxQdq9Ljn+7Ui6vyFRpiNDMzVf8yoZdiIsNcRwPQDJQwAPBTecXH9eBHO/TOhv1qHxWmeyb21uT0ZDbGAD9BCQMAP7dpX6n+94Nt+nznYSXGRuonF/fV9SOSeBQS4OOYqAAAPze4W6yenzlGL31/jDpGR+inf/1Slz+6VB9vKWTWAvBTlDAA8CMZvTvq7XvG6YnbR6imzuquBdm66akVys494joagGaihAGAnzHG6MohXfXRT8brv68brPwjJ3TjUyv0/flrtP3gMdfxADQR94QBgJ87UV2r55bn6qnFu1ReXasbRiTpJxf3Vbf2bVxHA4IeN+YDQBA4erxaTy7O0fysPMlIU9OTdffE3urQNtx1NCBoUcIAIIjsK6nQwx/v0BvrCtQ2PFSzJ/bS9HEpigoPdR0NCDqUMAAIQjsKj+kPH2zXoq2F6hQdofsm9dHNad0VFsLtwEBrYaICAIJQ34RoPTs1Ta/NTldyXJR+9eYmXfLwUv39ywPMWgA+gBIGAAFuVEqcXpudrmempCnUY3TPS+t07RPLlZVz2HU0IKhRwgAgCBhjdPHABH1w33j98cYhKjpWpdufXaXJc1Zp075S1/GAoMQ9YQAQhCpr6vT8ijw9sThHJSdqdPXQRD1wSV8lx7d1HQ0IKNyYDwA4pdKKGj29dJfmLNuj2jqr28f00I8u7KNO0RGuowEBgRIGADijwrJKPfrJTr2yZq8iQj36/vk9ddf5qYqODHMdDfBrlDAAQJPsLirXgx/t0N83HlBc23D96MLeun1MD0WEhriOBvglJioAAE3Ss1M7PXHHCL19zzj1S4jWf/5tiy56cIne/KJA9fX+9R/tgK+jhAEAvmVo9/Z66a4xmj9jtGIiw/STVzboyj8t02fbD7ExBrQQShgA4JSMMZrQt5Pe/VGmHr11mI5X1Wr6c2t069Mr9UX+UdfxAL9HCQMAnJHHY3TNsG5adP8E/efVg7SrqFzXPZml2c+vVc6hctfxAL/FjfkAgGYpr6rVnM/36Omlu1RZW6+bRibpvkl91SU20nU0wOfw7kgAQIs7XF6lxz/N0Yur8uQxRtPHpeoHE3opNopZC+ArlDAAgNfsPXJCD328Q2+t36eYyDDdPbGXpmakKDKMWQuAEgYA8Lot+8v0hw+3afH2InWNjdSPL+qjG0cmKSyE248RvNgJAwB43cDEGM2bPlovzxqrhJhI/fKNjbrowSV6Y12B6tgYA76FEgYAaFFje8brzbszNGdqmtpFhOr+Vzfo0keW6u9fHmDwFTgJJQwA0OKMMbpoQILe/VGmnrxjhCTpnpfW6co/LdOiLYUMvgKihAEAvMjjMbrivK768L7xeviWoTpRXavvL8jWtU9m6fOdRZQxBDVKGADA60I8RtcNT9Ki+yfof284T4ePVWnynNW65emVWr3niOt4gBO8OxIA0Oqqauv0ypq9+tOnOSo6VqXz+3TUA5f007Du7V1HA1oUExUAAJ9UUV2nF1bm6c9LdunI8WpNGpCg+y/uq4GJMa6jAS2CEgYA8GnlVbWat3yP/rJ0t45V1urKIV31k0l91LtztOtowDmhhAEA/ELpiRo9u2y35i7bo4qaOl07vJvuu6ivesRHuY4GnBVKGADArxSXV+kvS3drflau6uqtbkrrrh9d2FuJ7du4jgY0CyUMAOCXCssq9eRnOXppdb6MjG4f00N3X9BLnaMjXUcDmoQSBgDwawVHT+jxT3P02toChYUYTc1I0ezxvdShbbjraMAZUcIAAAEh9/BxPfrJTr21fp/ahodqRmaqZmamKrZNmOtowClRwgAAAWVH4TE9smiH3tt4ULFtwjRrfE9Ny0hR24hQ19GAb6CEAQAC0qZ9pXr44x36ZNshxbcN1w8m9tKdY5MVGRbiOhogiRIGAAhw6/KP6qGPdmhZzmElxETohxf20S1p3RUeytP54BYlDAAQFFbsKtaDH21Xdt5RdWvfRvdO6qPrh3dTaAhlDG6cqYTxTyUAIGCk94rXa7PTNX/GaMW3C9fP/vqlLn54qd5ev0919f510gGBjxIGAAgoxhhN6NtJb98zTk9PHqmIUI/ufXm9Ln90qT7YdED+dgUIgYsSBgAISMYYXTKoi9778fn6023DVVtvNfuFdbrq8WX6bNshyhic81oJM8bMNcYcMsZsOs3xicaYUmPM+saP//BWFgBA8PJ4jK4amqiP7huv/7tpqEpO1Gj6vDW64c9Zyso57Doegpg3z4TNk3TZd7zmc2vtsMaP//JiFgBAkAsN8ejGkUn69IGJ+u/rBmt/SaVuf3aVbnt6pdbmHXEdD0HIayXMWrtUEv9UAwB8SnioR3eMSdbin07Uf3xvoHYeOqYb/rxC055brY0Fpa7jIYi4vics3RizwRjzvjFm0OleZIyZZYzJNsZkFxUVtWY+AECAigwL0YzMVC392QX6+WX99UV+ia56fJn+5flsbT94zHU8BAGv7oQZY1IkvWutHXyKYzGS6q215caYKyQ9aq3t813fk50wAIA3lFXWaO6yPZrz+R6VV9fqqiGJum9SH/Xs1M51NPgxn9wJs9aWWWvLGz9/T1KYMaajqzwAgOAWExmm+yb11dKfXaDZE3rp4y2FmvTQEv30tQ3ae+SE63gIQM5KmDGmizHGNH4+ujFLsas8AABIUoe24fr5Zf219GcXaFpGqt7esF8XPrhY//7WRh0srXQdDwHEa5cjjTELJU2U1FFSoaTfSAqTJGvtU8aYH0r6gaRaSRWS7rfWZn3X9+VyJACgNR0ordDjn+bolTV75fEYTR6brB9M7KWO7SJcR4Mf4NmRAACco71HTujRT3bqjXUFigwL0eSxyZqSkaJu7du4jgYfRgkDAKCF7Coq16OLdurdL/dLki4emKBpGaka2zNOjXfZAF+jhAEA0MIKjp7QCyvz9fKafJWcqFG/hGhNzUjRtcMTFRUe6joefAQlDAAAL6msqdM76/fruaxcbT1QppjIUN0yqrumpKeoe1yU63hwjBIGAICXWWu1Jveo5mfl6oPNB1VvrS7qn6BpGSka1zueS5VB6kwljPOlAAC0AGOMRqfGaXRqnA6UVuiFlXlauHqvFm0tVO/O7TQ1I0XXD++mthH8Xy8acCYMAAAvqayp07tfHtD8rFxt3Feq6MhQ3TSyu6akJyulY1vX8dAKuBwJAIBD1lqtyy/RvKxcvb/xgOqs1cS+nTRtXKrO791RHg+XKgMVJQwAAB9RWFapF1fl66VV+TpcXqWeHdtqSnqybhiZpOjIMNfx0MIoYQAA+Jiq2jq9t/GA5mXlacPeErWLCNWNI5M0JT2Zh4YHEEoYAAA+bP3eEs3PytW7X+5XTZ3V+L6dNC0jWRP7duZSpZ+jhAEA4AcOHavUwlV79eKqPB06VqWU+ChNTk/RTWlJiuFSpV+ihAEA4Eeqa+v1weaDmp+Vq7V5RxUVHqLrR3TT1PQU9UmIdh0PzUAJAwDAT23aV6p5Wbl6Z8N+VdfWK7N3R03NSNGF/TsrhEuVPo8SBgCAnysur9LLa/bqhZV5OlBaqe5xbTR5bLJuSeuh2CguVfoqShgAAAGitq5eH20p1LzluVqde0RtwkJ07fBumpaRon5duFTpayhhAAAEoC37yzQ/K1dvrd+nqtp6je0Zp2kZKZo0IEGhIR7X8SBKGAAAAe3o8eqvL1XuK6lQt/ZtdOfYZN06qrs6tA13HS+oUcIAAAgCtXX1WrT1kOZn5WrF7mJFhHp0zbBETc1I0aDEWNfxghIlDACAILPtYJnmZ+XpzS8KVFlTr9EpcZqakaJLBiUojEuVrYYSBgBAkCo9UaNXs/dqwcpc7T1SoS4xkZqc3nCpMr5dhOt4AY8SBgBAkKurt/p0W8OlymU5hxUe6tFVQxI1LSNF5yVxqdJbzlTCQls7DAAAaH0hHqOLBybo4oEJyjl0TPOz8vT6ugK9vq5AI3q017Rxqbp8cBcuVbYizoQBABCkyipr9Fp2gRasyFVe8Ql1jo7QHWOSdfuYHuoUzaXKlsDlSAAAcFr19VZLdhRpXlauluwoUliI0ZXnddWNI7srvVc8j0c6B1yOBAAAp+XxGF3Qv7Mu6N9Zu4rK9fyKPL2+tkBvrd+vTtERumpIoq4ZlqghSbEyhkLWUjgTBgAAvqWypk6fbjukt9fv02fbilRdV6/Ujm119dCGQtazUzvXEf0ClyMBAMBZKz1Row82H9D/3969xchZ1nEc//7bLS09Cz3uttACCpTSThE5VZQIRA7S7QWeaQzxEhWMiYrRmHhtPFwQxaAGQ6NGhG6tKAQUlBhOttPSY6gcu9PSUlroCbbd/XsxQ9KiEqay+3Sm38/VzjOzM7/Jk333N+/7PvMuW1Xjsed2kgnzZkygu9LFdfOmM2X8qNIRj1mWMEmS9J7Y9tobrFhTY1m1l7W9rzMs4JLTJ7Go0slVc6cxftSI0hGPKZYwSZL0ntu8fS/Lq730rK7xws79nNAxjMvPmkJ3pZPLzpzCqBHDS0cszhImSZIGTWZSfWk3PdUaK9bUeGVvH+NGdXD13GksrnRx4WnH7wpLS5gkSRoSh/oH+Me/drKs2sv9a7exr6+fqePfWmHZxdyu8cfVCktLmCRJGnIH+vp5aOPL9FRrPLxpOwf7k9Mmj6F7fhfdlU5mTRpTOuKgs4RJkqSidu/v409rt7FsVS+PP/cqAPNnTmRxpZNr501nyrj2XGFpCZMkSceM2u4D/GF1jZ5qjfVb6yssF54xie5KFx8/Zyrj2miFpSVMkiQdk555eQ891Ro9q3t56dUDjOwYxhVnT6W70slHz5zMyI7WXmFpCZMkSce0zGTli7vpqfayYs1WXt3Xx4QTR3DNudNYNL+LC2efxLAWXGFpCZMkSS3jYP8Aj25+heXVGvev28b+vn6mjR/Fokr9kklzprfOCktLmCRJakn7+w7x4IbtLK/28vCmHRwaSM6YMpbFlU4Wze/ilJNHl474jixhkiSp5e3a18d9a7fSs6rGE8/XV1ied8pEuitdXDtvOpPGjiyc8D9ZwiRJUlvp3X2A5dUaPdVeNm7bw/BhwYfPmMTiBZ1cOWcaY0d2lI4IWMIkSVIb27RtDz3VXnqqNXp3H2DUiGFcOWca3fM7+cgHJnNCx7Bi2SxhkiSp7Q0MJCtf3MWyai9/XLOVXfsPMnH0CK45dzrd8zv50KyhX2FpCZMkSceVg/0D/P2ZHfRUazyw7mUOHOync8Iorqt0srjSxdnTxw9JDkuYJEk6bu178xAPbniZZat6+dszr9A/kJw5dRxfvHQ2nzp/5qC+9juVsGPjrDVJkqRBMmZkB92VLrorXezc+yb3Pb2VnmqN7a+/UTSXJUySJB03Th47kiUXz2LJxbMofTSw3HIBSZKkgkp/674lTJIkqQBLmCRJUgGWMEmSpAIsYZIkSQVYwiRJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYl1DESAAABM1JREFUJElSAZYwSZKkAixhkiRJBVjCJEmSCrCESZIkFRCZWTpDUyJiB/DCELzUJOCVIXgdDQ7nr/U5h63POWxtzt9749TMnPzf7mi5EjZUIuKpzDy/dA4dHeev9TmHrc85bG3O3+DzcKQkSVIBljBJkqQCLGH/289KB9D/xflrfc5h63MOW5vzN8g8J0ySJKkA94RJkiQVYAmTJEkqwBL2NhFxVURsiojNEfHN0nnUnIiYGRF/jYj1EbEuIm4unUnNi4jhEbEqIlaUzqLmRcTEiLg7IjZGxIaIuLh0JjUnIr7a2IaujYhfR8So0pnakSXsMBExHLgNuBqYA3w2IuaUTaUmHQK+lplzgIuAm5zDlnQzsKF0CB21HwN/zsyzgPk4ly0lIrqArwDnZ+ZcYDjwmbKp2pMl7EgXAJsz89nM7AN+A3QXzqQmZObWzFzZ+HkP9Y1/V9lUakZEzACuBe4onUXNi4gJwEeAnwNkZl9m7i6bSkehAzgxIjqA0UCtcJ62ZAk7Uhfw0mG3t+A/8JYVEbOABcDjZZOoST8Cvg4MlA6iozIb2AH8snFI+Y6IGFM6lN69zOwFvg+8CGwFXsvMB8qmak+WMLWliBgL/B64JTNfL51H705EfALYnpn/LJ1FR60DOA/4SWYuAPYBnl/bQiLifdSPAs0GOoExEXFD2VTtyRJ2pF5g5mG3ZzTG1EIiYgT1ArY0M+8pnUdNWQgsiojnqZ8O8LGIuKtsJDVpC7AlM9/aA3039VKm1nEF8Fxm7sjMg8A9wCWFM7UlS9iRngTeHxGzI+IE6iciLi+cSU2IiKB+LsqGzPxB6TxqTmbempkzMnMW9b+/v2Smn8BbSGZuA16KiDMbQ5cD6wtGUvNeBC6KiNGNberluLhiUHSUDnAsycxDEfEl4H7qq0F+kZnrCsdScxYCS4CnI6LaGPtWZt5XMJN0vPkysLTxYfZZ4MbCedSEzHw8Iu4GVlJfcb4KL2E0KLxskSRJUgEejpQkSSrAEiZJklSAJUySJKkAS5gkSVIBljBJkqQCLGGS9C5FxGURsaJ0DkntwRImSZJUgCVMUtuJiBsi4omIqEbE7RExPCL2RsQPI2JdRDwUEZMbj61ExGMRsSYi7m1cN4+IOCMiHoyI1RGxMiJObzz92Ii4OyI2RsTSxjeKS1LTLGGS2kpEnA18GliYmRWgH/g8MAZ4KjPPAR4Bvtv4lV8B38jMecDTh40vBW7LzPnUr5u3tTG+ALgFmAOcRv0qDZLUNC9bJKndXA58EHiysZPqRGA7MAD8tvGYu4B7ImICMDEzH2mM3wn8LiLGAV2ZeS9AZr4B0Hi+JzJzS+N2FZgFPDr4b0tSu7GESWo3AdyZmbceMRjxnbc97miv2fbmYT/343ZU0lHycKSkdvMQcH1ETAGIiJMi4lTq27vrG4/5HPBoZr4G7IqISxvjS4BHMnMPsCUiFjeeY2REjB7SdyGp7fkJTlJbycz1EfFt4IGIGAYcBG4C9gEXNO7bTv28MYAvAD9tlKxngRsb40uA2yPie43n+OQQvg1Jx4HIPNo98pLUOiJib2aOLZ1Dkt7i4UhJkqQC3BMmSZJUgHvCJEmSCrCESZIkFWAJkyRJKsASJkmSVIAlTJIkqYB/A7eyDDX3FUgDAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# dot_img_file = \"d_mod-\"+str(d_model)+\"-d_k-\"+str(d_k)+\"-d_v-\"+str(d_v)+\"-n_heads-\"+str(n_heads)+\"-enc_lays-\"+str(n_encoder_layers)+\"-dec_lays-\"+str(n_decoder_layers)+'-transformer.png'\n",
        "# tf.keras.utils.plot_model(transformer, to_file=dot_img_file, show_shapes=True)"
      ],
      "metadata": {
        "id": "RSK1eufbeQPL"
      },
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#transformer = keras.models.load_model('./Models/')"
      ],
      "metadata": {
        "id": "Oly3kYeU3FnV"
      },
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Testing"
      ],
      "metadata": {
        "id": "KJ_Gx7RVeNla"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def greedy_translate(model, source_sentence, target_sentence_start=sp_ben_bpe.bos_id()):\n",
        "  # Tokenizing and padding\n",
        "  source_seq = tokenize(source_sentence, sp_en_bpe)\n",
        "  source_seq = tf.keras.preprocessing.sequence.pad_sequences(source_seq, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "  predict_seq = [[target_sentence_start]]\n",
        "  \n",
        "  predict_sentence = [target_sentence_start] # Deep copy here to prevent updates on target_sentence_start\n",
        "  while predict_sentence[-1] != sp_ben_bpe.eos_id() and len(predict_sentence) < MAX_TOKENS:\n",
        "    predict_output = model([np.array(source_seq), np.array(predict_seq)], training=None)\n",
        "    predict_label = tf.argmax(predict_output, axis=-1) # Pick the label with highest softmax score\n",
        "    predict_seq = tf.concat([predict_seq, predict_label], axis=-1) # Updating the prediction sequence\n",
        "    predict_sentence.append(predict_label[0][0].numpy())\n",
        "  \n",
        "  if len(predict_sentence) == MAX_TOKENS:\n",
        "      return detokenize(predict_sentence, sp_ben_bpe, True, False)\n",
        "  return detokenize(predict_sentence, sp_ben_bpe)"
      ],
      "metadata": {
        "id": "zfg1AaKDnPQM"
      },
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow_probability as tfp"
      ],
      "metadata": {
        "id": "pLrtBO__iPPl"
      },
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I love you.\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", greedy_translate(transformer, sentence))"
      ],
      "metadata": {
        "id": "KtWeJl1JoapJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c5a93067-efea-437b-9778-f6e8c5743c7c"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  I love you.\n",
            "Bengali Sentence:  কিন্তু, আমি,, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি, আমি,\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentences = list(test_val_data['src'][:250])\n",
        "refs = [list(test_val_data['tgt'][:250])]\n",
        "sys = [greedy_translate(transformer, sentence) for sentence in sentences]"
      ],
      "metadata": {
        "id": "9XNckV-wBgiN"
      },
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "bleu = BLEU()\n",
        "bleu.corpus_score(sys, refs)"
      ],
      "metadata": {
        "id": "otRhPJC1VKyj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8628fc99-19a3-4fd1-e5fb-6e201d7b547e"
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "BLEU = 0.01 1.0/0.0/0.0/0.0 (BP = 1.000 ratio = 4.117 hyp_len = 14750 ref_len = 3583)"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "chrf = CHRF()\n",
        "chrf.corpus_score(sys, refs)"
      ],
      "metadata": {
        "id": "mlhToSrvYNqd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ceb015fe-5af5-46da-a4ce-492260e2515d"
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "chrF2 = 2.85"
            ]
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "ter = TER()\n",
        "ter.corpus_score(sys, refs)"
      ],
      "metadata": {
        "id": "DtIadLMJYe1w",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "88693974-f6ef-464b-9a62-d12b2f9cd40b"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TER = 211.89"
            ]
          },
          "metadata": {},
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def sampling_translate(model, source_sentence, temperature=0.6, target_sentence_start=sp_ben_bpe.bos_id()):\n",
        "  # Tokenizing and padding\n",
        "  source_seq = tokenize(source_sentence, sp_en_bpe)\n",
        "  source_seq = tf.keras.preprocessing.sequence.pad_sequences(source_seq, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "  predict_seq = [[target_sentence_start]]\n",
        "  logprob = 0\n",
        "  \n",
        "  predict_sentence = [target_sentence_start] # Deep copy here to prevent updates on target_sentence_start\n",
        "  while predict_sentence[-1] != sp_ben_bpe.eos_id() and len(predict_sentence) < MAX_TOKENS:\n",
        "    predict_output = model([np.array(source_seq), np.array(predict_seq)], training=None)\n",
        "    probs = predict_output[-1].numpy()[0]\n",
        "    log_probs = np.log(probs)\n",
        "    u = np.random.uniform(low=1e-6, high=1.0 - 1e-6, size=log_probs.shape)\n",
        "    g = -np.log(-np.log(u))\n",
        "    sample_idx = np.argmax(log_probs + g * temperature, axis=-1)\n",
        "    sample_prob = predict_output[-1].numpy()[0][sample_idx]\n",
        "    logprob += np.log(sample_prob)\n",
        "    predict_label = np.arange(predict_output.shape[-1])[sample_idx] # Sample the label softmax score\n",
        "    predict_seq = tf.concat([predict_seq, tf.cast([[predict_label]], dtype=tf.int32)], axis=-1) # Updating the prediction sequence\n",
        "    predict_sentence.append(predict_label)\n",
        "  \n",
        "  if len(predict_sentence) == MAX_TOKENS:\n",
        "      return detokenize(predict_sentence, sp_ben_bpe, True, False), logprob\n",
        "  return detokenize(predict_sentence, sp_ben_bpe), logprob"
      ],
      "metadata": {
        "id": "2dmVo_eQhE8v"
      },
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I love you.\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", sampling_translate(transformer, sentence))"
      ],
      "metadata": {
        "id": "CB5iHHlxh1cC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9c8aaec5-6b2e-4f5c-c962-2cb667ed10d6"
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  I love you.\n",
            "Bengali Sentence:  ('তবে তারা, তাদের মধ্যে তার সঙ্গে আবেদন করে।', -35.21933458652347)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def generate_samples(sentence, n_samples, model=None, temperature=0.6):\n",
        "    samples, log_probs = [], []\n",
        "    for _ in range(n_samples):\n",
        "        sample, logp = sampling_translate(model, sentence, temperature)\n",
        "        samples.append(sample)\n",
        "        log_probs.append(logp)\n",
        "    return samples, log_probs"
      ],
      "metadata": {
        "id": "xIxJ9HqTtxHY"
      },
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def rouge1_similarity(system, reference):\n",
        "    sys_counter = Counter(system)\n",
        "    ref_counter = Counter(reference)\n",
        "    overlap = 0\n",
        "    for token in sys_counter:\n",
        "        token_count_sys = sys_counter[token]\n",
        "        token_count_ref = ref_counter[token]\n",
        "        overlap += min(token_count_ref, token_count_sys)\n",
        "    precision = overlap / sum(sys_counter.values())\n",
        "    recall = overlap / sum(ref_counter.values())\n",
        "    if precision + recall != 0:\n",
        "        rouge1_score = 2 * ((precision * recall)/(precision + recall))\n",
        "    else:\n",
        "        rouge1_score = 0     \n",
        "    return rouge1_score"
      ],
      "metadata": {
        "id": "uhIlVoVwt7YC"
      },
      "execution_count": 41,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def weighted_avg_overlap(samples, log_probs):\n",
        "    scores = {}\n",
        "    for index_candidate, candidate in enumerate(samples):    \n",
        "        overlap, weight_sum = 0.0, 0.0\n",
        "        for index_sample, (sample, logp) in enumerate(zip(samples, log_probs)):           \n",
        "            if index_candidate == index_sample:\n",
        "                continue\n",
        "            sample_p = float(np.exp(logp))\n",
        "            weight_sum += sample_p\n",
        "            sample_overlap = rouge1_similarity(candidate, sample)\n",
        "            overlap += sample_p * sample_overlap\n",
        "        score = overlap / weight_sum\n",
        "        scores[index_candidate] = score\n",
        "    return scores"
      ],
      "metadata": {
        "id": "7KxiaRENt-C_"
      },
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def mbr_decode(sentence, n_samples, model=None, temperature=0.6):\n",
        "    samples, log_probs = generate_samples(sentence, n_samples, model, temperature)\n",
        "    scores = weighted_avg_overlap(samples, log_probs)\n",
        "    max_score_key = max(scores, key=scores.get)\n",
        "    return {sample:score for sample, score in zip(samples, scores.values())}"
      ],
      "metadata": {
        "id": "ny5EYLFWt_5u"
      },
      "execution_count": 43,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I love you.\"\n",
        "translated_sentence = mbr_decode(sentence, 2, transformer, 1)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "id": "lzrxKRMAaOgr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ccf4a7dd-2cbb-452e-e745-dee26d6049a1"
      },
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  I love you.\n",
            "Bengali:  {'কলকাতা, নোয়াখালী মতো-এর নিচে পূর্ণ করে তাদের কম কাছে খুশি করা এবং অন্যটি যা বিভিন্ন পরিশ্রম হয়।': 0.3739837398373983, 'তারা গোকীতে আমি এই ভূমিকা।': 0.37398373983739835}\n"
          ]
        }
      ]
    }
  ]
}