多臂老虎机.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BA2E2C14AAB24C118BAD0F36B3E30484",
    "jupyter": {},
    "mdEditEnable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# 多臂老虎机\n",
    "## 简介\n",
    "强化学习关注智能体和环境交互过程中的学习。在正式学习强化学习之前，我们通常需要先了解多臂老虎机问题，它可以认为是简化版的强化学习问题。与强化学习不同，多臂老虎机不存在状态信息，只有动作和奖励，可以算是最简单的“和环境交互中的学习”的一种形式。多臂老虎机中的探索与利用问题（Exploration and Exploitation）一直以来都是经典问题，理解它能够帮助我们之后学习强化学习的内容。\n",
    "## 问题介绍\n",
    "\n",
    "### 问题定义\n",
    "\n",
    "在多臂老虎机（Multi-Armed Bandit，MAB）问题中，有一个拥有 $K$ 根拉杆的老虎机，每一个拉杆都对应一个关于奖励的概率分布$\\mathcal{R}$。我们每次拉下其中一根拉杆，就可以获得一个从该拉杆对应的奖励概率分布中获得一个奖励$r$。我们的目标是: 在各个拉杆奖励的概率分布未知的情况下，从头开始尝试，并在操作 $T$ 次拉杆后，获得尽可能多的累积奖励。由于奖励的分布是未知的，我们就需要在“探索拉杆的获奖概率”和“根据经验选择获奖最多的拉杆”中进行权衡。于是，要采用怎样的操作策略，才能使获得的累积奖励最多，便是多臂老虎机问题。如果是你，会怎么做呢？\n",
    "    \n",
    "### 形式化描述\n",
    "    \n",
    "多臂老虎机问题可以表示为一个元组 $\\left\\langle \\mathcal{A}, \\mathcal{R} \\right\\rangle$，其中：\n",
    "* $\\mathcal{A}$ 为动作集合，其中一个动作表示拉动一个拉杆。若多臂老虎机一共有$K$个拉杆，那动作空间就是集合$\\{1, \\dots, K\\}$，我们用$a \\in \\mathcal{A}$ 表示任意一个动作。\n",
    "* $\\mathcal{R}$ 为奖励函数分布，每一个拉杆$a$都对应一个奖励分布$\\mathcal{R}( r | a) $，不同拉杆之间的奖励分布通常是不同的。\n",
    "\n",
    "假设每个时间步只能拉动一个拉杆，多臂老虎机的目标为最大化一段时间步$T$内累积的奖励: $\\max \\sum_{t=1}^{T} r_t,\\ r_t \\sim \\mathcal{R}\\left( \\cdot | a_t \\right)$。其中$a_t$表示在第$t$时间步拉动的拉杆，$r_t$表示拉动拉杆$a_t$后获得的奖励。\n",
    "\t\t\n",
    "### 累积懊悔 (Cumulative Regret)\n",
    "\n",
    "对于每一个拉杆a，我们定义其**期望奖励**为$Q(a) = \\mathbb{E}_{r\\sim\\mathcal{R}(\\cdot|a)} \\left[r\\right]$。于是，至少存在一个拉杆，它的期望奖励不小于其他任意一个拉杆。我们将该最优期望奖励表示为 $Q^* = \\max_{a \\in \\mathcal{A}} Q(a)$。为了更加直观、方便地观察一个拉杆离最优拉杆期望奖励的差距，我们引入**懊悔**概念。**懊悔**定义为当前拉杆$a$与最优拉杆的期望奖励差，即 $R(a) = Q^* - Q(a)$。**累积懊悔**即操作 $T$ 次拉杆后累积的懊悔总量，对于一次完整的T步决策$\\{a_1,a_2,...,a_T\\}$，累积懊悔为 $\\sigma_R = \\sum_{t=1}^T R(a_t)$。MAB问题的目标为最大化累积奖励，等价于最小化累积懊悔。\n",
    "\n",
    "### 估计期望奖励\n",
    "\n",
    "为了知道哪一个拉杆能获得更高的奖励，我们需要估计这个拉杆的期望奖励。只拉动一次获得的奖励存在随机性，所以需要拉动一个拉杆多次，然后计算得到的多次奖励的平均值。算法流程如下所示\n",
    "\n",
    "- 对于$\\forall a \\in \\mathcal{A}$,初始化期望奖励估值$\\ \\hat{Q}(a) = 0$和计数器 $N(a) = 0$\n",
    "- 主循环 $t = 1:T$\n",
    "\t- 选取某个拉杆 $a_t$\n",
    "\t- 得到奖励 $r_t$\n",
    "\t-\t更新计数器: $N(a_t) = N(a_t) + 1$\n",
    "\t- 更新估值: $\\displaystyle \\hat{Q}(a_t) = \\hat{Q}(a_t) + \\frac{1}{N(a_t)} \\left[ r_t - \\hat{Q}(a_t) \\right]$\n",
    "\t\n",
    "以上第4步我们如此更新估值，是因为这样我们可以进行增量式的平均数更新，公式如下。\n",
    "$$\n",
    "\\begin{align}\n",
    " Q_k &=\\frac{1}{k}\\sum_{i=1}^{k} r_i \\nonumber \\\\\n",
    " &= \\frac{1}{k}(r_k + \\sum_{i=1}^{k-1}r_i)  \\nonumber \\\\\n",
    "\t\t\t\t&= \\frac{1}{k}(r_k + (k-1)Q_{k-1}) \\nonumber \\\\\n",
    "\t\t\t\t&= \\frac{1}{k}(r_k + kQ_{k-1} - Q_{k-1}) \\nonumber \\\\\n",
    "\t\t\t\t&= Q_{k-1} + \\frac{1}{k}[r_k-Q_{k-1}]   \\nonumber\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "如果采用所有数求和再除以次数，存在一个缺点是每次更新的时间和空间复杂度均为$O(n)$。而采用增量式更新，时间和空间复杂度为 $O(1)$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "832A1BE0F15D4194B084572934B7A4B8",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "下面我们将用代码实现一个多臂老虎机，拉杆数为10。其中每个拉杆的奖励服从伯努利分布，即每次拉下拉杆有 $p$ 的概率获得奖励为1，剩下$1-p$ 的概率获得奖励为0。奖励为1代表获奖，奖励为0代表没有获奖。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "862C5FE8E2C34A3D839A6CCB4151B61D",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 导入需要使用的库，其中numpy是支持数组和矩阵运算的科学计算库，而matplotlib是绘图库\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "AA937B3EC836407E9347E71C2A2F08F8",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "随机生成了一个 10臂伯努利老虎机\n",
      "获得奖励概率最大的拉杆为1号，其获奖概率为0.7203\n"
     ]
    }
   ],
   "source": [
    "class BernoulliBandit:\n",
    "    \"\"\" 伯努利多臂老虎机，输入K为拉杆个数 \"\"\"\n",
    "    def __init__(self, K):\n",
    "        self.probs = np.random.uniform(size=K) # 随机生成K个0-1之间的数，作为每个拉杆的获奖概率\n",
    "        self.best_idx = np.argmax(self.probs) # 获奖概率最大的拉杆\n",
    "        self.best_prob = self.probs[self.best_idx] # 最大的获奖概率\n",
    "        self.K = K\n",
    "\n",
    "    def step(self, k):\n",
    "        # 当玩家选择了k号拉杆后，根据该老虎机k号拉杆获得奖励的概率返回 1（获奖）或 0（未获奖）\n",
    "        if np.random.rand() < self.probs[k]:\n",
    "            return 1\n",
    "        else:\n",
    "            return 0\n",
    "\n",
    "np.random.seed(1) # 设定随机种子，使实验具有可重复性\n",
    "K = 10\n",
    "bandit_10_arm = BernoulliBandit(K)\n",
    "print(\"随机生成了一个 %d臂伯努利老虎机\" % K)\n",
    "print(\"获得奖励概率最大的拉杆为%d号，其获奖概率为%.4f\" % (bandit_10_arm.best_idx, bandit_10_arm.best_prob))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "505287F0C4594BDA85283DA03201FD83",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "接下来我们以一个Solver基础类来实现上述的多臂老虎机的求解方案。根据前文的算法框架，我们需要实现下列函数：根据策略选择动作，根据动作获取奖励，更新奖励估值，更新累积懊悔和计数。在下面的MAB算法基本框架中，我们将 *根据策略选择动作*、*根据动作获取奖励* 和 *更新期望奖励估值* 放在 `run_one_step` 函数中，由每个继承Solver类的策略具体实现。而 *更新累积懊悔和计数* 则直接放在主循环 `run` 中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "D47B6E3385F44AF8BEAEDEC16A5137EE",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class Solver:\n",
    "    \"\"\" 多臂老虎机算法基本框架 \"\"\"\n",
    "    def __init__(self, bandit):\n",
    "        self.bandit = bandit                   \n",
    "        self.counts = np.zeros(self.bandit.K) # 每个拉杆的尝试次数\n",
    "        self.regret = 0. # 当前步的累积懊悔\n",
    "        self.actions = [] # 维护一个列表，记录每一步的动作\n",
    "        self.regrets = [] # 维护一个列表，记录每一步的累积懊悔\n",
    "\n",
    "    def update_regret(self, k):\n",
    "        # 计算累积懊悔并保存，k为本次行动选择的拉杆的编号\n",
    "        self.regret += self.bandit.best_prob - self.bandit.probs[k]\n",
    "        self.regrets.append(self.regret)\n",
    "\n",
    "    def run_one_step(self):\n",
    "        # 返回当前行动选择哪一个拉杆，由每个具体的策略实现\n",
    "        raise NotImplementedError\n",
    "\n",
    "    def run(self, num_steps):\n",
    "        # 运行一定次数，num_steps为总运行次数\n",
    "        for _ in range(num_steps):\n",
    "            k = self.run_one_step()\n",
    "            self.counts[k] += 1\n",
    "            self.actions.append(k)\n",
    "            self.update_regret(k)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "D6918C91CD28496C8C85824513AEE1A3",
    "jupyter": {},
    "mdEditEnable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 探索与利用问题\n",
    "在以上算法流程中，还没有一个策略告诉我们应该采取哪个动作，即拉动哪个拉杆，所以接下来我们将来学习如何设计一个策略。例如一个最简单的策略就是一直采取第一个动作，但这就非常取决于运气。如果运气绝佳，可能刚好是能获得最大期望奖励的拉杆，即最优拉杆；但如果运气很糟糕，就有可能是最小的期望奖励。于是在多臂老虎机任务中，一个经典的问题就是探索与利用的平衡问题。策略的设计就需要考虑这个问题。其中**探索(Exploration)** 是指尝试更多可能的拉杆，这个拉杆不一定会获取最优奖励，但能够清楚所有拉杆的获奖情况，也即对于一个10臂老虎机，我们要去把所有的拉杆都试一下才知道哪个拉杆可能获得最大的奖励。而**利用 (Exploitation)** 是指拉动已知平均奖励中最优的那个拉杆。由于已知的信息仅仅来自于有限次的交互观测，所以当前最优的拉杆不一定是真正全局最优。例如，对于一个10臂老虎机，我们只试过其中3个拉杆，接下来就一直拉动这3个拉杆中平均奖励最大的那个拉杆，但很有可能期望奖励最大的拉杆在剩下7个当中；即使我们对10个拉杆各自都尝试过了20次，发现5号拉杆的经验平均奖励是最高的，但仍然存在着微小的概率使得另一个6号拉杆的真实期望奖励是比5号拉杆更高的。\n",
    "\n",
    "于是在多臂老虎机问题中，我们需要平衡探索和利用的次数，使得能最大化累积奖励。一个比较大致的思路是在一开始的时候做比较多的探索，在对每个拉杆都有比较准确的估计后，再进行利用。目前存在一些比较经典的算法来解决这个问题，它们是$\\epsilon$-贪心算法、UCB算法、汤普森采样算法，我们接下来将分别着重介绍。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5F2D55E0CF274F6480276FFCBB0228F1",
    "jupyter": {},
    "mdEditEnable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## ϵ-贪心算法\n",
    "\n",
    "如果是完全的贪心算法，即在每一时刻采取期望奖励估值最大的动作（拉杆），这就是纯粹的利用，而没有探索。所以需要对完全贪心算法进行一些修改。其中比较经典的一种方法为$\\epsilon$-贪心算法 ($\\epsilon$-Greedy)，在完全贪心算法的基础上添加了噪声，每次以概率 $1 - \\epsilon$ 选择以往经验中期望奖励估值最高的那个拉杆，以概率 $\\epsilon$ **随机**选择一个拉杆。\n",
    "\t\t\n",
    "随着时间的不断推进，我们对各个动作的奖励估计得越来越准，此时我们就没太大必要继续花大力气进行探索。所以在$\\epsilon$-贪心算法的具体实现中，我们可以令$\\epsilon$ 随时间衰减，也即探索的概率将会不断降低。但是请注意，$\\epsilon$不会在有限的步数内衰减至0，因为基于有限步数观测的完全贪心算法仍然是一个局部信息的贪心算法，永远距离最优解有一个固定的差距。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "04867B5791134D748DBF1963F2066B20",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "我们接下来用代码实现一个$\\epsilon$-贪心算法，并用它去解决前文生成的10臂老虎机。我们设置$\\epsilon = 0.01$，以及$T=5000$。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "BCD67A732BFF4D1DB891284FAE4BB5CF",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class EpsilonGreedy(Solver):\n",
    "    \"\"\" Epsilon贪心算法，继承Solver类 \"\"\"\n",
    "    def __init__(self, bandit, epsilon=0.01, init_prob=1.0):\n",
    "        super(EpsilonGreedy, self).__init__(bandit)\n",
    "        self.epsilon = epsilon\n",
    "        self.estimates = np.array([init_prob] * self.bandit.K) #初始化对所有拉杆的奖励估值\n",
    "\n",
    "    def run_one_step(self):\n",
    "        if np.random.random() < self.epsilon:\n",
    "            k = np.random.randint(0, self.bandit.K)  # 随机选择一个拉杆\n",
    "        else:\n",
    "            k = np.argmax(self.estimates)  # 选择奖励估值最大的拉杆\n",
    "        r = self.bandit.step(k)  # 得到本次动作的奖励\n",
    "        self.estimates[k] += 1. / (self.counts[k] + 1) * (r - self.estimates[k])\n",
    "        return k"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BBDEC921EFEA4E669A9553226F252CCE",
    "jupyter": {},
    "mdEditEnable": false,
    "notebookId": "5f8f31cc46ba5e003077fa26",
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "为了更加直观地展示，我们把每一时间步的累积函数画出来。于是我们定义了以下画图函数，方便之后调用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "613E8777181E4711A3C956379BAEEB30",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def plot_results(solvers, solver_names):\n",
    "    \"\"\"生成累积懊悔随时间变化的图像。输入solvers是一个列表，列表中每个元素是一种特定的策略。而solver_names也是一个列表，包含每个策略的名称\"\"\"\n",
    "    for idx, solver in enumerate(solvers):\n",
    "        time_list = range(len(solver.regrets))\n",
    "        plt.plot(time_list, solver.regrets, label=solver_names[idx])\n",
    "    plt.xlabel('Time steps')\n",
    "    plt.ylabel('Cumulative regrets')\n",
    "    plt.title('%d-armed bandit' % solvers[0].bandit.K)\n",
    "    plt.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "ABC20D8DA4AE4E62896A8BA1DA50B632",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epsilon贪心算法的累积懊悔为： 25.526630933945313\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEWCAYAAABhffzLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAApX0lEQVR4nO3deXxU1f3/8deH7CxhBxGEuBVZiigpuJW6tGordbcu/bpgW1qr9Yu1+lW/bVX6s99WaWvVLmhFbYtr1RbRulalWooGRWTRIoqAbBGBsGSbzOf3x72JkxCSAXJnkpn38/GYR+6cu31OCJ+5c+6555i7IyIi2aNTugMQEZHUUuIXEckySvwiIllGiV9EJMso8YuIZBklfhGRLKPELxIBM7vXzP7fTtZdZGavpCiORucys61mtl8qzi3tlxK/pI2ZXWZmZWZWbWb3NrP+ODN7x8y2m9mLZjYkDWFmFHfv6u7vQ8sfTpLZlPglnVYD/w+Y3nSFmfUBHgN+BPQCyoCH2uKkFtDfvmQt/fFL2rj7Y+7+V2BDM6tPBxa5+yPuXgXcABxsZgc1dywz62lms8ys3Mw2hsuDEta/ZGY3mdmrwHZgPzNzM/uumS01sy1m9hMz29/M5phZhZk9bGb5CceYYGbzzWyTmf3LzEYlrDvEzN4Ij/MQUNhK9c3MbjezzeG3muMSVkw0syXhsd43s28nrDvazFaZ2ZVmtt7M1pjZxIT1vc1sZhj/a8D+TU7qZnaAmU0Cvg5cHTb/PNFKvJJBlPilvRoBvFX/xt23AcvC8uZ0Au4BhgCDgUrgjibbnA9MAroBH4ZlJwJjgMOAq4E7CRLiPsBI4FwAMzuU4JvJt4HewDRgppkVhB8OfwX+RPDt5BHgjFbqNw54H+gDXA88Zma9wnXrgQlAMTAR+FV4/np7Ad2BgcA3gN+YWc9w3W+AKmAAcHH42oG73wnMAG4Om3++2kq8kkGU+KW96gpsblK2mSBp78DdN7j7o+6+3d23ADcBX2iy2b3uvsjdY+5eG5b93N0r3H0RsBB41t3fd/fNwN+BQ8LtvgVMc/e57l7n7vcB1QQfGIcBecCt7l7r7n8BXm+lfusTtn8IeBc4KazLk+6+zAMvA88Cn0/YtxaYEu77FLAVGGpmOQQfOD92923uvhC4r5U4JAsp8Ut7tZXgijdRMbDFzAaHzRNbzWwrgJl1NrNpZvahmVUAs4EeYTKst7KZ86xLWK5s5n3XcHkIcGXYzLPJzDYRfCvYO3x95I1HPPyQljW3/d5hXb5sZv82s0/C83yF4JtBvQ3uHkt4vz2Msy+Q26SercUhWUiJX9qrRcDB9W/MrAtBe/Uid18RNk90dff6xHwlMBQY5+7FwPj6XROOuSdD0a4EbnL3Hgmvzu7+ALAGGGhmieca3Mrxmtt+tZkVAI8CU4H+7t4DeKpJPXamHIgRfCAlE4eG5s1SSvySNmaWa2aFQA6QY2aFZpYbrn4cGGlmZ4Tb/BhY4O7v7ORw3Qiu0DeFbeXXt3G4dwHfMbNxYa+gLmZ2kpl1A+YQJNzLwzqdDoxt5Xj9wu3zzOwsYBhBgs8HCgiTuJl9GTg+mQDdvY6gJ9QN4Teg4cCFLeyyDlCf/iykxC/p9EOCZH0N8F/h8g8B3L2coL36JmAjwc3Qc1o41q1AEfAx8G/g6bYM1N3LCNr57wjjeQ+4KFxXQ9AL6aJw3dkECbglc4EDw3hvAs4M71NsAS4HHg6PdR4wcxdCvYyg2WctcC/BDe+duRsYHjZd/XUXziEdnGkiFhGR7KIrfhGRLKPELyKSZZT4RUSyjBK/iEiWyW19k/Tr06ePl5SUpDsMEZEOZd68eR+7e9+m5R0i8ZeUlFBWVpbuMEREOhQza/bJbTX1iIhkGSV+EZEso8QvIpJlOkQbf3Nqa2tZtWoVVVVV6Q5FWlFYWMigQYPIy8tLdygiQgdO/KtWraJbt26UlJTQeJBDaU/cnQ0bNrBq1Sr23XffdIcjInTgpp6qqip69+6tpN/OmRm9e/fWNzORdqTDJn5ASb+D0L+TSPvSYZt6REQyxcpPtvOXeatobrTk0w4dxL59urTp+Tr0FX+65eTkMHr06IbXz372s10+RllZGZdffjkA9957L5dddtlux7N06VImTJjA/vvvz5gxYzjmmGOYPXv2bh+vJUcffbQeqhNpIw+8toJfv7CU2198b4fXhxu2tfn5IrviN7N9gD8CewFx4E53/7WZ3UAwoUV5uOl14YTRHU5RURHz58/fo2OUlpZSWlq6x7FUVVVx0kknMXXqVE4++WQAFi5cSFlZGePHj2+0bSwWIzdXX/ZE2ovtNXV0L8rjreuTmmxtj0X5vz8GXOnub4TT080zs+fCdb9y96kRnjutSkpKOPvss3nxxRcBuP/++znggAN45JFHuPHGG8nJyaF79+7Mnj2bl156ialTpzJr1qxGx/jwww+5+OKLKS8vp2/fvtxzzz0MHjyYiy66iOLiYsrKyli7di0333wzZ555JjNmzODwww9vSPoAI0eOZOTIkQDccMMNrF69muXLl9OnTx9+/etf853vfIcVK1YAcOutt3LkkUeybds2vve97/H2228Ti8W44YYbOOWUU6isrGTixIksXryYYcOGUVlZCcDdd9/NwoUL+dWvfgXAXXfdxZIlS/jlL38Z+e9ZpL3bXhPjjn+8x/aauha3+9eyjynKy0lRVBEmfndfQzAJNe6+xcyWAAOjONeNTyxi8eqKNj3m8L2Luf6rI1rcprKyktGjRze8v/baazn77LMBKC4u5rXXXuOPf/wjkydPZtasWUyZMoVnnnmGgQMHsmnTphaPfdlll3HBBRdw4YUXMn36dC6//HL++te/ArBmzRpeeeUV3nnnHU4++WTOPPNMFi1axKGHHtriMefNm8crr7xCUVER5513HldccQVHHXUUK1as4IQTTmDJkiXcdNNNHHvssUyfPp1NmzYxduxYvvjFLzJt2jQ6d+7MggULWLBgQcO5zjnnHEaNGsXNN99MXl4e99xzD9OmTWv5lyuSJcqWb+S3Ly2ja0EuOZ1a7uRw9NAdxlKLTEq+75tZCXAIwTyjRwKXmdkFQBnBt4KNzewzCZgEMHjw4FSEuctaauo599xzG35eccUVABx55JFcdNFFfO1rX+P0009v8dhz5szhsceCaVvPP/98rr766oZ1p556Kp06dWL48OGsW7eu2f1PO+00li5dymc+85mG45x88skUFRUB8Pzzz7N48eKG7SsqKtiyZQvPPvssM2fOZOrU4AtZVVUVK1asYPbs2Q33IkaNGsWoUaMA6NKlC8ceeyyzZs1i2LBh1NbW8tnPfrbFuolkmsqaOpaVb92h/J21wQXpXy45nIP2Kk51WDsVeeI3s67Ao8Bkd68ws98BPwE8/PkL4OKm+7n7ncCdAKWlpS1ODNzalXk6JHZhrF/+/e9/z9y5c3nyyScZPXr0Lt0fSDxeQUFBw3J9L4ARI0Y0upH7+OOPU1ZWxg9+8IOGsi5dPu0ZEI/HmTNnTsMHQeLxHn30UYYOHdpiDIm++c1v8tOf/pSDDjqIiRMnJl0nkUxx9aMLeOKt1Ttd36tzfgqjaV2kvXrMLI8g6c9w98cA3H2du9e5exy4CxgbZQzp8tBDDzX8PPzwwwFYtmwZ48aNY8qUKfTp04eVK1fudP8jjjiCBx98EIAZM2Zw1FFHtXi+8847j1dffZWZM2c2lG3fvn2n2x9//PHccccdDe/rP4ROOOEEbr/99oYPlDfffBOA8ePHM2PGDCC4abxgwYKGfceNG8fKlSu5//77G77piGSTj7dUc0C/rtx5/pgdXo9ecgT9igvTHWIjUfbqMeBuYIm7/zKhfEDY/g9wGrAwqhii1rSN/8QTT2zo0lldXc24ceOIx+M88MADAFx11VUsXboUd+e4447j4IMP5uWXX2722LfddhsXX3wxt9xyS8PN3ZYUFRUxa9Ysvv/97zN58mT69+9Pt27d+OEPf7jT41966aWMGjWKWCzG+PHj+f3vf8+PfvQjJk+ezKhRo3B3SkpKmDVrFpdccgkTJ05k1KhRjB49mrFjG39ef+1rX2P+/Pn07Nkz2V+fSMaoqYvTr1sBx4/YK92hJMWae2CgTQ5sdhTwT+Btgu6cANcB5wKjCZp6lgPfTvggaFZpaak37TO+ZMkShg0b1rZBt5H6iWP69OmT7lBSZsKECVxxxRUcd9xxza5vz/9eInvq5DteoVeXfO6d2L4aMMxsnrvv0F88yl49rwDNNQp3yD770rz6nj8HH3zwTpO+SEfm7twwcxGrNlbudJv3y7exVztrzmmJnuKJwPLly9MdQsr06NGD//znP+kOQyQyFVUx7pvzIQO6F9K7a/M3aUv6dOZLw/unOLLd16ETv7trALAOIKrmRJFUqK0LWqovOXp/Lji8JL3BtJEOO1ZPYWEhGzZsUFJp5+rH4y8s7Dhfg0USxeqCHJOX02HT5Q467BX/oEGDWLVqFeXl5a1vLGlVPwOXSEdUf8Wf28qTtx1Jh038eXl5mtFJRCKzrTpG3J3NlbUA5Ofqil9EJGPNmPsh//t440eMCnJTN4ha1JT4RUSa+KB8G/k5nbj6xGDokoK8HL7wmdQNohY1JX4RkSZq6uIU5efwzc/vl+5QIpE5jVYiIm2kti6eUW36TemKX0Sy0qbtNUyb/T7VtfEd1pUt30h+BnXfbEqJX0Sy0kvvlvO7l5bRJT+HTs08CHrssH5piCo1lPhFJKvE6uIsK9/Ge+uDiVNevOpo+nXLrgcMlfhFJKtMm/0+tzzzLhA8lFVcmJfmiFJPiV9EssqazZV0K8jllrNGMaB7EYUpnOS8vVDiF5Gs8o8l6ykuyuPEkQPSHUraKPGLSFbp1MnI9kF9lfhFJONs3l7Lgo82NbuuorKWk0fvndqA2hklfhHJODfOWsRjb3y00/V9u2ZXL56mlPhFpN355bPvMmtBi1Nxt2jN5iqGDyhmyikjdlhnZowcWLwn4XV4Svwi0u48v2Q922pijN23927tP2Jgd0767ABKS3q1cWSZQYlfRNqd2ro4Y4b05PZzD0l3KBkpcwejEJEOq6YunlFTHbY3+s2KSLtTG4tn9CBp6aamHhFJi9q6OKf+5lU+2lS5w7rNlbUZPSxyuinxi0habK6sZdHqCsbt24uD9urWaJ2ZcVbpoDRFlvmU+EUkLWpiwTj4px0ykHPGDk5zNNlF36VEJC2qw8RfkKc0lGq64heRSN0/dwWzFqzeoXx7TR0A+TnZNzpmuumjVkQi9Zd5K1n40WZq6+KNXnk5xucP7MPowT3SHWLW0RW/iEQqFnfGDOnJPRPHpjsUCUV2xW9m+5jZi2a2xMwWmdl/h+W9zOw5M1sa/uwZVQwikn41MT2M1d5E+a8RA65092HAYcClZjYcuAZ4wd0PBF4I34tIhqrVU7jtTmT/Gu6+xt3fCJe3AEuAgcApwH3hZvcBp0YVg4ik17bqGNWxoD1f2o+UfAybWQlwCDAX6O/uayD4cAD6pSIGEUmtrdUxPnfT86zaWElRvnrutCeRJ34z6wo8Ckx294pd2G+SmZWZWVl5eXl0AYpIJDZX1rK9po7TDx3IZccemO5wJEGkid/M8giS/gx3fywsXmdmA8L1A4D1ze3r7ne6e6m7l/bt2zfKMEUkArXhA1qfP7APA3sUpTkaSRRlrx4D7gaWuPsvE1bNBC4Mly8E/hZVDCKSPrF4kPhzO+nGbnsTZT/+I4HzgbfNbH5Ydh3wM+BhM/sGsAI4K8IYRCRNamIOoB497VBkid/dXwF2div/uKjOKyLpEauL83DZKrZVx4Bg3luA/Fz16Glv9OSuiLSJN1du4rrH325Ulpdj7K32/XZHiV9E2sSWqloAHvjWYXx2UHcAcjsZhXnqytneKPGLZIDqWB3TX1ne0MySDsvKtwLQu2s+XQuUWtoz/euIZIC3Vm7m50+/QycLZq9Kl77dCtire2Hazi/J2aXEb2adgK678iCWiESvsjYY2/6R7xzBmCEa91Ba1mo/KzO738yKzawLsBh418yuij40EUlWdZj4CzRBuSQhmSv+4e5eYWZfB54C/geYB9wSaWQiWW5dRRXzPtyY1LZvhNvlK/FLEpJJ/Hnh0AunAne4e2062xBFssWUWYt5csGapLfvZNCzc36EEUmmSCbxTwOWA28Bs81sCLA5yqBEBDZtr2H4gGJ+dfbopLbvXpRH324F0QYlGSGZxP+Eu99W/8bMVgAXRxeSSHbZsLWaF5asJ+7eqHzVxkr26dmZoXt1S1NkkqmSSfyPAofWv3F3N7MHgTGRRSWSRe7913Ju/8d7za47Yv/eKY5GssFOE7+ZHQSMALqb2ekJq4oBddQVaSPbquvokp/D81d+YYd1/brpv5q0vZau+IcCE4AewFcTyrcA34owJpGsUlsXJz+3EwO6a0wbSY2dJn53/xvwNzM73N3npDAmkawSi2syckmtZP7aNpjZC2a2EMDMRpnZDyOOSyRr1MRciV9SKpm/truAa4FaAHdfAJwTZVAi2aS+qUckVZL5a+vs7q81KUvfEIAiGSYWj5PbSQ9FSuok053zYzPbH3AAMzsTSP5xQhGhqraO99ZvbXbdhq01auqRlEom8V8K3AkcZGYfAR8AX480KpEMc+MTi3ngtRU7XT9u314pjEayXYuJ38xygEvc/Yvh6Jyd3H1LakITyRzlW6oY3KszP5owvNn1wwbo6VxJnRYTv7vXmdmYcHlbakISaf9WbNjOqo3bk95+bUUV/boV8KXh/SOMSiQ5yTT1vGlmM4FHgIbk7+6PRRaVSDt32m9fZcO2ml3a54QRSvrSPiST+HsBG4BjE8ocUOKXrFQXdzZsq+GsMYM4c8ygpPc7aK/iCKMSSV6rid/dJ6YiEJFU21Yd4+q/LKCiqnaX9qsfRfMz/bsxbj8NoiYdT6uJ38xua6Z4M1AWDusg0iG9s3YLT769hv37dqG4KG+X9h1b0ovDNXKmdFDJNPUUAgcRtPEDnAEsAr5hZse4++SIYhOJVFU4T+3/nT6KsepOKVkkmcR/AHCsu8cAzOx3wLPAl4C3I4xNJBLrKqqY/OB81lVUAVCUl5PmiERSK5nHBQcCXRLedwH2dvc6oDqSqEQitPCjzcx5fwM9u+Rz8sF7c2D/rukOSSSlkrnivxmYb2YvAQaMB34aPtD1fISxieyy99ZvoXxLy90s31oVTBn98zNGcUA/JX3JPsn06rnbzJ4CxhIk/uvcfXW4+qoogxPZFZsraznx1n8Si3ur23Yy6NUlPwVRibQ/yfTqMeA4YD93n2Jmg81sbDMjdopE4l/vfcyUWYt3mIy8qdo6JxZ3Lj/uQA5vpZtl7675SvyStZJp6vktECd4gGsKwdSLjwKfa2knM5tOMHXjencfGZbdQDBtY3m42XXu/tRuRS5ZY877G3hn7Ra+PHKvVrcdvU8Pzj9sCH27FaQgMpGOKZnEP87dDzWzNwHcfaOZJXOpdC9wB/DHJuW/cvepuxamZLPKmmAy8t/915h0hyKSEZJJ/LXhKJ314/H3JfgG0CJ3n21mJXsWnmSjVRu3c8H019heHfSz31xZS5cCdbkUaSvJJP7bgMeBfmZ2E3AmsCdz7l5mZhcAZcCV7r6xuY3MbBIwCWDw4MF7cDrpSLZWx7jy4bd4v3wbJ47Yi+7hE7WHDO6R3sBEMkhr4/F3Iph45WqCG7wGnOruS3bzfL8DfkLw7eEnwC+Ai5vb0N3vJJgAhtLS0ta7aUhGeGHJOuZ+8AkAPz9zVEPiF5G209p4/HEz+4W7Hw68s6cnc/d19ctmdhcwa0+PKZlle03QvPPK/xyjpC8SkWSe3H3WzM4Iu3XuETMbkPD2NGDhnh5TMktNLLh9pGEURKKTTBv/9wmGaYiZWRVBc4+7e4uDi5vZA8DRQB8zWwVcDxxtZqMJmnqWA9/e7cgl47g7c5ZtAKBAiV8kMsk8ubtbk4G6+7nNFN+9O8eS7LB0/VaeXrQWgMLcZL6Misju0P8uaTc2bQ8mRLn5jFHk5uhPUyQq+t8l7UZ9+35Jny6tbCkie0KJX9qNmrqgR0++mnlEIpXMzV3M7CjgQHe/J3xyt6u7fxBtaJLpauvifHfGG6zfEkzrUFEZNPXkq5lHJFLJjM55PVAKDAXuAfKAPwNHRhuaZLq1m6t4bvE6DtqrG/2LC+lRlMeoQd3Zr6+aekSilMwV/2nAIcAbAO6+2sx2q6ePSKL6h7W+d+yBnDRqQCtbi0hbSSbx17i7m1n9IG26HJM9MmPuh/zhnx9QHU52XpSvph2RVEom8T9sZtOAHmb2LYKxde6KNizJZLP/U87HW6s5Zmg/OufnMGZIr3SHJJJVknmAa6qZfQmoIGjn/7G7Pxd5ZJKxauuckt5duO3cQ9IdikhWSubm7hXAI0r20lZq6+Lk5ezx0E8ispuSaVwtBp4xs3+a2aVm1j/qoCSz1cTi5KnLpkjatPq/z91vdPcRwKXA3sDLZvZ85JFJxpr7wSfk6opfJG125bJrPbAW2AD0iyYcyQbFhbnEW528U0Si0mriN7NLzOwl4AWgD/Atdx8VdWCSuerizvC9WxzVW0QilEx3ziHAZHefH3EskiVq61zj8Yik0U4Tv5kVu3sFcHP4vlFna3f/JOLYJAO5OzV1urkrkk4tXfHfD0wA5hHMmJV4N86B/SKMSzLU1/8wF4DCPCV+kXTZaeJ39wnhz31TF45kuv+s2wLAGYcOSnMkItkrmZu7LyRTJpKMurhz/mFD6F9cmO5QRLJWS238hUBngsnSe/JpU08xQX9+kV0Wq3P14RdJs5ba+L8NTCZI8vP4NPFXAL+JNizJVLXxuCZaEUmzltr4fw382sy+5+63pzAmyRD1o3AmqonFdcUvkmbJjM55u5mNBIYDhQnlf4wyMOnY1lVUccH015pdp/Z9kfRKdurFowkS/1PAl4FXACV+2amN22sAmHLKCL7wmb4N5Z3MGNSzKF1hiQjJPbl7JnAw8Ka7TwxH5/xDtGFJR/DO2grO/N0cKsOZtBK5OwD79unCkN6atE2kPUkm8Ve6e9zMYmZWTDBYmx7eEpau28rW6hjnjRtMr875O6zvXJDD50o0u5ZIe5NM4i8zsx4E0y3OA7YCzTfeStb417KPeWbRWgAuPeYABvZQ841IR5HMzd3vhou/N7OngWJ3XxBtWNLeXfXIAj7aVEnPznnNXu2LSPvV0gNch7a0zt3fiCYk6QgqKms5/7Ah/GjCcI20KdLBtHTF/4sW1jlwbBvHIh3EJ9tq2FIdo7goV0lfpANq6QGuY/bkwGY2nWB0z/XuPjIs6wU8BJQAy4GvufvGPTmPpN6HG7YB0K+b+uOLdETJDNJ2QXOvJI59L3Bik7JrgBfc/UCCGb2u2eWIJe0qa4Lum0P36pbmSERkdyTTq+dzCcuFwHHAG7TyAJe7zzazkibFpxA8DAZwH/AS8D9JxCApNuWJxfzp38ubXRcPuujTtSCZPx8RaW+S6dXzvcT3ZtYd+NNunq+/u68Jj7vGzHY6abuZTQImAQwePHg3Tye7a+FHm+nXrZBTRjc/EGtxUR7DBmjeXJGOaHcu2bYDB7Z1IE25+53AnQClpaUe9fmkseq6OAf068rVJx6U7lBEpI0lM1bPEwS9eCC4JzAceHg3z7fOzAaEV/sDCJ4ClnaoJhZXjx2RDJXMFf/UhOUY8KG7r9rN880ELgR+Fv78224eRyK2ZE0F+/XVGDsimSiZNv6XAcJxenLD5V7u/klL+5nZAwQ3cvuY2SrgeoKE/7CZfQNYAZy1R9FLJLbXxADYVh1LcyQiEoVkmnomAT8BKoE4wUxcTisDtbn7uTtZddwuxigpVlUbB+DohOGURSRzJNPUcxUwwt0/jjoYaR9qYkHiz8/NSXMkIhKFZO7eLSPoySNZorauPvHr5q5IJkrmiv9a4F9mNhdomEDV3S+PLCqJzJML1jD5oTepi++8h2z9msI8JX6RTJRM4p8G/AN4m6CNXzqwJWsqqIs7lx5zQIvbFeR2ajRloohkjmQSf8zdvx95JLJbnl64lhWfbEt6+9c++IQuBblcefzQCKMSkfYsmcT/Ytiz5wkaN/W02J1TolcTi/PdGfNoodWmWYcM7hFJPCLSMSST+M8Lf16bUNZqd06JXnWsjrjDVScM5aIjSpLerzBPvXVEslkyD3Dtm4pAZNfV1gWX+l3yc+iikTJFJEnJPMDV7Nj77t7isMwSvU+7XeoKXkSSF9l4/BKt6lgd01/9AIC8HEtzNCLSkaR6PH5pI2+t3My0l9+na0Eu+/frmu5wRKQD2Z0ndFIyHr+0rDoWTH94z8TPcejgnmmORkQ6klSPxy9tJBbe2M3L0dO1IrJrUj0ev7SRmvDGbm4nte+LyK7ZaeI3swMI5sh9uUn5582swN2XRR6d7FT9Fb8GUhORXdXSFf+twHXNlFeG674aQTxZ7zcvvsef//1hq9tV1gZt/GrqEZFd1VLiL3H3BU0L3b3MzEqiCyl7bKuO8dDrK6mOfTr23YOvr8Adxn+mT6v79+pSwOBenaMMUUQyUEuJv7CFdUVtHUg2eundcqbMWrxD+aTx+3HdV4alISIRyQYtJf7Xzexb7n5XYmE4X+68aMPKDvVz2v7jyi+wd49PP0s1lo6IRKmlxD8ZeNzMvs6nib4UyAdOiziujLZxWw2rN1fywYZgOOUenfOV7EUkZXaa+N19HXCEmR0DjAyLn3T3f6Qksgx21rQ5vLd+KxB0x+ycr6QvIqmTzJANLwIvpiCWrLFucxVfHNaPs0r3YUD3Ql3ti0hKaSzf3VRVW8eUWYvZUhXb5X231sQYNqCYE0bsFUFkIiItU+LfTYtWV3D/3BUM6F5I0S5ese/ftyvj9u0dUWQiIi1T4m/GO2srWF9R3eI2C1dvBuCO8w5hzJBeqQhLRKRNKPE3sa06xoTbXiGW5ES2/bq19LiDiEj7o8TfxLaaGLG4M2n8fpwwon+L23YvymMfPTkrIh2MEn8T9fPY7t+3i5pwRCQjaYSvJmpj9fPY6lcjIpkp46/4t9fEePGdcmLxeOsbA2s3VwEa9VJEMldaEr+ZLQe2AHVAzN1LozrXY298xA//unCX99NNWxHJVOm84j/G3T+O+iQVVbUA/P2/P09Bks03hXk5jQZNExHJJBnf1FNdGzTxDO3fjU6aplBEJG03dx141szmmdmk5jYws0lmVmZmZeXl5bt9oqpYHfm5nZT0RURC6Ur8R7r7ocCXgUvNbHzTDdz9TncvdffSvn377vaJqmvjFKqHjohIg7RkRHdfHf5cDzwOjI3qXNWxOgo0+qWISIOUJ34z62Jm3eqXgeOBXe92k6TaOidfXTNFRBqk4+Zuf4KZverPf7+7Px3VyeriTo7a90VEGqQ88bv7+8DBqTpfTIlfRKSRjG8DqYvHlfhFRBJkQeJ3cpX4RUQaZEXi1xW/iMinMj7xq41fRKSxjE/8uuIXEWksKxK/2vhFRD6V8YlfTT0iIo1lfOJXU4+ISGNZkvgzvpoiIknL+IyoNn4RkcYyPvGrjV9EpLGMT/x18Tg5psQvIlIvCxK/k5OjxC8iUi8rEr/a+EVEPpXxiT8WdzX1iIgkyPjEr378IiKNZUXiz1Ubv4hIg6xI/LriFxH5VMYnfrXxi4g0lvGJP64hG0REGsn4jBhTG7+ISCMZn/jVxi8i0ljGJ/6YhmwQEWkkoxO/uxN3dMUvIpIgoxN/XdwBNGSDiEiCjE78b67cBEAnJX4RkQYZnfhf++ATAA7br1eaIxERaT8yOvHXxOIAHDq4Z5ojERFpPzI78dfFycsxTL16REQaZHTir43FycvJ6CqKiOyytGRFMzvRzN41s/fM7JqozlNbFyc/V4lfRCRRbqpPaGY5wG+ALwGrgNfNbKa7L27rcw0bUExVbbytDysi0qGlPPEDY4H33P19ADN7EDgFaPPEf87YwZwzdnBbH1ZEpENLRzvIQGBlwvtVYVkjZjbJzMrMrKy8vDxlwYmIZLp0JP7mutj4DgXud7p7qbuX9u3bNwVhiYhkh3Qk/lXAPgnvBwGr0xCHiEhWSkfifx040Mz2NbN84BxgZhriEBHJSim/uevuMTO7DHgGyAGmu/uiVMchIpKt0tGrB3d/CngqHecWEcl2erpJRCTLKPGLiGQZc9+hJ2W7Y2blwIe7uXsf4OM2DKcjUJ2zg+qcHfakzkPcfYf+8B0i8e8JMytz99J0x5FKqnN2UJ2zQxR1VlOPiEiWUeIXEcky2ZD470x3AGmgOmcH1Tk7tHmdM76NX0REGsuGK34REUmgxC8ikmUyOvGnaorHqJnZdDNbb2YLE8p6mdlzZrY0/NkzYd21YZ3fNbMTEsrHmNnb4brbrB3PQm9m+5jZi2a2xMwWmdl/h+UZW28zKzSz18zsrbDON4blGVtnCGblM7M3zWxW+D6j6wtgZsvDeOebWVlYlrp6u3tGvggGgFsG7AfkA28Bw9Md127WZTxwKLAwoexm4Jpw+Rrg5+Hy8LCuBcC+4e8gJ1z3GnA4wZwIfwe+nO66tVDnAcCh4XI34D9h3TK23mF8XcPlPGAucFgm1zmM9fvA/cCsbPjbDuNdDvRpUpayemfyFX/DFI/uXgPUT/HY4bj7bOCTJsWnAPeFy/cBpyaUP+ju1e7+AfAeMNbMBgDF7j7Hg7+YPybs0+64+xp3fyNc3gIsIZipLWPr7YGt4du88OVkcJ3NbBBwEvCHhOKMrW8rUlbvTE78SU3x2IH1d/c1ECRJoF9YvrN6DwyXm5a3e2ZWAhxCcAWc0fUOmz3mA+uB59w90+t8K3A1EE8oy+T61nPgWTObZ2aTwrKU1TstwzKnSFJTPGagndW7Q/4+zKwr8Cgw2d0rWmjCzIh6u3sdMNrMegCPm9nIFjbv0HU2swnAenefZ2ZHJ7NLM2Udpr5NHOnuq82sH/Ccmb3TwrZtXu9MvuLP9Cke14Vf9Qh/rg/Ld1bvVeFy0/J2y8zyCJL+DHd/LCzO+HoDuPsm4CXgRDK3zkcCJ5vZcoKm2GPN7M9kbn0buPvq8Od64HGCpumU1TuTE3+mT/E4E7gwXL4Q+FtC+TlmVmBm+wIHAq+FXx23mNlh4Z3/CxL2aXfCGO8Glrj7LxNWZWy9zaxveKWPmRUBXwTeIUPr7O7Xuvsgdy8h+P/5D3f/LzK0vvXMrIuZdatfBo4HFpLKeqf77naUL+ArBL1BlgH/m+549qAeDwBrgFqCT/lvAL2BF4Cl4c9eCdv/b1jnd0m4yw+Uhn9gy4A7CJ/cbo8v4CiCr60LgPnh6yuZXG9gFPBmWOeFwI/D8oytc0K8R/Npr56Mri9BT8O3wtei+tyUynpryAYRkSyTyU09IiLSDCV+EZEso8QvIpJllPhFRLKMEr+ISJZR4peMY2a9w1EP55vZWjP7KFzeama/TVEMo83sK6k4l8iuyuQhGyRLufsGYDSAmd0AbHX3qSkOYzRBH+unUnxekVbpil+yhpkdnTDm+w1mdp+ZPRuOjX66md0cjm3+dDhcRP145y+Hg2k9U/9IfZPjnmVmCy0YR392+KT4FODs8JvG2eHTmtPN7HULxp4/Jdz3IjP7W3jOd83s+rC8i5k9GR5zoZmdnbrflGQ6XfFLNtsfOIZgvPM5wBnufrWZPQ6cZGZPArcDp7h7eZh8bwIubnKcHwMnuPtHZtbD3WvM7MdAqbtfBmBmPyUYkuDicFiG18zs+XD/scBIYDvwenjeIcBqdz8p3L97ZL8FyTpK/JLN/u7utWb2NsHEPU+H5W8DJcBQgoT8XDgqaA7B0BlNvQrca2YPA481sx6C8VhONrMfhO8LgcHh8nNh8xRm9hjBcBVPAVPN7OcEQxn8c7drKdKEEr9ks2oAd4+bWa1/On5JnOD/hgGL3P3wlg7i7t8xs3EEE4rMN7PRzWxmBN8o3m1UGOzXdNwUd/f/mNkYgvGJ/s/MnnX3KbtYP5FmqY1fZOfeBfqa2eEQDBNtZiOabmRm+7v7XHf/MfAxwRC6WwimjKz3DPC9cBRFzOyQhHVfsmC+1SKCGZReNbO9ge3u/mdgKsHUmyJtQlf8IjsRttWfCdwWtrHnEswYtajJpreY2YEEV/UvEIy6uAK4xoLZtP4P+Em474Iw+S8HJoT7vwL8CTgAuN/dyyyYUPsWM4sTjMp6SUTVlCyk0TlF0sjMLiLhJrBIKqipR0Qky+iKX0Qky+iKX0Qkyyjxi4hkGSV+EZEso8QvIpJllPhFRLLM/wcQHqK5hc5mkQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(1)\n",
    "epsilon_greedy_solver = EpsilonGreedy(bandit_10_arm, epsilon=0.01)\n",
    "epsilon_greedy_solver.run(5000)\n",
    "print('epsilon贪心算法的累积懊悔为：', epsilon_greedy_solver.regret)\n",
    "plot_results([epsilon_greedy_solver], [\"EpsilonGreedy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "795061BFE87742918BD471E40D0135CD",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "通过上面实验我们发现，在经历了开始的一小段时间后，$\\epsilon$-贪心算法的累积懊悔几乎是线性增长的。这是$\\epsilon=0.01$时的结果，因为一旦做出了随机拉杆的探索，那么带来的懊悔值是固定的。那其他不同的 $\\epsilon$ 取值又会带来怎样的变化呢？我们继续使用该10臂老虎机，我们尝试不同的参数: $\\{10^{-4}, 0.01, 0.1, 0.25, 0.5\\}$，查看相应实验结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "FFCBC0D7DD7E4C958A36274545B5FA16",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABRpklEQVR4nO3dd3yV9fn4/9eV5GTvBYEkJGzChrAUHCAqooDbunerba2davv5tlY7/HT9Ouyn1da6qqJVRLTKEEXZmLAkYQQSRkhIIHud/f79cQ4YICQBkpzk5Ho+HiHnvO913SG5r3Pf9/u+3mKMQSmllGpNgK8DUEop1f1pslBKKdUmTRZKKaXapMlCKaVUmzRZKKWUapMmC6WUUm3SZKFUNyEiL4nIL84w7W4RWdNFcZy0LRGpF5GBXbFt1X1pslA9ioh8S0RyRMQmIi+1MH2WiOwSkUYR+VREBvggTL9ijIk0xhRC6wlN+TdNFqqnKQF+Afzr1AkikggsAv4fEA/kAG92xEbFQ/9eVK+lv/yqRzHGLDLGLAYqWph8HZBnjPmPMcYKPAmMFZHhLa1LROJE5AMROSoiVd7Xqc2mrxKRX4rIWqARGCgiRkQeFpECEakTkadFZJCIrBeRWhF5S0SCm63jahHZKiLVIrJORMY0mzZeRDZ71/MmENrG7ouI/EVEarxnT7OaTbhHRHZ611UoIl9vNu0SESkWke+LSLmIlIrIPc2mJ4jIEm/8m4BBp2zUiMhgEXkQuA34kffS1PttxKv8iCYL5U9GAtuOvzHGNAD7vO0tCQBeBAYA6UAT8Owp89wBPAhEAQe8bVcCE4GpwI+A5/EcRNOAUcDXAERkAp4zoK8DCcBzwBIRCfEmlMXAq3jOgv4DXN/G/k0BCoFE4GfAIhGJ904rB64GooF7gP/Pu/3j+gIxQH/gPuCvIhLnnfZXwAqkAPd6v05jjHkeeA34jffS1DVtxKv8iCYL5U8igZpT2mrwHOhPY4ypMMa8Y4xpNMbUAb8ELj5ltpeMMXnGGKcxxuFt+19jTK0xJg/YASw3xhQaY2qAj4Dx3vkeAJ4zxmw0xriMMS8DNjxJZipgAf5ojHEYY94Gvmhj/8qbzf8msBuY692X/xpj9hmPz4DlwIxmyzqAp7zLfgjUA8NEJBBPkvqpMabBGLMDeLmNOFQvpMlC+ZN6PJ+sm4sG6kQk3XvppF5E6gFEJFxEnhORAyJSC3wOxHoPoMcdamE7Zc1eN7XwPtL7egDwfe8lqGoRqcZz9tHP+3XYnFzJ8wCta2n+ft59mSMiG0Sk0rudq/CcgRxXYYxxNnvf6I0zCQg6ZT/bikP1QposlD/JA8YefyMiEXiuv+cZYw56L51EGmOOH8y/DwwDphhjooGLji/abJ3nU5b5EPBLY0xss69wY8wbQCnQX0Sabyu9jfW1NH+JiIQA7wC/A/oYY2KBD0/ZjzM5CjjxJLH2xKFlqnspTRaqRxGRIBEJBQKBQBEJFZEg7+R3gVEicr13np8C240xu86wuig8ZwLV3mv/P+vgcP8BfENEpnh7U0WIyFwRiQLW4zlIP+Ldp+uAyW2sL9k7v0VEbgRG4EkKwUAI3gO/iMwBLm9PgMYYF54eZE96z7SygLtaWaQM0GcueiFNFqqn+R88B/jHgdu9r/8HwBhzFM/1918CVXhuCN/Syrr+CIQBx4ANwNKODNQYk4PnvsWz3nj2And7p9nx9N662zvtZjwH7dZsBIZ44/0lcIP3vksd8AjwlnddtwJLziLUb+G5JHUEeAnPTf8zeQHI8l5WW3wW21A9nOjgR0oppdqiZxZKKaXapMlCKaVUmzRZKKWUapMmC6WUUm0KanuWnikxMdFkZGT4OgyllOpRcnNzjxljkk5t99tkkZGRQU5Ojq/DUEqpHkVEWnyCXy9DKaWUapMmC6WUUm3SZKGUUqpNnXbPQkT+hae+frkxZpS3LR7PyGUZwH7gJmNMlXfaE3jq7LuAR4wxy7ztE/GUIAjDUwfnO+YcHzt3OBwUFxdjtVrPfceUz4WGhpKamorFYvF1KEr1Gp15g/slPDVxXmnW9jiw0hjzjIg87n3/mLd42S14BqnpB3wsIkO9Rc7+hmfwmQ14ksWVeMYMOGvFxcVERUWRkZHBycU7VU9hjKGiooLi4mIyMzN9HY5SvUanXYYyxnwOVJ7SPJ+vBlZ5GVjQrH2hMcZmjCnCU3BtsoikANHGmPXes4lXmi1z1qxWKwkJCZooejARISEhQc8OlepiXX3Poo8xphTA+z3Z296fkwdfKfa29fe+PrW9RSLyoIjkiEjO0aNHzzTPuUevugX9P1Sq63WXG9wt/fWbVtpbZIx53hiTbYzJTko67ZkSpZTyaxUl9axfvA/j7vhq4l2dLMq8l5bwfi/3thdz8khdqUCJtz21hfZea8mSJTzzzDMAPPnkk/zud7/rtG3l5uYyevRoBg8ezCOPPMKZ+hX8+te/ZvDgwQwbNoxly5a1ubzNZuPmm29m8ODBTJkyhf37959Y5sorryQ2Nparr7660/ZLKX+0beUh/vOrHHauLaG+2tbh6+/qZLGEr0bhugt4r1n7LSISIiKZeAZ42eS9VFUnIlO9w0ne2WyZXmnevHk8/vjjXbKthx56iOeff56CggIKCgpYuvT0sYHy8/NZuHAheXl5LF26lIcffhiXy9Xq8i+88AJxcXHs3buX7373uzz22GMn1vfDH/6QV199tUv2Tyl/4LC5+OyN3az5TwHJGVHc8Hg2UfGhHb6dTksWIvIGnqEjh4lIsYjcBzwDzBaRAmC29z3GmDw8o3zl4xmt7JvenlAADwH/xHPTex/n2BOqu/j3v//N5MmTGTduHF//+tdxuVxERkby/e9/nwkTJjBr1iyO32/585//TFZWFmPGjOGWWzwDvr300kt861vfOm29W7duZerUqYwZM4Zrr72WqqoqAC655BIee+wxJk+ezNChQ1m9enW74iwtLaW2tpZp06YhItx5550sXrz4tPnee+89brnlFkJCQsjMzGTw4MFs2rSp1eXfe+897rrL85nhhhtuYOXKlSfOOmbNmkVUVNRZ/UyV6o3cbkPOR/t55Sfr2PHZYcZelsaC744nOiGsU7bXaV1njTFfO8OkWWeY/5d4hoo8tT0HGNWBoQHw8/fzyC+p7dB1ZvWL5mfXjDzj9J07d/Lmm2+ydu1aLBYLDz/8MK+99hoNDQ1MmDCB3//+9zz11FP8/Oc/59lnn+WZZ56hqKiIkJAQqqurW932nXfeyV/+8hcuvvhifvrTn/Lzn/+cP/7xjwA4nU42bdrEhx9+yM9//nM+/vhjdu/ezc0339ziulatWsXhw4dJTf3qCmBqaiqHDx8+bd7Dhw8zderU0+azWCxnXP7w4cOkpXmuOgYFBRETE0NFRQWJiYmt7qNSysPa4GDFC3kczK8keUAUl987krSs+E7dpt8WEuyOVq5cSW5uLpMmTQKgqamJ5ORkAgICThy4b7/9dq677joAxowZw2233caCBQtYsGDBGddbU1NDdXU1F198MQB33XUXN95444npx9c3ceLEE/cHhg0bxtatW8+4zpbuT7TUC+lM87W2fHvXrZQ63d7ccla8kIfbbZgyL5OJV2YgAZ3/99Nrk0VrZwCdxRjDXXfdxa9//euT2p9++umT3h8/cP73v//l888/Z8mSJTz99NPk5eWd03ZDQkIACAwMxOl0ArR5ZpGamkpx8Ve9louLi+nXr99p86ampnLo0KHT5mtt+ePLpKam4nQ6qampIT6+cz8VKdXTOWwuNizex/ZPi0lIjeSim4fSb0hsl22/u3Sd7RVmzZrF22+/TXm5pxNYZWUlBw4cwO128/bbbwPw+uuvM336dNxuN4cOHeLSSy/lN7/5DdXV1dTX17e43piYGOLi4k7cj3j11VdPnGWcyfEzi5a+YmNjSUlJISoqig0bNmCM4ZVXXmH+/PmnrWfevHksXLgQm81GUVERBQUFTJ48udXl582bx8sve57NfPvtt5k5c6aeWSjVCnuTk/f/vJXtnxYz4oIUbnhsYpcmCujFZxa+kJWVxS9+8Qsuv/xy3G43FouFv/71r0RERJCXl8fEiROJiYnhzTffxOVycfvtt1NTU4Mxhu9+97vExsaecd0vv/wy3/jGN2hsbGTgwIG8+OKL5x3v3/72N+6++26ampqYM2cOc+bMATzdd3NycnjqqacYOXIkN910E1lZWQQFBfHXv/6VwMDAVpe/7777uOOOOxg8eDDx8fEsXLjwxDZnzJjBrl27qK+vJzU1lRdeeIErrrjivPdFqZ7IGEP+mhJyPtpPY7Wdy+8fyZDsPj6JRc6xJl+3l52dbU4d/Gjnzp2MGDHCRxGdWWRk5BnPGlTLuuv/pVIdxdrgYNVru9m3uZyImGAuvWMEA0YldPp2RSTXGJN9arueWSilVDdSV2llw+J97NlUBsD4y9OZtmBQl9zEbo0mi25AzyqUUtYGB4d2VvLpq7tw2FwMmpDEqIv6kzq8e3T+0GShlFI+5HK62bzsAJuXHsDpcBMeE8yC740neUC0r0M7iSYLpZTykYqSej59dRdlRbXE9Q1nxs1DSRkUQ1BwoK9DO40mC6WU6mJOh4tP/72LPRvLCAoO4JLbhpE1vV+37kKuyUIppbpQfZWVD57dRsXhBoZO7sOU+QM7rZ5TR9KH8nqYnlSi/Cc/+QlpaWlERkZ2WoxK9SRHD9ax8OlNVJY2ctk9Wcy+d2SPSBSgyaLH6Uklyq+55ho2bdrUJbEq1d0Vbj3Ku7/fTECgcN0PJjBsSl9fh3RWNFl0sd5Sohxg6tSppKSknMuPSSm/4XK62bikkI+e+5LIuBCu+8FE+g6M8XVYZ6333rP46HE48mXHrrPvaJjzzBkn96YS5UopqC5vZNk/dnDsUD3Dpvbl4luHYemGPZ3ao/cmCx/oTSXKlerNjNuw6YMitq44CAHCnK+PZuD4JF+HdV56b7Jo5Qygs/SmEuVK9VZul5uPntvB/u3HyBidwNQFg0jo3/M7eeg9iy7Um0qUK9UbHcyv4LWfbWD/9mOMn53O3G+O9YtEAb35zMIHeluJ8h/96Ee8/vrrNDY2kpqayv3338+TTz553nEp1d24nG7WvbOX7Z8WE5UQyqy7RjBsas/q7dQWLVHeDWiJ8rPXXf8vVe9Te6yJj1/Kp3RvDcOn9mXGLUMJDu25n8O1RLlSSnWgipJ68j4vIW/NYQJETpxN+GsHD00W3YCeVSjVc9RXWcldeoAdn3m6iA/JTmbinAy/uTdxJposlFKqHZx2Fzkf7Sf3owMgMGRSH6ZdO4io+FBfh9YlNFkopVQbDuZX8Omru6ivsjEkO5nsuZnEp0T4OqwupclCKaXOwG51suq13RR8UUZc33CueWQs6VmdPw52d6TJQimlTmFrdJC3uoQvPyumvsrG6EtTueC6QQRZemapjo6gD+X1MN2tRHlFRQWXXnopkZGRLRY4VKoncbvcbFt5iNd+toH17+4jJMzCdT+YyEU3D+3ViQL0zKLHmTdvHvPmzeuSbR0vUT516lSuuuoqli5deuLBvONCQ0N5+umn2bFjBzt27OiSuJTqaIf3VLFrwxFK9lRRe8xK8oAornxwNP2GxPo6tG5Dzyy6mL+VKI+IiGD69OmEhvaOHiHKfzTV2fn8jd28/MRaFv9hC3tzy4lODGP2fVnc8Fi2JopT9Nozi//d9L/sqtzVoescHj+cxyY/dsbp/liiXKmextboYPfGI2z6oAhbg5PkjGhGzujH6EvTCAnrtYfENulPpgv5Y4lypXoKt8vNpveL2LryEC6Hm8i4EOY+PJaUQT1vICJf8EmyEJHvAvcDBvgSuAcIB94EMoD9wE3GmCrv/E8A9wEu4BFjzLLT13p2WjsD6Cz+WKJcqZ6guqyRj1/Kp6yolrQRcYybnU7q8HgCAvQDUHt1+T0LEekPPAJkG2NGAYHALcDjwEpjzBBgpfc9IpLlnT4SuBL4PxHpkd0S/LFEuVLd3e6NR3jjqY0cPVDHpXcM55pHxpGelaCJ4iz56jJUEBAmIg48ZxQlwBPAJd7pLwOrgMeA+cBCY4wNKBKRvcBkYH0Xx3ze/LFEOUBGRga1tbXY7XYWL17M8uXLycrKOu/tK3U+bI0OvvjvfratPERC/wiueGAUcX1711PXHcknJcpF5DvAL4EmYLkx5jYRqTbGxDabp8oYEycizwIbjDH/9ra/AHxkjHm7hfU+CDwIkJ6ePvHAgQMnTe+uZa21RPnZ667/l8r33C43uzceYe07e7E1OMkYncDs+0b26LLhXanblCgXkTg8ZwuZQDXwHxG5vbVFWmhrMcMZY54HngfPeBbnF6lSqiexNjhY985e9m0ux251kdA/gvnfGU9SepSvQ/MLvki1lwFFxpijACKyCLgAKBORFGNMqYikAOXe+YuBtGbLp+K5bOU39KxCqXPndLjYuuIgOR8ewOV0kzEmkcETkhg0MbnXP3XdkXyRLA4CU0UkHM9lqFlADtAA3AU84/3+nnf+JcDrIvIHoB8wBNjU1UErpbqfkr3VfPb6bipLGkjoH8mldwynT0a0r8PyS12eLIwxG0XkbWAz4AS24Ll0FAm8JSL34UkoN3rnzxORt4B87/zfNMa4ujpupVT3kr+2hE9f3UV4TDBXPDCKQROS9FmgTuSTOz7GmJ8BPzul2YbnLKOl+X+J54a4UqqXq6+y8vFLOzm8u4rkjGjmPzpOb153Af0JK6V6BLvVya71R9jw3j6MgSnzBzL6klRNFF1ECwn2MN2tRPmKFSuYOHEio0ePZuLEiXzyyScnpl1yySUMGzaMcePGMW7cuBMPIyp1tgpyynj1f9az+s09xCaHc8OPJpI9J0NrOXUh/Un3MN2tRHliYiLvv/8+/fr1Y8eOHVxxxRUnFRx87bXXyM4+rcu2Uu3SVG9n1Wu7KdxylPDoYK7+9ljSs+L13oQP6JlFF/O3EuXjx48/UTNq5MiRWK1WbDbbufxolDpJ/toSFj61icItRxkzM5U7f3kBA0YmaKLwkV57ZnHkV7/CtrNjS5SHjBhO3x//+IzT/b1E+TvvvMP48eNPFC4EuOeeewgMDOT666/nf/7nf/QPXbXJ7XKz9p29bP+kmOjEUOY/Oo7U4fG+DqvX67XJwhf8uUR5Xl4ejz32GMuXLz/R9tprr9G/f3/q6uq4/vrrefXVV7nzzjvPuA7VuxljOJhfyZq3Cqgua2T0palceP1gAoP0Akh30GuTRWtnAJ3FX0uUFxcXc+211/LKK68waNCgE+39+/cHICoqiltvvZVNmzZpslAtsjY4+PBv2yndW0N4dDBzvjGazLGJeibajWjK7kL+WKK8urqauXPn8utf/5oLL7zwRLvT6eTYsWMAOBwOPvjgA0aNGnX2PzTl14wx7P/yGG88tZHSvTVMviaT238xjYHj9AG77qbXnln4gj+WKH/22WfZu3cvTz/99IkzpOXLlxMREcEVV1yBw+HA5XJx2WWX8cADD5x3TMp/VJc1svqtPRzMqyQsysK874wjbYTem+iuzqpEuYgEAJHGmNrOC6ljZGdnm5ycnJPaumtZay1Rfva66/+lap+SvdUsfe5LbE1Oxl2WzuS5mQRa9EJHd3DOJcpF5HXgG3iGNM0FYkTkD8aY33Z8mEopf1ZztIl1i/ZSuOUokfEh3PL9yTogUQ/RnstQWcaYWhG5DfgQz+h1uYAmiw6iZxWqNzi8u4qPX8rHWu9g9MX9mXhVBhExIW0vqLqF9iQLi4hYgAXAs8YYh954Ukq1l9tt2PjePjYvO0h4dDDX/mACyQO0jHhP055k8RywH9gGfC4iA4CazgxKKdXzuVxudq0rJefD/dRX2Rg4PolLbxtOaKTF16Gpc9CeZPG+MebPx9+IyEHg3s4LSSnV0+WtPszGJYU01TlITIsk+6oMsqb30+6wPVh7ksU7wITjb4wxRkQWAhM7LSqlVI9UX2Vly4qDbP+kmNg+4cy4aSiDJiYTEKBJoqc7Y7IQkeHASDy9n65rNikaCO3swFTLlixZQn5+Po8//jhPPvkkkZGR/OAHP+iUbeXm5p54zuKqq67iT3/602mfDPfv38+IESMYNmwYAFOnTuXvf/97p8Sjuq+jB+v4fOEejhR6rlAPndKHS28frmNg+5HWziyGAVcDscA1zdrrAH26yke6W4lygEGDBrVaZ0r5L2MMuR/tZ+OSIkIjLYydmcbg7GT6ZEbrJSc/c8anYIwx7xlj7gGuNsbc0+zrEWPMui6M0a/4W4ly1Xu5XW5W/CufjUuKGDA6gZt/MpnpNw2h78AYTRR+qD33LCpEZCXQxxgzSkTGAPOMMb/o5Ng61eq39nDsUMc+35CYFsmMm4aecbq/ligvKipi/PjxREdH84tf/IIZM2a0/oNSPd7Rg3V8+u9dHD1Yx4QrBzB1/kBNEH6uPcniH8AP8XShxRiz3ftUd49OFr7gjyXKU1JSOHjwIAkJCeTm5rJgwQLy8vKIjtZ+9P6ouryRL/5bxJ6NZQRZArj4a0MZdXFq2wuqHq89ySLcGLPplAOFs5Pi6TKtnQF0Fn8sUR4SEnJi/RMnTmTQoEHs2bNHh1L1M421djYuKWTXulICgoTRl6YyfnY6UfHa16W3aE+yOCYigwADICI3AKWdGpWfmjVrFvPnz+e73/0uycnJVFZWUldXd6JE+S233NJiifLp06fz+uuvt6tE+YwZM86qRPmZxMbGnihRPmXKFF555RW+/e1vnzbf0aNHiY+PJzAwkMLCQgoKChg4cOBZ/VxU91a49Sifv7Gbhho7Iy/qz6S5WqajN2pPsvgm8DwwXEQOA0XAbZ0alZ/yxxLln3/+OT/96U8JCgoiMDCQv//978THa5lpf1BxuJ7PF+6hpKCahP6RXPXwGC3T0Yu1WqJcRAKBZ4wxPxSRCCDAGFPXZdGdBy1R7t+66/+lP7BbnexcV8rGJYUEBAgTrhjA2FlpOrxpL3FOJcqNMS4Rmeh93dBZwSmluoeSgiqWv5BPQ7WN+H4RXP2tsXpfQgHtuwy1RUSWAP8BTiQMY8yiTouql9GzCuVrLqebde/s5ctVxUTEhjDn694xsLVMh/JqT7KIByqAmc3aDNAjk4UxRvuD93BnM7qjapvd6mTZP3ZwMK+S4RekMP2GwYSEa2VYdbI2k4X3KW6/EBoaSkVFBQkJCZoweihjDBUVFYSG6qWRjtBUZ2fJn7dSUVzPpbcPJ2v66d2jlYL2Dav65xaaa4AcY8x7HR9S5zn+7MDxchqqZwoNDT3p6XJ1bg7lV7Lq9V001NiZ+82xDBiV4OuQVDfWnstQocBwPPcsAK4H8oD7RORSY8yjnRRbh7NYLGRmZvo6DKV8qupIA6vfKuBQfiVhURau+sZo0kdqolCta0+yGAzMNMY4AUTkb8ByYDbw5blsVERigX8Co/Dc/7gX2A28CWTgGZnvJmNMlXf+J4D7ABfwiDFm2blsV6nezBjDhvcK2bzsAEHBgYydmcakazIJCWvPYUD1du35LekPRPDVUKoRQD9vt1rbOW73T8BSY8wNIhIMhAM/BlYaY54RkceBx4HHRCQLuAXP2Br9gI9FZKgxxnWO21aqV9q87ACblx4gfWQ8M+8YQUSsPoWt2q89yeI3wFYRWQUIcBHwK+9Deh+f7QZFJNq7jrsBjDF2wC4i84FLvLO9DKwCHgPmAwuNMTagSET2ApOB9We7baV6o7KiWjYu2cehnVVkjEnkqodGawcPddba0xvqBRH5EM8BWoAfG2NKvJN/eA7bHAgcBV4UkbFALvAdPCXQS73bLBWRZO/8/YENzZYv9radRkQeBB4ESE9PP4fQlPIfNUcb2bC4kL255VhCApkyfyBjZ6ZpolDnpD29oQSYBQw0xjwlIukiMtkYs+k8tjkB+LYxZqOI/AnPJaczhtBCW4sd7Y0xz+OpY0V2drZ2xle9kttt+OKDIjYvO0BAoJA9N4Pxl6UTrPcm1Hloz2/P/wFuPA/lPYVnWNV3gEnnuM1ioNgYs9H7/m08yaJMRFK8ZxUpQHmz+dOaLZ8KlKCUOo3D5mLly/ns23yUIdnJXHjjEK0QqzpEeyqDTTHGfBOwAnh7KAWf6waNMUeAQyIyzNs0C8gHlgB3edvuAo4/w7EEuEVEQkQkExgCnOtZjVJ+y9rg4INnt7Fv81GmLhjI5feP0kShOkx7ziwc3uqzx8ezSMJzpnE+vg285u0JVQjcgydxvSUi9wEHgRsBjDF5IvIWnoTiBL6pPaGUOllBThmr39xDU72D6TcOYeystLYXUuostCdZ/Bl4F0gWkV8CNwD/cz4bNcZsBVoaSm3WGeb/JfDL89mmUv7I5XCz8f1Ctiw/SGJaJFd/a6yOOaE6RavJQkQC8Ax29CM8B3IBFhhjdnZBbEqpMzBuw5efHWbLigPUV9oYOqUPl94+nCBLoK9DU36qrfEs3CLye2PMNGBXF8WklDoDt9tQvr+WDYv3cXhPNckZ0Vx0yzAyxyT6OjTl59pzGWq5iFwPLDJaG1opn6koqWfpczuoLmskJDyIyddkkn1Vhj43obpEe5LF9/CU+HCKiBXPpShjjNELo0p1Abfb8PnCPexaV4olNJALbxjM4Il9iIzTnk6q67TnCe6orghEKXU6a72DFS/mczCvgoHjkph+0xAd5lT5hD7SqVQ3ZLc6Wf1WAQVflOFyuBl/eTrTrh2kl5yUz2iyUKqb2be5nHXv7qP2aBPDpvRl/OXpJPSP9HVYqgcwLhe43RAU1OEfLDRZKNVN2K1OPn11F3tzy4lJCmP+d8eTOizO12Gpbs5+8CD1q1dj3ZFH7dKlmKYmhm3bioR07D2tdiULEZkODDHGvOh9gjvSGFPUoZEo1Qu53YaDOyrYs+kIhwuqaayxM+ri/lx4w2B9ZkKdkXG7qV+1ivpPV1H9zjvgdiMWCxEXX0TYyJFIYMf/7rSn6uzP8DxtPQx4EbAA/wYu7PBolOol6qts5C7dz74tR2mqtRNoCSBjVAIjZ/QnLSve1+Gpbsg4HNj27cO6Ywc1S96ncZOnRF7U7MtI/v73Cerbl4DQzuv80J4zi2uB8cBmAGNMiYhoDymlzoFxG/ZsOsLqtwqwNTlJz0pg6OQ+ZI5NJDhUrwqr09n376fqzbeoWbwYV1UVAAExMSR9/3vE33knAR18uelM2vPbaTfGGBE5XkgwopNjUsovOWwulv9zB/u/rCAxLZL5d44gKU0/d6nTGWOwFRRw7P/+Rt3SpSBCxLSpRMy4iLBxYwkbM6ZTLjW1pj3J4i0ReQ6IFZEHgHuBf3RuWEr5l8N7qlj+Qh5NtXamXTuIsZelERjYnhECVG/hqqmh7uOVNKxdS/2aNbhrawGIvmoOiQ89RMiQIT6Nrz0P5f1ORGYDtXjuW/zUGLOi0yNTyg+U7a/l84V7OHqwjuiEUC779ji9J6FOsO7eQ+1HH9Kwdh3WXbvA4UDCw4mcPp2wceOImn0ZwWndo9x8e25wfxf4jyYIpc5OzodFfPHBfkKjLIyfncaEKwYQEm7xdVjKx2yFRTR+8QV1y5bSsG49iBAyeBAx8+cRc808wieMRyzd7/ekPZehooFlIlIJLATeNsaUdW5YSvVcxhhWL9zDl58dZtD4JC65fTihEd3vj191DeNy0bB+A40bN1C/6jNsBQUASHg48ffeS/xdd2Hpk+zjKNvWnstQPwd+LiJjgJuBz0Sk2BhzWadHp1QPU19lZeXLOyneVUXm2EQuv38kAXpvotcwLhf2/ftpWLMG+6Fi7AcO0LR5M+6GBgBChg+nz4+fIHTkSEJHjyYg+JxHqO5yZ9NXrxw4AlQA3T8NKtXFjhXX89Hft9NQY2fKvEwmXpmBBGgtJ39ljMFeVETD6tVYd+/BUVpC07btmMZGAAKiogjqk0zU5ZcTPmUykdOnE5TYc8cdac89i4fwnFEkAW8DDxhj8js7MKV6CrvVyaYPiti28hABgcK8R8bRf6iW6fBXtoICqt58i9qPPsJVUQFAYEwMltRUombNInzyJMLGjiV06FAfR9qx2nNmMQB41DtutlKqmUM7K/n4xXwaa+2Muqg/469IJzohzNdhqQ5mXC4a1qyh/E9/wpa/E0QInzKFqAcfIHLmTCypqX5fEfiMyUJEoo0xtcBvvO9P6u9njKns5NiU6raMMWxbeYi17+wlJCyIud8cQ8bonnuJQZ3OGEPD559Tu3w59as+w1VRQWBcHMk/+hHRV83B0revr0PsUq2dWbwOXA3kAgbPCHnHGWBgJ8alVLdkjOFgfiW5H+2ndG+N50nsR8drbyc/4bZaqV+1isZNm6j75FOcR45AUBBRl15C5KxZRF12GYGRvbNc/BmThTHmau/3zK4LR6nuybgNW1YcZNsnh2issRMSHsSYmalMv2GI3sT2A26rlcqXXqbixRdx19QAEJ6dTcL99xMz7xoCo3UU6fbc4F5pjJnVVptS/sja4CB/TQlbVx6iqdZO34HRTLh8AMMvSCEkTAv/9XTWPXuoW76C6jffxHn0KKEjRxL/0/9H5PTpBMbE+Dq8bqW1exahQDiQKCJxfHUZKhro1wWxKeUzTfV2tq44yLaVxbicbhLTIrnwukEMndLX729k+jNnVRWNm76gMSeHxg3rsRXsBSBk6FCSH3+MmLlzfRxh99XaR6OvA4/iSQy5fJUsaoG/dm5YSvmG2+Vm57pS1i/eh63BScaYRMbPTqPfEO0K25M5ysqoWbSIo3959sSwoyGDB5P0/e8RfeWV3ab+UnfW2j2LPwF/EpFvG2P+0oUxKeUThVuOsn7xPqrLGklKj2L6NwZrkuih7MWHqVm0iKatW2ncsgXT1ARA6KhRJH37W4RPndpl40D4i/aU+/iLiIwCsoDQZu2vdGZgSnUVp91Fzof7yV12gOiEUOZ8fTSZ4xL1clMPYozB3dBA/WefUf3Wf2jMyQGXi6B+KUTPmUPIkCGEDB5ExJQpSA8qsdGdtHdY1UvwJIsPgTnAGkCTherRnHYXuzceIXfpAeoqrAyf2peLbh2GJVjHvu4pbEVF1CxaRO3y5TgOHAQgIDycuFtuIe722wjJ1M6cHaU93TluAMYCW4wx94hIH+CfnRuWUp2rrtLK0ue+pPxAHWFRFq55ZCzpWQm+Dku1wbjdWHfswJqXR+1HS0+MQx0ybBiJDz+EJT2d6Cuv7NSxqHur9iSLJmOMW0ScIhKNp6CgPpCneqyGahtvP5OD3eZi9r1ZDBqfTKBFK8N2Z8bhoP6zz6h46SWacnIBCEpJIe7Wr5Fw//1Y+mkHzc7WnmSRIyKxeIZSzQXqgU3nu2ERCQRygMPGmKu95UTeBDKA/cBNxpgq77xPAPcBLuARY8yy892+6p0aamws+l0ujbV2LdHRjTmOHKExN5fGjZs8ZxL5ntqlQUlJ9Pnxj4mYNpXggQO7fBzq3qw9N7gf9r78u4gsBaKNMds7YNvfAXbieW4D4HFgpTHmGRF53Pv+MRHJAm4BRuLpxvuxiAw1xrg6IAbVS7hdbvLXlrLp/UJsjU7mfH20JopuxllVRfWbb9Gwdi2NmzeDy4WEhxM6fDjxd99N2JjRRF52WY8aA8KftPZQ3oTWphljNp/rRkUkFZgL/BL4nrd5Pp4b6QAvA6uAx7ztC40xNqBIRPYCk4H157p91bvUVVr55BXPgESJaZHM+fpoUgbH+jqsXs9efJiGNatp3LQJ+/4DX509JCeTcO+9RF58kWeAIO3i2i20dmbx+1amGWDmeWz3j8CPgKhmbX2MMaUAxphSETk+wFJ/YEOz+Yq9bacRkQeBBwHS09PPIzzlL4p3V/HpqzupPWZl2nWDGD87XbvE+oBxu2nctInGnFyaNudi27sPZ3k5AAGRkYQMH0b8XXcROWsmEZMn+zha1ZLWHsq7tDM2KCJXA+XGmFwRuaQ9i7TQZlqa0RjzPPA8QHZ2dovzqN7B5XSz5q0Cdnx+mIBAYc7XRzNwfJKvw+pVjDFYv/ySuuXLqV26DEdxMQBBffsSNnECIUOGEH355Z57DwHawaC7a89zFne21H4eD+VdCMwTkavwPOQXLSL/BspEJMV7VpGCp9cVeM4kmj+LnwqUnOO2VS9QUlDFypc9ZxOjL0ll8tWZhEZqCfGu4rZaqXr9DSr++U9clZUQGEjYqFHE3303ERdMI2SgdqbsidrTG2pSs9ehwCxgM+f4UJ4x5gngCQDvmcUPjDG3i8hvgbuAZ7zf3/MusgR4XUT+gOcG9xA6oDeW8k95qw/z+Rt7CA4L4vL7RjJkUh9fh9QrOI8e5eizf6Vp82ZsBQUABA8aRNKj3yH68ssJjI31bYDqvLWnN9S3m78XkRjg1U6I5RngLRG5DzgI3Ojdfp6IvAXkA07gm9oTSp3KGMMnr+5i17pSkjOiuebbY3VAok5kjMFetB9r3g5qly6j/pNPwBgiLphGxIwZREybSsT06Xp/yI+IMWd3aV9ELMB2Y8yIzgmpY2RnZ5ucnBxfh6G6wNGDdeQuPcC+zeWMnZnGtGsH6UN2ncRWUEDl66/TsG7difIaEhJC7PXXETN/PmFjx/o4QnW+RCTXGJN9ant77lm8z1c3lAPw1Ih6q2PDU+rsOewuVryQR9G2YwRaAhg/O51p1w3ST7OdwJqfz5Ff/oqmXM/T0+GTJhF3002ET5xIyNChBISH+zhC1dnac8/id81eO4EDxpjiTopHqXYpKahm1eu7qSptIHtuBuNmpRESrpedOlLT1q3ULHmfuk8+8YxFbbGQ8MD9xN16K5aUFF+Hp7pYe+5ZfAbgrQsV5H0db4yp7OTYlGpRQU4ZH7+YT3BoELPvy2LopL6+DslvuGprqVu+nOr/vE3Ttm0gQuSllxJ+x+3ELFhAUIIWW+yt2nMZ6kHgaaAJcON57sGgxQRVFzNuw9q397Ltk0PE9Q1nwfcmEB6tpR86gq2wkKbt2yn71a9x19YSGBtL4je/SfxddxIYHd32CpTfa89lqB8CI40xxzo7GKXOpPxALWveKqB0Xw1ZF6Zw4Y1DCA5tz6+vao27oYGy3/2O6jcWAp4H5vr/7rdETJuGWPSynvpKe/7a9gGNnR2IUi1xOlysXriH/LWlBFoCmHR1JpPmZuhN7PPgqqmhbuUnVL3xBta8PHC7iZk/j9ibbyF0xHACwsJ8HaLqhtqTLJ4A1onIRsB2vNEY80inRaUUUF9l44Nnt1JxuIHRF/dnyvyBehP7HBljsO3aRe3SZVT84x/gdhMyZDBxt95K1GWzCJ8yRROwalV7ksVzwCfAl3juWSjV6Rpr7Sz50xZqj1mZ843RDByndZ3OhTGG6jffpObdxZ4b1kDEjBkk3Hcf4ZOydTwI1W7tSRZOY8z32p5NqfNnjGH9u/vY/kkxxhiu/uZY0rLifR1Wj2PduZPqt9+h9qOPcFVWEjxwIMk/+hFRl83CkpamZxHqrLUnWXzq7RH1PidfhtKus6pDFe+qZNXru6kpbyJlcAwXXj+EPpnaE+ds2AoKKPvNb2lYvRqAiIsvInrOHGLmz9cEoc5Le5LFrd7vTzRr066zqsNYGxysX7yP/DUlREQHc+kdwxkxLQUJ0INbe7itVho2bKDmnUXUrVgBQNztt5Nw7z06NrXqMO15KC+zKwJRvY8xhr255ax5q4CmegdjZ6Yx+epMgsO0S2xbXPUN1H/6CY2bvqDmgw8wTU1IcDAxCxYQf889hA4b6usQlZ/xxXgWSrFrfSlbPz5ExeF6ImKCue4HE+g7MMbXYXVrjrIybLt20bBpE1Wvv4FpagIRwsaOJe7WrxF12WVao0l1mi4fz0L1bvYmJ2vf2Uv+mhKiE0OZumAg42enExCoVWJbYlwu7AcOUPHPF6hZtMjTKELERTNIuPtuwrOz9eE51SW603gWys8d2FHBp6/upKHWzvjL05kybyCBQZokTmWcTmx79nD02b96xokACAoi9sYbiZk/D0taGpY+OqiT6lrncnG4Ec9odUq1i9vlZuOSIjYvO0B0YijX/WAiKYP0klNz7sZGbHv3Uv2ft6n98EPcDQ1IcDBxt99OcFoqkTNnEpyW1vaKlOokOp6F6jROh4u9OeVsXn6QqtIG0kcmMOfrowgK1gfBjmv6cgflv/0tjbm54HJ5qrzOmknkxRcTdcklBCXpw4iqe9DxLFSnqC5v5MO/fUlVaQMxSWHMvHM4Iy7QbpzgGSeibuVKGnM307R1KxIaSvyddxI6fBjhkyfrWBGqWzpjshCRwUCf4+NZNGufISIhxph9nR6d6nGMMeStLmHDe/vAwOX3j2TQhGQCevEzE8blwrZnD3Ufr6T6nXc8AwkBoWPHEH/H7cTfex+WPsk+jlKp1rV2ZvFH4McttDd5p13TCfGoHsq4DXu+KGPTB0XUHm0itk84c74xmviUCF+H5jO2wkIqX36FumXLcFVXAxAyYgTxd9xOzLXXEhSvZUxUz9Fassgwxmw/tdEYkyMiGZ0XkuppqssaWfbPHRw7VE9iWmSvfgLbVV1N9aJ3qVmyBNuuXQCET5tK1OzZRF5wAcEZGb4NUKlz1FqyCG1lmha8VwAUbT/Gx//KIyAogNn3ZjEku0+vSxLGbqd+7VpqFr1L/apVGIeDkKwRJH7rW8QsWEBwan9fh6jUeWstWXwhIg8YY/7RvFFE7gNyOzcs1d0ZY9i4pJDcjzzdYa/+1lji+vaeS06OsjJq3n2X+jVrsO3chbuhgcCYGCIvuYTYW24m4oILtHCf8iutJYtHgXdF5Da+Sg7ZQDBwbSfHpbox4zZ8+toudq4tZdD4JGbeNaLXDHFq27uX8t//gfpVq8AYLOnpRF15BZEzLiLyohlabkP5rTP+hRtjyoALRORSYJS3+b/GmE+6JDLVLTnsLpb/M4/9248xdlYaF14/2O8vO7ntdupWrKD2/Q+oX7UKCQkh4YEHiLzkYsLGjUMC9Cl05f/aU+7jU+DTLohFdXNHD9ax/IU8qssamXxNJtlX+e9Y2I4jR2jauo3aDz+kfs0aTGMjgbGxxN97L3Ffu0Wfpla9Tu+4dqDO25blB1m/eB+WkEBm3T2C4VP988Exe/Fh6j9ZSflvf4dxOACImj2bqMtnE33FFUhwsI8jVMo3NFmoVjVU29jw3j52rT9C2og4Zt6ZRWRciK/D6lCOsnJqP/oQa14+te+/D0BI1giSv/99QocNIygx0ccRKuV7mixUi4zbkLemhPWL9mK3uRg/21sl1uIf1+eNy0X9559T/9ln1Cx+D2O1EhARQeyNNxA9dy7hEydq6W/V7bmNG6vTSq29lsP1hzlUd4hqazV3jbyrwy8Ra7JQp3G53Hz8r3z25paTMjiGGTcPJSktytdhnTe3zYZtTwENG9ZT+eJLuCo9w8hHXHghSd/7LqFZWX57D0b1XMYYau21HKo7REVTBRXWCtaXrGdL+RaONh3FbdynLXPTsJsIt3RszzxNFuokdZVWPnt9Nwd2VDBpbgaT5mb2+N5O9atXU/GPf9K4adOJtuCBA+nzxONEzZxJQETveT5EdR9Wp5WShhLKG8uxu+w4XA4qrBVUNFVQaa3kk4OfUGOvwe6yY04U/vYIDQxlcspkroi+gsSwRMKDwkkOTyYzJpPk8GTCgjr+uekuTxYikoZnlL2+gBt43hjzJxGJB94EMoD9wE3GmCrvMk8A9wEu4BFjzLKujrs3OLCjgo/+/iUGw/QbhzB2Vs/t8WOMoW75CipfeommLVsITEgg7o47CBs1kuCMDEJHj9Yur6rLGGMoqi2iwd7AgboD/GP7PyisKTzj/JYAC0PjhnLVwKsIDgwmyhJFcngy6dHpRAVH0S+iH5bArr1M6oszCyfwfWPMZhGJAnJFZAVwN7DSGPOMiDwOPA48JiJZwC3ASKAf8LGIDDXGuHwQu9/at7mc5f/KIyo+lLkPj+nRT2NXL15M5Qv/wlZQQGBsLIkPP0zC/ffpA3Oq0zjdTo41HeNQ3SGONBzhWNMxCmsKKakv4UjDEcoay7C5bCfmjwqO4p6R95AalUpqVCqRlkiCA4MJCwojNTKVwIDuN+ZLlycLY0wpUOp9XSciO4H+wHzgEu9sLwOrgMe87QuNMTagSET2ApOB9V0buX+yW52seauAnetKie8XwRUPjOpxicIYg6u6mqatWzn6579g27kTS2oqfZ/8GbHXXafdXVWHcrldFNUUUVBdwPaj28kpy6GopuikZAAQHBDMiIQRDIwZyJSUKQyIHkBmTCbhQeFkJWR1+D2FzubTexbe6rXjgY14xs44nkRKReR4gf/+wIZmixV721pa34PAgwDp6emdFLX/KNlbzWev76aypIFRF/dn+o1DesyY2MbppOKFf9GwZg22wkJcFRUABMbFkfzDHxJ/911IYPf7dKZ6FpfbxbGmY1hdVorrillXso7/7PkPTc4mAARhRMIIrsy4kmHxw0iNTKVvRF9So1KJsEQQID3j76k9fJYsRCQSeAd41BhT20ovlJYmmBbaMMY8DzwPkJ2d3eI8Chpr7eT8t4i81SWERFq45pGxpGcl+DqsVhljaNq6lcacHOx791Hz3/+C00lgYiIR06YRmpVF8MBMwseNIzA21tfhqh7C6rRyrOkYNpeNQ3WHqLPXUVRTRElDCTsrdlLaUHoiMRw3MmEk8wbNIyMmgwnJEwgNaq1At//wSbIQEQueRPGaMWaRt7lMRFK8ZxUpQLm3vRhofqc1FSjpumj9R12llS3LD7J74xGcNheDJiRxyW3DCQ7rvp3ijMtFw5o1VP77NRpWrwYgIDqasNGjib/jdqKuvFJvVKszqrJWUWOroaShhNyyXNaXrMfhduAyLhrsDZQ2lJ7W0wggNiSWgTEDGZkwkkGxg0gKTyIuJI6hcUPpE9HHB3vie77oDSXAC8BOY8wfmk1aAtwFPOP9/l6z9tdF5A94bnAPATahzkrh1qMs+8cO3G5DelYCk+Zm0HdgjK/DOiNHaSl1y5dT8cK/cJaXExAeTsIDDxB3261Y+vb1dXiqG3K4HawvWU9uWS7rStZxsPYgjc7Gk+YZGDOQ9Oh0AgggKCCIuQPnkhaVRkhgCNEh0aRHpRNhiSAhrHufafuCLz5SXgjcAXwpIlu9bT/GkyTe8o6XcRC4EcAYkycibwH5eHpSfVN7QrXf8eFOP3tjN/H9IrjywVHEJHXfG2vOigqOPPU0dcs8vaNDhg0j6dFHibr8cgIje9aNd9Xx7C471bZq8o7lcbTp6ImeR3X2OjYe2UidvQ7wJIV5g+YRFxpHSkQKfcL7kBadRlpUz+0O7mtijH9e2s/OzjY5OTm+DsOn6qtsfPxSHod3VxPfL4KrHhpDTFL3HOTQVV1N5euvU/Xqv3E3NBD3ta8RfdUcfR6il6qyVrH96HYKawppdDZS3lhOSX0J245uO+keQoAEEBMcQ2RwJJkxmVyWfhmzBswiyhKlT+OfIxHJNcZkn9refS9Wq/NSXd7Iot9txtHkZPI1mYyfnU5QcPfrHeQoKaHst7+l7uOV4HAQNm4cfX78BGFjxvg6NNUJjDE0OhvZdnQbdfY67C47dped8sZy8ivyqbRVUtlUSXF98UnLhQeFkxmTyaz0WYxKHEX/yP4Mjx9OUlhSt3wmwR9psvBDFSX1fPh/23E73Vz/WDaJqZG+Dukk7oYG6levpnHzZqrfWIhxOgmfNImkRx8lfMJ4X4enOoAxhpKGEkrrSyltKGV/7X42l21md+Vu6hx1LS4THxrPwJiBDIsfxnVDriM9Op2pKVOJCo7yqy6oPZUmCz+ze+MRPn4xn+CwIK7+5phukyiM203T1q3Ur/qMqjffxF1TA0DkrFkkf/97hAwc6OMI1bmyu+zkV+RTbaumsKaQwupC1hxeQ4W14qT5+kf2Z2b6TDJjMhkQPYAB0QMIDgwmJDCECEsEUcE9v1ilP9Nk4SeOFNWQ89/9HNhRQXJGNJffl+XzG9nGbsd+4AAVL79M3dJluOvrAQjNyiLxf58h8oIL9OnqHsrusrN472Jyy3JZW7KWGlvNiWnBAcGMTx7PrSm3MjJhJAlhCaREpBAT0n1736m2abLo4VxON+sX72Pbx4cICBImXDmAiVcM8MmzE8YYnCUlWHfvpuK552natu3EtIgLphF99TVEXHghlj7JraxFdSdWp5VqWzWlDaVUWis5WHuQPVV7WFq0FKdxEh8az5DYISwYvIDMmEz6RfYjMUwHi/JHmix6MFuTk+X/2MHB/EoGjU9i+k1DiIzruqdJjd1OY04O9Z+v9jxZXVSEu6EBgMCkROLvvRdLan8ipkwhZNCgLotLnZ1aey21tlpsLhuH6w+zt3ovx5qOsaVsCzsqdpw2f3BAMBenXcyVmVdyxYArtNdRL6HJogcyxvDlqmJyPjqAtc7OJbcNY+SMFstldShXTQ2NW7Zgzc+nYfUamnbsAIcDRAgdMYLoq68meMAAQkcMJ2zCBAJC/Gv41Z6u1l7LtvJtbCnfQmlDKXX2uhPJ4VSWAAsDogdw09CbGBY/jL4RfYkJiSE9Kp3YkFhNEL2QJosepqHGxmev76Zo2zGSM6KZ+9AY+mRGd8q2jNuNq7qahtWrqf/sM2pXfOxJDoClXz/ibryBkKHDiL56LoGR3eNGem9mjMHqsmJ1WimqKeLLY19yuP4wxXXFVNuq+fLYl4Cn+F3fiL7EhsSSGJbI7AGzSYlIITQolAhLBOOSx+lzCuo0mix6iPoqKzkfHWBvThm2RicTrkhn6oJBnfIH7Sgt5dhzz1G3bDmuqioAAmNjibrkYmIWLCBszBiCkpI6fLvq7ByviFpQXcDHBz7m00OfUmmtPGmeIAliSNwQooKjeGD0A4xLHkd2n+weVx5b+Z4mi26u5mgj6xfto3DbMQToNzSWKfMGdkpdp8bcXI49/zwNn30OQPikSUTOnEnI4EFEXHCBlvzuIi63i/LG8hNVUIvri2lyNtHobKS0vpTyxnKK64upslbhcHvO9EIDQxmTNIZJfScRHRxNWFAY0/pN04fWVIfRZNGNVZTU8+7vN+NyuBl1cX9GX9y/UwYmavziC47+5VkaN21CwsOJu/VW4r52CyFDhnT4tpSHMYaDdQcpqCpgX/U+Nh7ZyL7qfdTaanEaZ4vLBEgAKREpJIYlMiF5AikRKfSL7Ee/yH6MSx5HdHDnXI5UCjRZdEt2q5MNiwvJW32Y0EgLNzyWTWxyx142ME4ndSs/oeKFF7Bu305gYiIJDzxAwv33ERij/eHPR729nnpHPceajrGrchcHaw9yrOkYDreDWnst+2v2U2WrOqnGUf/I/mT3ySY9Ov3EQ2qZMZlEB0eTHJ6sZwjK5zRZdDN2q5N3f7+ZY4fqGTKpD5OvzuywRGHsduo+XUXdsqU05uR6Sn9HRpL4yLeJu/lmghK0LPPZsrvs7Di2g01HNrGrchf5FfmUNpSeNI8g9InoQ2hgKCGBIWQlZNE3oi99I/oyIn4EaVFp9I3oqzeUVbemyaKbcLnc5K8uYcPifThsLq54YBSDJ57/w2vOqirqli2n9sMPadq2DWOzERAVRdiE8SR977tEz5mjXVxb4XA7qLHV4HK7cBonlU2VrDm8hvzKfPZV76O0vvTEZaMoSxRjksYwb9A8+kb0JS40jj7hfRgWPwxLgMXHe6LU+dFk0Q1UlzWy8uV8jhTWkpQexQXXDSJ1ePw5r88YQ+2HH1K75H0av/gCd2MjgbGxxFx3LRHTphF54YUEROjYEABOt5PS+lIanY0cqD1AlbWKssYyCqoLqLPXsbNi52kD6ABkRGeQlZDF7AGzGRo31HMPITLFB3ugVNfQZOFDxm349LVd7FxbiiUkkEvvGM6IC1LO6XKE89gxmrZupfrdxTRt2YKrspKAmBgiLryQuFu/Rvjkyb26N5MxhrLGMt7a/RaFNYVYXVbKGsoorCnEbdynzZ8elU5yeDIXp13M2KSxhASGECiBBAUEMT55PKlRqT7YC6V8R5OFjzTW2vn01Z3s/7KC4RekMHX+QCJizu5ykDEGe2EhTVu3cuTnT2HsdgIiIoiYPp3IGdOJmTevVxXqO9Z0jL3Ve9lSvuVEz6IGRwNu4+Zg3UFq7bUApEWlERMcQ0JYAqMSRzEyYeSJS0b9IvsRaYnU5xCUOoUmCx+oOtLAB89uo6HGzgXXD2b87PSzWt7Y7Rz7+3NULVyIq9LzEFbwwIH0/dnPCB2Z5fdPUx8vU9HgaCC3LJftR7ez6cimk3oXJYcl0zeyLxGWCAIkgItjLmZo3FBGJo5kUt9JPoxeqZ5Jk0UXKt5VSe7SAxzeXYUlJJAF3xtP38y2u6m67XZsu3ZRt+JjGtatw7pnj2dUubFjiX7oIUKHDyN05EgCwv3n03C9vZ6NRzby2aHPqLRWYnVZqWiqoKyx7MQ4y8fFhsQyNmksk/pOIi0qjQv7X6jPHCjVwTRZdAHjNqxdtJdtHx8iIiaY8ZcPIGt6v1bHwzZOJ9YdOzj29+do2LgR0+T51Bw6ahRxN95I+OTJRF1xeY/tbmmMocnZRFFNETuO7SC/Mp/DdYexuWwcaTzCkYYjgKfb6aDYQURaIkkOT2Zin4kkhCWQEZ1BTEgMfcP7MjBWB05SqrNpsuhkdZVWlv8zjyOFNWTN6MeF1w8mOLTlH7vbasW2ezdNX+7g2F//iquqCrFYiJpzJeETJhIx/UKCU7v3jVWX20VuWS55FXmUN5bjMi7cxo3LuHC5XVRZqzhQd4BqazVVtqoTy4UGhjI0bihhljCy4rO4bvB1jE0ay5ikMUQG+/dlNaV6Ak0WncTa4GDVa7sp2nYUDMy4eSijL+l/2pmAu7GR2mXLsRfuo/rtd04U7rMMSCfxoYeIvmoOQYm+H0zG4XJQaa3EZVw43U7KGsuotlWzrmQdNbYa9tfu51DtIawu64llwoLCsARYCJRAAgMCCZAALAEWBsUOYmzSWNKi0kgOT2ZUwigyYzL1CWWlujFNFp3g0K5KVvwrH2udnawZ/Rk5vR9J6V+NL2zsdhq3bKVp2zaq33wTx+HDYLEQPCCdxG98nbAJEwgdNapLLjE53A6sTisOtwOHy4HNZWNf9T6ONB5hd+VuSupLPJeLaotOGjrzOEuAhbSoNCItkVwz6BoSwxJJDk9mev/p9Anv02MvkymlTqbJogMZY9i64hDrFu0ltk84cx4cRcrgWADshw5h27eP2iXv07B+/YkziODBg+j3298SPfcqJCCgU+NzGzeF1YUcrDtIWWMZXxz5gk8PfYrT3XLhuiAJOlGobnLfyUzsM5HwoHCCAoKIsESQHJ5MenS63kxWqhfQZNEBrA0O1i/aS9H2YzTVOUhKj+Kab48lLCoYe/FhjvzsZzSsXeuZOSiIiMmTiLr8CiIvmoGlX79Oja2kvoS397zN9mPb2Vy2+URJa4C4kDguS7+MrIQsQoNCsQRYsARYTiSI5PBkLVOhlAI0WZwX4zbkrT7MhvcKsTU6ScuKZ+C4JIaPj8G6eT0HX3uNhtVrkOBg4u+9l8gZ0wkdPbpTn4M4PiDOvpp9LCpYxLL9ywBPVdMrM64ku282g2MHExsSS1pUml4mUkq1iyaLc9BQY+PAjgoKviijeFcVyQOiyJ4cSvgXH2J7rZC9396IsdkIjI0l7rbbiL/jdoIHDOiw7VdaKzlYe5BKayXljeVYnVZKGkrYUr6FopoibC7biXnnDZrHnVl3Mix+WIdtXynV+2iyOAuNtXZ2fFbM5uUHcTnchEZamHJBGP32/Zeab76JFbD070/UrJlEX30NEdOmEhB25mcpzsTldlFYU0itvZZ6ez35lfnU2esorS/lUN0hdlftPm2ZQAlkaNxQLhtwGWMSxxAfGs+0ftOICdGxKZTqUMaAywFuJxiX57vb5f1ygtMK9gbAgHF75jfmlPduz3uXHeqPet97uZ1Qf8SzvjMt1+Jrvprv8l9AYMce3jVZtINxG7avKmbD4n047W76ZUYw0rKTgI//g+ODQmpEiJg+nb5PPklwav8zrqe8sZxdlbsoriumxlZDrb32xHMIh+sPc6zpGOWN5aeNowwQHhROTEgM/SP7c/uI2xmdOJqUyBSSwpKIC40jJDCEoAD971QdwGnzHuxodrBq73f3V69t9VBbAm6H5+DqaARH0xk26mXcUFPsmddl9yxXX+45AB9fb/O42nxNs/haek3b8xxfl8sGDitYa8DZxn50CgER7/eA019LwFfzXPakJouu5rC7WPlSPvs2HyU1I4QhjV8gLz6HCwgaNoykRx8l/o7bT5T8Nsaws3InB2sPUmOrobi+mLyKPKqsVeyt3nvSuo/3LAqUQEKDQhkSN4SBMQMZED2AxLBE0qLSCAsKIzMmU88QzofTDpWFnj9wl8NzwLLVnuFTG6d/anM0eNbRElsdWKu96zjly94AtaWeT5/H19vqgas9B0DvP26nZ9ttLdOubXHycvb6r977Slg8BAZ7voIjICIRjt9eO3GfTdr3+sQy7XnNmdslEEIiIcAC0Sme7wGBEBDkmRZw/MsCodHNDt4tHMybvw+L8+xjcxGJYIn4ar5ucG9Rk8UZ2K1O1v5nD7vWH8HtNgxvWE/KS68hQOSllxJ/z90wbiRHGo9QWJtHbkEuB+oO8MWRLyhvLD+xHkEYFj+MpLAkLkq9iGn9ppERnUGkJbLlJ5PdLs8BzXhPa40LnE5wlHvaTzrItfDH32IbLc/n9n5qO/Fp8JR5oPX1nmm60+75ZNhiHK3FfYZ1N1Z4PtGd2A0X1JV54j/pwO7+6lOou9nPz3H6eBQdyhLh+RQnAad/hSd4DganHchaOXC192AYEg0BAe1Y5iy3FWiByL6nfHptablTvjc/GB4/SMYNAEu4Z52BwRAS9VUMZxIcfvrBU/mcmBN/uN2biFwJ/AkIBP5pjHmmtfmzs7NNTk7OOW2r7sBB3v7VOholmT5lm+hXug5bWBGH0i2smhLMrkgbVuPGwcnjIPQlkGQTwJWNDibbHSS4DFHGENL81Py075z8KdZp/eqTaK9z6kHI2xYUChEJnHSQCYv1HCxP/dQWEun5VHriE1+A58AT2Qei+3k+9QVaPMsfP6Cf6bQePK+Pf7ptSUCg9wColH8QkVxjTPap7T3izEJEAoG/ArOBYuALEVlijMnv6G0V79vLR7/YhDMonphj/2DF0O0UzYSoEDeRbgcWGriwzpDsdBNiDP1cBogmwxFMtAnCTSD1gdEcCYjkCIKbABAwIhjvwc54D0rmlC8Ap1hoCgjHTQBGAnATiJsA3BKAmwCaAiI80/Cuw3tMO/EeaXYBQVqZz/O6PiAae0DoSTEcn++r9951eg+grc8LLgnEKuHN5m8pvq/Wdc6n2KePWQQOoP6U2Qwcq7Phcjf/YHT60+iq+6pqtFNva/nhUXW6zf9vNqGWji2f0yOSBTAZ2GuMKQQQkYXAfKBDk4XDamPpzz/BETqAqpB3+O+suwkglDACcdqh2jvfUWDP8f+H4z9BXw1j3d4TwzPN19IBt1M0u7zkA7HhFqJD9QHDnmpwYCR9o0MJ8P2l+x4hsBN+UD0lWfQHDjV7XwxMOXUmEXkQeBAgPf3sBhQCCLIEEhhYQLRlJw//5flzDFUppfxPT0kWLaXJ0z6mGmOeB54Hzz2Ls95IYBD3vPjbs49OKaX8XOdWrus4xUBas/epQImPYlFKqV6npySLL4AhIpIpIsHALcASH8eklFK9Ro+4DGWMcYrIt4BleLrO/ssYk+fjsJRSqtfoEckCwBjzIfChr+NQSqneqKdchlJKKeVDmiyUUkq1SZOFUkqpNmmyUEop1aYeU0jwbInIUeDAOS6eCBzrwHB6At3n3qG37XNv2184/30eYIxJOrXRb5PF+RCRnJaqLvoz3efeobftc2/bX+i8fdbLUEoppdqkyUIppVSbNFm0rDeWnNV97h162z73tv2FTtpnvWehlFKqTXpmoZRSqk2aLJRSSrVJk0UzInKliOwWkb0i8riv4zkfIvIvESkXkR3N2uJFZIWIFHi/xzWb9oR3v3eLyBXN2ieKyJfeaX8WOdcBszufiKSJyKcislNE8kTkO952v91vEQkVkU0iss27zz/3tvvtPgOISKCIbBGRD7zv/X1/93tj3SoiOd62rt1nY4x+ee7bBAL7gIFAMLANyPJ1XOexPxcBE4Adzdp+Azzuff048L/e11ne/Q0BMr0/h0DvtE3ANDyjFX4EzPH1vrWyzynABO/rKGCPd9/8dr+98UV6X1uAjcBUf95nb6zfA14HPuglv9v7gcRT2rp0n/XM4iuTgb3GmEJjjB1YCMz3cUznzBjzOVB5SvN84GXv65eBBc3aFxpjbMaYImAvMFlEUoBoY8x64/lNe6XZMt2OMabUGLPZ+7oO2Iln/Ha/3W/jUe99a/F+Gfx4n0UkFZgL/LNZs9/ubyu6dJ81WXylP3Co2ftib5s/6WOMKQXPgRVI9rafad/7e1+f2t7tiUgGMB7PJ22/3m/vJZmtQDmwwhjj7/v8R+BHgLtZmz/vL3g+ACwXkVwRedDb1qX73GMGP+oCLV276y39is+07z3yZyIikcA7wKPGmNpWLsv6xX4bY1zAOBGJBd4VkVGtzN6j91lErgbKjTG5InJJexZpoa3H7G8zFxpjSkQkGVghIrtambdT9lnPLL5SDKQ1e58KlPgols5S5j0Vxfu93Nt+pn0v9r4+tb3bEhELnkTxmjFmkbfZ7/cbwBhTDawCrsR/9/lCYJ6I7MdzqXimiPwb/91fAIwxJd7v5cC7eC6bd+k+a7L4yhfAEBHJFJFg4BZgiY9j6mhLgLu8r+8C3mvWfouIhIhIJjAE2OQ9ta0TkaneXhN3Nlum2/HG+AKw0xjzh2aT/Ha/RSTJe0aBiIQBlwG78NN9NsY8YYxJNcZk4Pkb/cQYczt+ur8AIhIhIlHHXwOXAzvo6n329V3+7vQFXIWnB80+4Ce+juc89+UNoBRw4PlEcR+QAKwECrzf45vN/xPvfu+mWQ8JINv7i7kPeBbvU//d8QuYjue0ejuw1ft1lT/vNzAG2OLd5x3AT73tfrvPzeK9hK96Q/nt/uLpobnN+5V3/NjU1fus5T6UUkq1SS9DKaWUapMmC6WUUm3SZKGUUqpNmiyUUkq1SZOFUkqpNmmyUAoQkQRvRc+tInJERA57X9eLyP91UQzjROSqrtiWUmdLy30oBRhjKoBxACLyJFBvjPldF4cxDk8/+A+7eLtKtUnPLJRqhYhc0mzMhCdF5GURWe4dX+A6EfmNd3yApd5SI8fHDPjMW/Rt2fGSDKes90YR2SGecSg+91YNeAq42XtGc7P3yd1/icgX4hm7Yb532btF5D3vNneLyM+87REi8l/vOneIyM1d95NS/k7PLJQ6O4OAS/GMGbAeuN4Y8yMReReYKyL/Bf4CzDfGHPUesH8J3HvKen4KXGGMOSwiscYYu4j8FMg2xnwLQER+haecxb3ekh6bRORj7/KTgVFAI/CFd7sDgBJjzFzv8jGd9lNQvY4mC6XOzkfGGIeIfIlnwKyl3vYvgQxgGJ6D+ApvtdtAPGVXTrUWeElE3gIWtTAdPDWA5onID7zvQ4F07+sV3ktniMgiPKVOPgR+JyL/i6cMxupz3kulTqHJQqmzYwMwxrhFxGG+qpfjxvP3JECeMWZaaysxxnxDRKbgGcRnq4iMa2E2wXPmsvukRs9yp9bpMcaYPSIyEU89rF+LyHJjzFNnuX9KtUjvWSjVsXYDSSIyDTwl00Vk5KkzicggY8xGY8xPgWN4SkrX4RkO9rhlwLe9FUIRkfHNps0WzxjMYXhGO1srIv2ARmPMv4Hf4RlWV6kOoWcWSnUg772HG4A/e+8ZBOEZ2S3vlFl/KyJD8Jw9rMRTUfQg8Lh4Rr37NfC0d9nt3oSxH7jau/wa4FVgMPC6MSZHRK7wrteNp9rwQ520m6oX0qqzSvUwInI3zW6EK9UV9DKUUkqpNumZhVJKqTbpmYVSSqk2abJQSinVJk0WSiml2qTJQimlVJs0WSillGrT/w9X94ioOAuKMwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(0)\n",
    "epsilons = [1e-4, 0.01, 0.1, 0.25, 0.5]\n",
    "epsilon_greedy_solver_list = [EpsilonGreedy(bandit_10_arm, epsilon=e) for e in epsilons]\n",
    "epsilon_greedy_solver_names = [\"epsilon={}\".format(e) for e in epsilons]\n",
    "for solver in epsilon_greedy_solver_list:\n",
    "    solver.run(5000)\n",
    "plot_results(epsilon_greedy_solver_list, epsilon_greedy_solver_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "080F7BBD6F454ECBA4B1B6CAD0C5E279",
    "jupyter": {},
    "mdEditEnable": false,
    "notebookId": "5f8f31cc46ba5e003077fa26",
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "通过实验结果，我们发现基本上无论$\\epsilon$取值多少，累积懊悔都是线性增长的。在这个例子中，随着$\\epsilon$的增大，累积懊悔增长的速率也会增大。\n",
    "\n",
    "接下来我们尝试$\\epsilon$值随时间衰减的$\\epsilon$-贪心策略，采取的具体衰减形式为线性衰减，具体公式为 $\\epsilon_t = \\frac{1}{t}$。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "77C854B4F61F4B2E8B0036F8C0FF2D91",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class DecayingEpsilonGreedy(Solver):\n",
    "    \"\"\" epsilon取值随时间衰减的Epsilon贪心算法，继承Solver类 \"\"\"\n",
    "    def __init__(self, bandit, init_prob=1.0):\n",
    "        super(DecayingEpsilonGreedy, self).__init__(bandit)\n",
    "        self.estimates = np.array([init_prob] * self.bandit.K)\n",
    "        self.total_count = 0\n",
    "\n",
    "    def run_one_step(self):\n",
    "        self.total_count += 1\n",
    "        if np.random.random() < 1 / self.total_count: # epsilon随时间衰减\n",
    "            k = np.random.randint(0, self.bandit.K) \n",
    "        else:\n",
    "            k = np.argmax(self.estimates)  \n",
    "        \n",
    "        r = self.bandit.step(k) \n",
    "        self.estimates[k] += 1. / (self.counts[k] + 1) * (r - self.estimates[k])\n",
    "\n",
    "        return k"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "77E5ABF5CF9240768BC2D4C72CF50B08",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epsilon衰减的贪心算法的累积懊悔为： 10.114334931260183\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEWCAYAAABhffzLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAlxElEQVR4nO3deZxU5ZX/8c+h2RFEBBwUoTFxNIjdiA0EdQgBBaOGGHHcFzBKzKgETVz4/YioGZ0MYebniEkUI6KJxl0kuAQ1ooNh0EYJAUERBxEl0qLsi91d5/fHvVVdvdAUTS1dt77v16tfXXXX82B76qnnPvdcc3dERKRwtMh1ACIikl1K/CIiBUaJX0SkwCjxi4gUGCV+EZECo8QvIlJglPhFMsDMZpnZv+5h3VgzW5ClOGqdy8y2mdkR2Ti3NF9K/JIzZna1mZWb2W4zm9XA+hFmttLMdpjZq2bWOwdhRoq7H+DuH0LjH04SbUr8kkufAv8KzKy7wsy6Ak8DPwO6AOXAY+k4qQX0ty8FS3/8kjPu/rS7zwY2NrD6LGC5uz/h7ruAW4BSMzu6oWOZ2UFmNtfMKszsy/B1z6T1883sdjN7A9gBHGFmbmb/YmarzGyrmf3czL5mZgvNbIuZPW5mrZOOcYaZLTGzTWb2FzMrSVp3nJm9HR7nMaDtXppvZjbdzDaH32pGJK0YZ2YrwmN9aGY/TFo3zMzWmdlPzGyDma03s3FJ6w82szlh/G8CX6tzUjezr5vZeOBC4IZw+OePe4lXIkSJX5qrY4C/xt+4+3Zgdbi8IS2AB4DeQC9gJ3B3nW0uBsYDHYGPwmWnAscD3wRuAGYQJMTDgX7A+QBmNoDgm8kPgYOBe4E5ZtYm/HCYDfyO4NvJE8CYvbRvMPAh0BWYAjxtZl3CdRuAM4BOwDjg/4Xnj/sH4EDgMOAHwK/M7KBw3a+AXUAP4LLwpx53nwE8DEwNh3++u5d4JUKU+KW5OgDYXGfZZoKkXY+7b3T3p9x9h7tvBW4HvlVns1nuvtzdq9y9Mlz27+6+xd2XA8uAee7+obtvBl4Ajgu3uwK4190XuXu1uz8I7Cb4wPgm0Aq4090r3f1J4K29tG9D0vaPAe8Bp4dtec7dV3vgNWAe8E9J+1YCt4X7Pg9sA44ysyKCD5yb3X27uy8DHtxLHFKAlPiludpG0ONN1gnYama9wuGJbWa2DcDM2pvZvWb2kZltAV4HOofJMO7jBs7zWdLrnQ28PyB83Rv4STjMs8nMNhF8Kzg0/PnEa1c8/IjGNbT9oWFbvmNm/2NmX4TnOY3gm0HcRnevSnq/I4yzG9CyTjv3FocUICV+aa6WA6XxN2bWgWC8erm7rw2HJw5w93hi/glwFDDY3TsBQ+O7Jh1zf0rRfgzc7u6dk37au/sfgPXAYWaWfK5eezleQ9t/amZtgKeAacAh7t4ZeL5OO/akAqgi+EBKJQ6V5i1QSvySM2bW0szaAkVAkZm1NbOW4epngH5mNibc5mZgqbuv3MPhOhL00DeFY+VT0hzufcCVZjY4nBXUwcxON7OOwEKChDshbNNZwKC9HK97uH0rM/tn4BsECb410IYwiZvZd4CRqQTo7tUEM6FuCb8B9QUubWSXzwDN6S9ASvySS5MJkvVNwEXh68kA7l5BMF59O/AlwcXQ8xo51p1AO+Bz4H+AF9MZqLuXE4zz3x3G8wEwNlz3FcEspLHhunMJEnBjFgFHhvHeDpwdXqfYCkwAHg+PdQEwZx9CvZpg2OfvwCyCC957cj/QNxy6mr0P55A8Z3oQi4hIYVGPX0SkwCjxi4gUGCV+EZECo8QvIlJgWu59k9zr2rWrFxcX5zoMEZG8snjx4s/dvVvd5XmR+IuLiykvL891GCIiecXMGrxzW0M9IiIFRolfRKTAKPGLiBSYvBjjb0hlZSXr1q1j165duQ5FIqxt27b07NmTVq1a5ToUkbTJ28S/bt06OnbsSHFxMbWLHIqkh7uzceNG1q1bR58+fXIdjkja5O1Qz65duzj44IOV9CVjzIyDDz5Y3yolcvI28QNK+pJx+huTKMrYUI+ZzSR4bugGd+8XLusCPAYUA2uAc9z9y0zFICKyfXcVDy5cw66vqnMdSpN8f0BP+nTtkNZjZnKMfxZB7fKHkpbdBLzi7r8ws5vC9zdmMIaMKioq4thjj6WyspKWLVty6aWXMnHiRFq0yPwXqZtvvpmhQ4dy8sknN2n/eOxx5513HjfddNM+HaO8vJyHHnqIu+66i1mzZlFeXs7dd9d9vnlqVq1axbXXXsuKFSvo3LkznTp14tZbb2Xo0KF733kfDRs2jGnTplFWVpb2Y0vzs3D1Rqa++B4A+fgFbkDvg/In8bv762ZWXGfx94Bh4esHgfnkceJv164dS5YsAWDDhg1ccMEFbN68mVtvvTXj577tttv2a//k2JuqrKwsLclz165dnH766UybNo3Ro0cDsGzZMsrLy+sl/qqqKlq2zNs5CZIDVbEYAC/8+J/4Ro+6j3EuTNke4z/E3dcDhL+7Z/n8GdO9e3dmzJjB3XffjbtTXV3N9ddfz8CBAykpKeHee+9NbDt16lSOPfZYSktLE73s++67j4EDB1JaWsqYMWPYsWMHW7dupU+fPlRWVgKwZcsWiouLqaysZOzYsTz55JNAUNJiypQpDBgwgGOPPZaVK4OnE1ZUVHDKKacwYMAAfvjDH9K7d28+//zzRttRXFzMjTfeyKBBgxg0aBAffPABAE888QT9+vWjtLQ0kYznz5/PGWecUe8YH330ESNGjKCkpIQRI0awdu1aAMaOHcuECRM44YQTOOKIIxLxP/zwwwwZMiSR9AH69evH2LFjAbjlllsYP348I0eO5JJLLqGiooIxY8YwcOBABg4cyBtvvAHA9u3bueyyyxg4cCDHHXcczz77LAA7d+7kvPPOo6SkhHPPPZedO3cCcP/993Pttdcmznnfffdx3XXXNf4fWvJOLHzWVIt87O5nSLPtOpnZeGA8QK9ejT+3+tY/LufdT7ek9fx9D+3ElO8es0/7HHHEEcRiMTZs2MCzzz7LgQceyFtvvcXu3bs58cQTGTlyJCtXrmT27NksWrSI9u3b88UXXwBw1llnccUVVwAwefJk7r//fq655hqGDRvGc889x5lnnsmjjz7KmDFjGpxT3rVrV95++21+/etfM23aNH77299y6623Mnz4cCZNmsSLL77IjBkzEtvv3LmT/v37J95PmjSJc889F4BOnTrx5ptv8tBDDzFx4kTmzp3Lbbfdxp/+9CcOO+wwNm3a1Oi/w9VXX80ll1zCpZdeysyZM5kwYQKzZ88GYP369SxYsICVK1cyevRozj77bJYvX86AAQMaPebixYtZsGAB7dq144ILLuDaa6/lpJNOYu3atYwaNYoVK1Zw++23M3z4cGbOnMmmTZsYNGgQJ598Mvfeey/t27dn6dKlLF26NHGu+IfB1KlTadWqFQ888ECtD2iJhlj4lMEWyvsJ2U78n5lZD3dfb2Y9gA172tDdZwAzAMrKyvLm+ZDxR1nOmzePpUuXJnq1mzdvZtWqVbz88suMGzeO9u3bA9ClSxcgGNqYPHkymzZtYtu2bYwaNQqAyy+/nKlTp3LmmWfywAMPcN999zV43rPOOguA448/nqefDh73umDBAp555hkATj31VA466KDE9o0N9Zx//vmJ3/Ee8YknnsjYsWM555xzEufak4ULFyZiuPjii7nhhhsS684880xatGhB3759+eyzzxrc//vf/z6rVq3iH//xHxPHGT16NO3atQPg5Zdf5t13301sv2XLFrZu3cq8efOYM2cO06ZNA4IhpLVr1/L6668zYcIEAEpKSigpKQGgQ4cODB8+nLlz5/KNb3yDysrKWtc9JBqqwy6/ZmjVyHbinwNcCvwi/P1sOg66rz3zTPnwww8pKiqie/fuuDvTp09PJPC4F198scE/wLFjxzJ79mxKS0uZNWsW8+fPB4KEu2bNGl577TWqq6vp169fg+du06YNEFy0raqqAmo+hPZVcnzx1/fccw+LFi3iueeeo3///vt0fSD5ePE4k+M75phjeP311xPLn3nmGcrLy/npT3+aWNahQ83FrVgsxsKFCxMfBMnHe+qppzjqqKMajSHZ5Zdfzh133MHRRx/NuHHjUm6T5A9PDPXkNo7mJGNj/Gb2B2AhcJSZrTOzHxAk/FPMbBVwSvg+EioqKrjyyiu5+uqrMTNGjRrFb37zm8T4/Pvvv8/27dsZOXIkM2fOZMeOHQCJoZ6tW7fSo0cPKisrefjhh2sd+5JLLuH888/f58R00kkn8fjjjwPBN5Avv0xt5uxjjz2W+D1kyBAAVq9ezeDBg7ntttvo2rUrH3/88R73P+GEE3j00UeBYPz+pJNOavR8F1xwAW+88QZz5sxJLIv/+zRk5MiRtWYPxT+ERo0axfTp0xMfKO+88w4AQ4cOTfybLlu2jKVLlyb2HTx4MB9//DGPPPJI4puOREvNUI8yf1wmZ/Xs6f+iEZk6Z7bFx8nj0zkvvvjixMXByy+/nDVr1jBgwADcnW7dujF79mxOPfVUlixZQllZGa1bt+a0007jjjvu4Oc//zmDBw+md+/eHHvssWzdujVxngsvvJDJkyfvc2KaMmUK559/Po899hjf+ta36NGjBx07dqwVe9ypp57KL34RfA7v3r2bwYMHE4vF+MMf/gDA9ddfz6pVq3B3RowYQWlpKa+99lqD573rrru47LLL+OUvf0m3bt144IEHGo2zXbt2zJ07l+uuu46JEydyyCGH0LFjRyZPnrzH41911VWUlJRQVVXF0KFDueeee/jZz37GxIkTKSkpwd0pLi5m7ty5/OhHP2LcuHGUlJTQv39/Bg0aVOt455xzDkuWLKk1FCbREb+4W6Qufw13b/Y/xx9/vNf17rvv1lsWVU888YRfdNFF+7zfrl27vLKy0t3d//KXv3hpaele9+ndu7dXVFTs87ny2emnn+4vv/zyHtcX0t9aFD321lrvfeNc//iL7bkOJeuAcm8gpzbbWT0SuOaaa3jhhRd4/vnn93nftWvXcs455xCLxWjduvUeLwwXqvjMn9LSUkaMiMwXUanDNdRTjxJ/Mzd9+vQm73vkkUcmxrlTtWbNmiafL9907tyZ999/P9dhSIZpHn99eV2kLf5JLpIp+hvLf5rHX1/eJv62bduyceNG/Y8pGeNhPf62bdvmOhTZD/Eev+bx18jboZ6ePXuybt06Kioqch2KRFj8CVySv+KdQ83qqZG3ib9Vq1Z6KlIEfbJpJ6OnL2Db7qpch5JkZa4DkP0Qv3O3SD3+hLxN/BJNazfuYOP2r/hu6aEc2llDLJIeh3Vux4Ht9dzkOCV+aVbivbOLv9mbQX265DgakWjK24u7Ek3x2ukti/S1XCRTlPilWYn3+FvqQpxIxijxS7NSFdMMDJFMU+KXZqWmx68/TZFM0cXdHFj04UZWV2zPdRjN0tJ1mwD1+EUySYk/B654qJwtu5rTPPXmpXXLFnTp0DrXYYhElhJ/DuyqjHHJkN5c9e2v5zqUZql96yI6ttWca5FMUeLPgapYjE5tW3FIJ92gJCLZpytoWebuxFxj2CKSO0r8WVat6YoikmNK/FmmeeoikmtK/FkWfyiE7kwVkVxR4s8y9fhFJNeU+LMspsQvIjmmxJ9l6vGLSK4p8WeZZvWISK4p8WeZHgMnIrmmO3ebqLI6xmvvVbCrqnqf9tu47StAPX4RyR0l/iZ6/f0KLn+ovMn7H3yAipCJSG4o8TfRtt1Bdc2ZY8s4/KD2+7Rv65Yt6NVl3/YREUkXJf4mqqwOxuq/3q0jvQ5WEheR/KGLu01UVa2HgotIflLib6LK+CMClfhFJM/kZKjHzK4FLgcc+Bswzt135SKWhnxYsW2vT8hauzF4dGIrPRtWRPJM1hO/mR0GTAD6uvtOM3scOA+Yle1YGrJ24w6G/8drKW3bsoXRtlVRhiMSEUmvXF3cbQm0M7NKoD3waY7iqGfzzkoAJow4kuMO79zott07taFdayV+EckvWU/87v6JmU0D1gI7gXnuPq/udmY2HhgP0KtXr6zFFy+b3P/wA/n20d2zdl4RkWzJ+gC1mR0EfA/oAxwKdDCzi+pu5+4z3L3M3cu6deuWtfiqw8TfQiUVRCSicnFl8mTgf929wt0rgaeBE3IQR4NciV9EIi4XiX8t8E0za29mBowAVuQgjgaF0/NVS0dEIivrid/dFwFPAm8TTOVsAczIdhx7Eh/jV4dfRKIqJ7N63H0KMCUX596b+BOyNNQjIlGlu4/qCPO+hnpEJLKU+OuomdWT40BERDJEib+OmGb1iEjEKfHXoemcIhJ1+3Rx18xaAAe4+5YMxZMVH3+xg0ffWpuYupnso7D4msb4RSSq9pr4zewR4EqgGlgMHGhm/+nuv8x0cJny1Nvr+NWrq2ndsuEvPF0PaMMhndpmOSoRkexIpcff1923mNmFwPPAjQQfAHmb+Hd+VU3bVi1Y+fPv5DoUEZGsS2WMv5WZtQLOBJ4Nyyzktd1VMdq0VFVNESlMqfT47wXWAH8FXjez3sDmTAaVCVf+bjHL1wdhf7m9kg5tlPhFpDClkvj/6O53xd+Y2VrgssyFlBmvrPyMPl070O/QAwE4vvigHEckIpIbqST+p4AB8Tfu7mb2KHB8xqLKgKqYM+qYf+AnI4/KdSgiIjm1x8RvZkcDxxDM4jkraVUnIK+mvMRijrvm5ouIQOM9/qOAM4DOwHeTlm8FrshgTGkXL8PQUnPzRUT2nPjd/VngWTMb4u4LsxhT2lWHldeKipT4RURSmc650cxeMbNlAGZWYmaTMxxXWsUTv3r8IiKpJf77gElAJYC7LwXOy2RQ6ValGvsiIgmpJP727v5mnWVVmQgmEzbvrOTxtz4G1OMXEYHUEv/nZvY1wAHM7GxgfUajSqM5f/2U258PHul72EHtcxyNiEjupTKP/yqCZ+IebWafAP8LXJjRqNJod2U1AAsnDafHge1yHI2ISO41mvjNrAj4kbufbGYdgBbuvjU7oaVH/MEqHdu2ynEkIiLNQ6OJ392rzez48PX27ISUXvGa+0W6sCsiAqQ21POOmc0BngASyd/dn85YVGmUeJSinjUmIgKklvi7ABuB4UnLHMiLxJ+4eUs9fhERIIXE7+7jshFIpiQSv6ZyiogAqT168a4GFm8GysOyDs1azB0zMPX4RUSA1ObxtwX6A6vCnxKC4Z8fmNmdGYssTapjrmEeEZEkqYzxfx0Y7u5VAGb2G2AecArwtwzGlhbV7rTQMI+ISEIqPf7DgA5J7zsAh7p7NbA7I1GlUUw9fhGRWlLp8U8FlpjZfMCAocAd4Q1dL2cwtrSojunCrohIslRm9dxvZs8DgwgS//9x90/D1ddnMrh0iLmjvC8iUmOvQz0WTIcZAZS6+2ygpZkNynRg6VIdc/X4RUSSpDLG/2tgCHB++H4r8Kv9OamZdTazJ81spZmtMLMh+3O8xlS7Er+ISLJUxvgHu/sAM3sHwN2/NLPW+3ne/wJedPezw2NlrF5yLOZ6AIuISJJUEn9lWKUzXo+/GxBr6gnNrBPBBeKxAO7+FfBVU4+3NxrqERGpLZWhnruAZ4DuZnY7sAC4Yz/OeQRQATxgZu+Y2W/DGUK1mNl4Mys3s/KKioomn6za1eMXEUnWaOI3sxYED165Afg3gidvnenuT+zHOVsCA4DfuPtxBBU/b6q7kbvPcPcydy/r1q1bk0/mrsqcIiLJ9laPP2Zm/+HuQ4CVaTrnOmCduy8K3z9JA4k/XVSyQUSktlT6wvPMbIylqcqZu/8d+NjMjgoXjQDeTcexG6KSDSIitaVycfc6gjINVWa2i+AmLnf3Tvtx3muAh8MZPR8CGSv9rJINIiK1pXLnbsd0n9TdlwBl6T5uQzSrR0Sktshf9oxpVo+ISC2RT/zq8YuI1Bb9xO/o4q6ISJKUEr+ZnWRm48LX3cysT2bDSp/g4m6uoxARaT5Sqc45BbgRmBQuagX8PpNBpZPG+EVEakulx/99YDTBHbaEtfjTPtMnU9xR4hcRSZJK4v/K3Z2aIm316uo0ZzH34M4DEREBUkv8j5vZvUBnM7uC4HGL92U2rPRxlPdFRJKlcgPXNDM7BdgCHAXc7O4vZTyydNGsHhGRWvaa+M3sWuCJvEr2SWLutNQYv4hIQipDPZ2AP5nZf5vZVWZ2SKaDSidHF3dFRJLtNfG7+63ufgxwFXAo8JqZvZzxyNIk5o7yvohIjX25c3cD8HdgI9A9M+GknzukqaK0iEgkpHID14/MbD7wCtAVuMLdSzIdWLq4u2b1iIgkSaUef29gYlhKOe8EY/y5jkJEpPnYY+I3s07uvgWYGr7vkrze3b/IcGxpEYzxK/OLiMQ11uN/BDgDWEz9+6AcOCKDcaWNbtwVEaltj4nf3c8If+dNJc6G6OKuiEhtqVzcfSWVZc2VpnOKiNTW2Bh/W6A90NXMDqJmxKQTwXz+vKGLuyIiNRob4/8hMJEgyS+mJvFvAX6V2bDSJ+aOaZRfRCShsTH+/wL+y8yucffpWYwprdyhReQfMCkikrpUqnNON7N+QF+gbdLyhzIZWLqoxy8iUlsq1TmnAMMIEv/zwHeABUBeJP7g6TG5jkJEpPlIZRDkbGAE8Hd3HweUAm0yGlU66dGLIiK1pJL4d7p7DKgys04Exdry4uYtiA/1iIhIXCq1esrNrDPB4xYXA9uANzMZVDqpVo+ISG2pXNz9l/DlPWb2ItDJ3ZdmNqz0Ua0eEZHaGruBa0Bj69z97cyElF5ByYZcRyEi0nw01uP/j0bWOTA8zbFkRFCkTZlfRCSusRu4vp3NQDLFVatHRKSWVObxX9LQ8vy5gUsXd0VEkqUyq2dg0uu2BHP632Y/b+AysyKgHPgkXgI6ExzduSsikiyVWT3XJL83swOB36Xh3D8GVhBU+8wY1eoREamtKSlxB3Dk/pzUzHoCpwO/3Z/jpCKmmg0iIrWkMsb/R8KSNwQfFH2Bx/fzvHcCNwAdGznveGA8QK9evfbjVK4xfhGRJKmM8U9Lel0FfOTu65p6QjM7A9jg7ovNbNietnP3GcAMgLKyMt/TdnsT0zx+EZFaUhnjfw0grNPTMnzdxd2/aOI5TwRGm9lpBBeLO5nZ7939oiYer1HuriJtIiJJUnnm7ngz+wxYSjALZ3H4u0ncfZK793T3YuA84M+ZSvoQ9vgzdXARkTyUylDP9cAx7v55poPJBFetHhGRWlJJ/KsJZvKknbvPB+Zn4tiJc6AxfhGRZKkk/knAX8xsEbA7vtDdJ2QsqjRSrR4RkdpSSfz3An8G/gbEMhtO+gUXd3MdhYhI85FK4q9y9+syHkmGaDqniEhtqdy5+2o4s6eHmXWJ/2Q8sjRxNJ1TRCRZKj3+C8Lfk5KWOXny3N2Yo/mcIiJJUrmBq082AskYXdwVEakl8vX4XbV6RERqyVk9/mzRxV0RkdpyWY8/K1SrR0SktpzU488m1eoREaktV/X4s8I9CFu1ekREamS9Hn82hXlfY/wiIkn2mPjN7OvAIfF6/EnL/8nM2rj76oxHt58SX1OU+UVEEhob478T2NrA8p3humYvFh/qyXEcIiLNSWOJv9jdl9Zd6O7lQHHGIkojDfWIiNTXWOJv28i6dukOJBMcXdwVEamrscT/lpldUXehmf2A4PGLzZ56/CIi9TU2q2ci8IyZXUhNoi8DWgPfz3BcaRFP/Lq4KyJSY4+J390/A04ws28D/cLFz7n7n7MSWRro4q6ISH2plGx4FXg1C7GknaZziojU15SSDXkj0eNX3hcRSYh04o+P8YuISI1IJ350cVdEpJ5IJ34N9YiI1BfpxK+LuyIi9UU68avHLyJSX6QTf82du8r8IiJxEU/8uoFLRKSuaCf+8Lc6/CIiNaKd+DWdU0SknkgnftXqERGpL+uJ38wON7NXzWyFmS03sx9n6lyazikiUl8qD1tPtyrgJ+7+tpl1BBab2Uvu/m66TxSLxaf1pPvIIiL5K+s9fndf7+5vh6+3AiuAwzJ5TvX4RURq5HSM38yKgeOARQ2sG29m5WZWXlFR0aTja4xfRKS+nCV+MzsAeAqY6O5b6q539xnuXubuZd26dWvSORKzeiJ9CVtEZN/kJCWaWSuCpP+wuz+dqfPU9PjV5xcRicvFrB4D7gdWuPt/ZvJcuoFLRKS+XPT4TwQuBoab2ZLw57RMnGjLzkoAqmN6IouISFzWp3O6+wKydL01PtTTpmVRNk4nIpIXIn3Zs6o6SPyd27fKcSQiIs1HpBN/ddjj1zx+EZEakU78sVjwu6iFEr+ISFykE3+8x18U6VaKiOybSKfEeK0eDfWIiNSIdOKvChN/S926KyKSEOmMGJ+/r7wvIlIj0ikxlhjj11CPiEhcpBN/vMdfpDF+EZGESCd+9fhFROqLdOKP37mrxC8iUiPSiV937oqI1BfpxB+fx68ev4hIjUgn/mqN8YuI1BPpxK8ev4hIfZFO/FWazikiUk+kE3/NnbtK/CIicZFO/JrHLyJSX6QTf3W8Hr+GekREEiKd+NXjFxGpL9KJX3fuiojUF+nEX3Pnbo4DERFpRiKd+GMxp4WBaYxfRCQh0om/2l3DPCIidUQ78ceU+EVE6op+4tcwj4hILZFP/LprV0Sktkgn/pjG+EVE6mmZ6wAy6ZhDO7GrsjrXYYiINCuRTvznDuzFuQN75ToMEZFmJdJDPSIiUl9OEr+ZnWpm75nZB2Z2Uy5iEBEpVFlP/GZWBPwK+A7QFzjfzPpmOw4RkUKVix7/IOADd//Q3b8CHgW+l4M4REQKUi4S/2HAx0nv14XLajGz8WZWbmblFRUVWQtORCTqcpH4G5pY7/UWuM9w9zJ3L+vWrVsWwhIRKQy5SPzrgMOT3vcEPs1BHCIiBSkXif8t4Egz62NmrYHzgDk5iENEpCCZe71Rlsyf1Ow04E6gCJjp7rfvZfsK4KMmnq4r8HkT981XanNhUJsLw/60ube71xsrz0nizyYzK3f3slzHkU1qc2FQmwtDJtqsO3dFRAqMEr+ISIEphMQ/I9cB5IDaXBjU5sKQ9jZHfoxfRERqK4Qev4iIJFHiFxEpMJFO/FEp/2xmM81sg5ktS1rWxcxeMrNV4e+DktZNCtv8npmNSlp+vJn9LVx3l1nzfRK9mR1uZq+a2QozW25mPw6XR7bdZtbWzN40s7+Gbb41XB7ZNkNQsdfM3jGzueH7SLcXwMzWhPEuMbPycFn22u3ukfwhuDlsNXAE0Br4K9A313E1sS1DgQHAsqRlU4Gbwtc3Af8evu4btrUN0Cf8NygK170JDCGol/QC8J1ct62RNvcABoSvOwLvh22LbLvD+A4IX7cCFgHfjHKbw1ivAx4B5hbC33YY7xqga51lWWt3lHv8kSn/7O6vA1/UWfw94MHw9YPAmUnLH3X33e7+v8AHwCAz6wF0cveFHvzFPJS0T7Pj7uvd/e3w9VZgBUEV18i22wPbwretwh8nwm02s57A6cBvkxZHtr17kbV2Rznxp1T+OY8d4u7rIUiSQPdw+Z7afVj4uu7yZs/MioHjCHrAkW53OOyxBNgAvOTuUW/zncANQCxpWZTbG+fAPDNbbGbjw2VZa3eUH7aeUvnnCNpTu/Py38PMDgCeAia6+5ZGhjAj0W53rwb6m1ln4Bkz69fI5nndZjM7A9jg7ovNbFgquzSwLG/aW8eJ7v6pmXUHXjKzlY1sm/Z2R7nHH/Xyz5+FX/UIf28Il++p3evC13WXN1tm1oog6T/s7k+HiyPfbgB33wTMB04lum0+ERhtZmsIhmKHm9nviW57E9z90/D3BuAZgqHprLU7yok/6uWf5wCXhq8vBZ5NWn6embUxsz7AkcCb4VfHrWb2zfDK/yVJ+zQ7YYz3Ayvc/T+TVkW23WbWLezpY2btgJOBlUS0ze4+yd17unsxwf+ff3b3i4hoe+PMrIOZdYy/BkYCy8hmu3N9dTuTP8BpBLNBVgP/N9fx7Ec7/gCsByoJPuV/ABwMvAKsCn93Sdr+/4Ztfo+kq/xAWfgHthq4m/DO7eb4A5xE8LV1KbAk/Dktyu0GSoB3wjYvA24Ol0e2zUnxDqNmVk+k20sw0/Cv4c/yeG7KZrtVskFEpMBEeahHREQaoMQvIlJglPhFRAqMEr+ISIFR4hcRKTBK/BI5ZnZwWPVwiZn93cw+CV9vM7NfZymG/mZ2WjbOJbKvolyyQQqUu28E+gOY2S3ANnefluUw+hPMsX4+y+cV2Sv1+KVgmNmwpJrvt5jZg2Y2L6yNfpaZTQ1rm78YlouI1zt/LSym9af4LfV1jvvPZrbMgjr6r4d3it8GnBt+0zg3vFtzppm9ZUHt+e+F+441s2fDc75nZlPC5R3M7LnwmMvM7Nzs/UtJ1KnHL4Xsa8C3CeqdLwTGuPsNZvYMcLqZPQdMB77n7hVh8r0duKzOcW4GRrn7J2bW2d2/MrObgTJ3vxrAzO4gKElwWViW4U0zezncfxDQD9gBvBWetzfwqbufHu5/YMb+FaTgKPFLIXvB3SvN7G8ED+55MVz+N6AYOIogIb8UVgUtIiidUdcbwCwzexx4uoH1ENRjGW1mPw3ftwV6ha9fCoenMLOnCcpVPA9MM7N/Jyhl8N9NbqVIHUr8Ush2A7h7zMwqvaZ+SYzg/w0Dlrv7kMYO4u5XmtlgggeKLDGz/g1sZgTfKN6rtTDYr27dFHf3983seIL6RP9mZvPc/bZ9bJ9IgzTGL7Jn7wHdzGwIBGWizeyYuhuZ2dfcfZG73wx8TlBCdyvBIyPj/gRcE1ZRxMyOS1p3igXPW21H8ASlN8zsUGCHu/8emEbw6E2RtFCPX2QPwrH6s4G7wjH2lgRPjFpeZ9NfmtmRBL36VwiqLq4FbrLgaVr/Bvw83HdpmPzXAGeE+y8Afgd8HXjE3csteKD2L80sRlCV9UcZaqYUIFXnFMkhMxtL0kVgkWzQUI+ISIFRj19EpMCoxy8iUmCU+EVECowSv4hIgVHiFxEpMEr8IiIF5v8Djh1Aa21TIRkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(1)\n",
    "decaying_epsilon_greedy_solver = DecayingEpsilonGreedy(bandit_10_arm)\n",
    "decaying_epsilon_greedy_solver.run(5000)\n",
    "print('epsilon衰减的贪心算法的累积懊悔为：', decaying_epsilon_greedy_solver.regret)\n",
    "plot_results([decaying_epsilon_greedy_solver], [\"DecayingEpsilonGreedy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hide_input": false,
    "id": "B10E4CB1163B448196345118F1715110",
    "jupyter": {},
    "mdEditEnable": false,
    "notebookId": "5f8f31cc46ba5e003077fa26",
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "从上图我们看到，随着时间做指数衰减的$\\epsilon$-贪心策略能够使得累计懊悔与时间的关系变成次线性 (Sublinear) 的，这明显优于固定$\\epsilon$值的$\\epsilon$-贪心策略。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E2A9124205E14D178D124F342AC7C504",
    "jupyter": {},
    "mdEditEnable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 上置信界算法 (Upper Confidence Bound, UCB)\n",
    "\n",
    "设想这样一种情况：对于一台2臂老虎机，其中第一个拉杆只尝试过一次，得到奖励为0；第二个拉杆尝试过很多次，对它的奖励分布有了大致的把握。这时你会怎么做？或许你会进一步尝试第一个拉杆，来更加确定其奖励分布。这种思路主要是基于不确定性，因为此时第一个拉杆只试了一次，它的不确定性很高。如果一个拉杆它的不确定性越大，那么其就越具有探索的价值，因为探索之后可能发现它的奖励很大。我们在此引入不确定性度量$U(a)$，其会随着一个动作尝试的次数增加而减小。因此可以使用一种基于不确定性的策略为，综合考虑现有的期望奖励估值和不确定性，其核心问题是如何估计不确定性。\n",
    "\n",
    "上置信界算法UCB (Upper Confidence Bound) 是一种经典的基于不确定性的策略算法。它的思想用到了一个非常著名的数学原理：霍夫丁不等式 *(Hoeffding's Inequality)*。在霍夫丁不等式中，令 $X_1,...,X_n$ 为 $n$ 个独立同分布的随机变量，取值范围为 $[0,1]$，其经验期望为 $\\bar{x}_n = \\frac{1}{n}\\sum_{j=1}^{n}X_j$，则有\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\mathbb{P}\\left\\{ \\mathbb{E}\\left[X\\right] \\geq \\bar{x}_t + u \\right\\} \\leq e^{-2nu^2} \n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "现在我们将霍夫丁不等式运用于多臂老虎机问题中。 将$\\hat{Q}_t(a)$ 代入 $\\bar{x}_t$，并认为不等式中的参数 $u = \\hat{U}_t(a)$ 代表不确定性度量。给定一个概率 $p= e^{-2N_t(a)U_t(a)^2}$，根据不等式，$Q_t(a) < \\hat{Q}_t(a) + \\hat{U}_t(a)$ 至少以概率 $1-p$ 成立。当$p$很小时，$Q_t(a) < \\hat{Q}_t(a) + \\hat{U}_t(a)$就以很大概率成立，此时$\\hat{Q}_t(a) + \\hat{U}_t(a)$便是期望奖励的上界。于是，上置信界算法便选取期望奖励上界最大的动作，即$a = \\underset{a\\in\\mathcal{A}}{\\text{argmax}} \\left[\\hat{Q}(a) + \\hat{U}(a)\\right]$。那其中$\\hat{U}_t(a)$具体是什么呢？根据$式子e^{-2N_t(a)U_t(a)^2}$，解之即得 $\\hat{U}_t(a) = \\sqrt{\\frac{-\\log p}{2N_t(a)}}$。所以设定一个概率$p$后，就可以计算相应的不确定性度量$\\hat{U}_t(a)$了。更直观地说，UCB方法在每次选择拉杆前，先估计每根拉杆的期望奖励的上界，使得每根拉杆的期望奖励只有一个较小的概率 $p$ 超过这个上界。接着选出期望奖励上界最大的拉杆，从而选择最有可能是期望奖励最大的拉杆。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2C38EF37C1F14CDF89D77E495360F726",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "我们下面将用代码实现UCB算法，并且仍然使用上文定义的10臂老虎机来观察实验结果。在具体实现过程中，我们设置$p=\\frac{1}{t}$，并且在分母中每个拉杆的次数加上常数1，以免出现除以0的情形，即此时$\\hat{U}_t(a) = \\sqrt{\\frac{\\log t}{2(N_t(a)+1)}}$。并且我们设定一个系数$c$来控制不确定性的比重，此时$a = \\underset{a\\in\\mathcal{A}}{\\text{argmax}} \\left[\\hat{Q}(a) + c \\cdot \\hat{U}(a)\\right]$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "E3617E5B1341466C801F28C2F240FEED",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class UCB(Solver):\n",
    "    \"\"\" UCB算法，继承Solver类 \"\"\"\n",
    "    def __init__(self, bandit, coef, init_prob=1.0):\n",
    "        super(UCB, self).__init__(bandit)\n",
    "        self.total_count = 0\n",
    "        self.estimates = np.array([init_prob] * self.bandit.K)\n",
    "        self.coef = coef\n",
    "\n",
    "    def run_one_step(self):\n",
    "        self.total_count += 1\n",
    "        ucb = self.estimates + self.coef * np.sqrt(np.log(self.total_count) / (2 * (self.counts + 1))) # 计算上置信界\n",
    "        k = np.argmax(ucb) # 选出上置信界最大的拉杆\n",
    "        r = self.bandit.step(k)\n",
    "        self.estimates[k] += 1. / (self.counts[k] + 1) * (r - self.estimates[k])\n",
    "        return k"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "522F36C0E2B54C208DD0F284676E9643",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "上置信界算法的累积懊悔为： 4.360254345326592\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAhbElEQVR4nO3deZRdZZnv8e+PUBAMhEBSSkhIgpFmypKpDCC0HVG7GSLYSi9RVECXETpOLVeu2F5AbOirnfYioIaoXEBRxEsYxIAggoDNVIEAiYEmIEgZJEWADAaQmOf+sd+Cc07tqtqpql3T/n3WOivn7PF5AznPeYf9vooIzMysurYY7ADMzGxwORGYmVWcE4GZWcU5EZiZVZwTgZlZxTkRmJlVnBOBWckkXSLp37rYd6KkOwcojrp7SVov6c0DcW8b2pwIbEiQ9GlJrZJekXRJzv53SXpE0gZJt0qaOghhjigRsW1EPAHdJysb+ZwIbKhYCfwbcHHjDkkTgIXA/wJ2BFqBn/bHTZXxvwOrNP8DsCEhIhZGxDXA6pzd7weWRcTPIuJl4CxgH0l75F1L0g6SrpfULumF9H5yzf7bJJ0j6bfABuDNkkLSP0t6TNI6SV+TNF3SXZLWSrpS0lY115gtaYmkFyX9l6S31uzbT9L96To/BUb3UHxJukDSmlTreVfNjpMkLU/XekLSp2r2zZLUJulUSaskPSPppJr94yVdl+K/F5jecNOQ9BZJc4DjgdNSc9HPe4jXRhgnAhsO9gYe7PgQEX8GHk/b82wB/F9gKjAFeAm4sOGYjwJzgO2Ap9K2w4EDgIOA04AFZF+QuwAzgA8BSNqfrObyKWA8cBFwnaStU7K4BvghWe3lZ8AHeijfgcATwATgTGChpB3TvlXAbGAscBLwf9L9O+wEbA9MAj4BfFvSDmnft4GXgYnAx9Ork4hYAFwOfCM1F723h3hthHEisOFgW2BNw7Y1ZF/inUTE6oi4KiI2RMQ64Bzg7xoOuyQilkXExoh4NW37ekSsjYhlwFLgpoh4IiLWADcA+6XjPglcFBH3RMRfI+JS4BWyBHIQ0AScFxGvRsT/A+7roXyrao7/KfAocFQqyy8i4vHI/Aa4CfjbmnNfBc5O5y4C1gO7SxpFloDOiIg/R8RS4NIe4rCKciKw4WA92S/iWmOBdZKmpOaM9ZLWA0h6g6SLJD0laS1wOzAufTl2eDrnPs/WvH8p5/O26f1U4NTULPSipBfJag07p9cfo342x6foXt7xO6eyHCHpbknPp/scSVZz6LA6IjbWfN6Q4mwGtmwoZ09xWEU5EdhwsAzYp+ODpDFk7d3LIuIPqTlj24jo+KI+FdgdODAixgLv6Di15pp9mXb3aeCciBhX83pDRPwEeAaYJKn2XlN6uF7e8SslbQ1cBcwD3hQR44BFDeXoSjuwkSxBFYnD0xBXmBOBDQmStpQ0GhgFjJI0WtKWaffVwAxJH0jHnAE8FBGPdHG57ch+wb+Y2trP7OdwvwecLOnANOpojKSjJG0H3EX2BfzZVKb3AzN7uN4b0/FNkv4J2JPsC38rYGvSl7qkI4C/LxJgRPyVbKTVWamGtBdwQjenPAv4mYKKciKwoeIrZF/eXwI+kt5/BSAi2snau88BXiDrXD2um2udB2wDPAfcDdzYn4FGRCtZP8GFKZ4VwIlp31/IRjmdmPZ9kOwLuTv3ALuleM8Bjk39HOuAzwJXpmt9GLhuM0L9NFkz0Z+AS8g60LvyA2Cv1NR1zWbcw0YAeWEaM7Nqc43AzKzinAjMzCrOicDMrOKcCMzMKm7Lng/pm/QQTyvZQzOzG/bNAq4Ffp82LYyIs7u73oQJE2LatGn9H6iZ2Qi2ePHi5yKiOW9f6YkA+BywnM5Phna4ozFBdGfatGm0trb2S2BmZlUhqcsny0ttGkozPh4FfL/M+5iZWe+V3UdwHtksjpu6OeZgSQ9KukFS7mySkuYoW7Sktb29vYw4zcwqq7REIGk2sCoiFndz2P3A1IjYB7iAbPreTiJiQUS0RERLc3NuE5eZmfVSmTWCQ4CjJT0JXAEcJulHtQekKX/Xp/eLgCZlq1GZmdkAKS0RRMTpETE5IqaRzQvz64j4SO0xknbqmHVR0swUT94KVWZmVpKBGDVUR9LJABExHzgWOEXSRrJJxo4LT35kZjaght2kcy0tLeHho2Zmm0fS4ohoyds34DWCwbRs5Rp+ufRPgx2GmVmvtEzbkXf8Tf8PmKlUIpj/myf4+YMrUZH1nczMhpiT/266E0FfbfzrJv7mTdty0780rmNuZlZdlZp0LgK2cHXAzKxOpRLBpmHWMW5mNhAqlghcIzAza1SpRADhjmIzswaVSgSuEZiZdVapRBARbOE8YGZWp1KJYFOA24bMzOpVLBG4RmBm1qhSiQDAecDMrF6lEkFWI3AqMDOrValEEOEuAjOzRpVKBJsikDOBmVmdiiUC3FlsZtag9EQgaZSkByRdn7NPks6XtELSQ5L2LzWYALm72MyszkDUCD4HLO9i3xHAbuk1B/humYFsimCLStWBzMx6VurXoqTJwFHA97s45BjgssjcDYyTNLGseALXCMzMGpX9+/g84DRgUxf7JwFP13xuS9vqSJojqVVSa3t7e6+DyTqLe326mdmIVFoikDQbWBURi7s7LGdbp0UDImJBRLREREtzc++XafOkc2ZmnZVZIzgEOFrSk8AVwGGSftRwTBuwS83nycDK0iJyjcDMrJPSEkFEnB4RkyNiGnAc8OuI+EjDYdcBH0ujhw4C1kTEM2XF5BqBmVlnA754vaSTASJiPrAIOBJYAWwATirz3kG4q9jMrMGAJIKIuA24Lb2fX7M9gLkDEQPApk34yWIzswaVGlXvUUNmZp1VKhGAp5gwM2tUqUTgaajNzDqrVCLwNNRmZp1VKhF4Gmozs84qlQgivFSlmVmjaiUC/ECZmVmjSiUCDx81M+usUokgPMWEmVknlUoErhGYmXVWqUQQXqrSzKyTiiWC8JPFZmYNKpUINvmBMjOzTiqVCAJPMWFm1qhSiSCrETgRmJnVKnPN4tGS7pX0oKRlkr6ac8wsSWskLUmvM8qKBzzXkJlZnjIXpnkFOCwi1ktqAu6UdENE3N1w3B0RMbvEOGp4hTIzs0alJYK0+tj69LEpvaKs+xURg3p3M7OhqdQ+AkmjJC0BVgE3R8Q9OYcdnJqPbpC0dxfXmSOpVVJre3t7H2Pq0+lmZiNOqYkgIv4aEfsCk4GZkmY0HHI/MDUi9gEuAK7p4joLIqIlIlqam5t7H0+vzzQzG7kGZNRQRLxItnj94Q3b10bE+vR+EdAkaUKJcfjJYjOzBmWOGmqWNC693wZ4N/BIwzE7KY3nlDQzxbO6rJiy+5R5dTOz4afMUUMTgUsljSL7gr8yIq6XdDJARMwHjgVOkbQReAk4LnUyl8JNQ2ZmnZU5aughYL+c7fNr3l8IXFhWDHlcITAzq1epJ4s9fNTMrLOKJQIvXm9m1qhSicDMzDqrVCJwy5CZWWeVSgTg4aNmZo02KxFI2kLS2LKCKZ2rBGZmnfSYCCT9WNJYSWOA3wGPSvpi+aH1v8BrFpuZNSpSI9grItYC7wMWAVOAj5YZVJncNGRmVq9IImhK6wm8D7g2Il4tN6TylPjQspnZsFUkEVwEPAmMAW6XNBVYU2ZQZXKFwMysXpFE8POImBQRR6Z5gP4AfLzkuErh+oCZWWdFEsFVtR9SMriinHDK5z4CM7N6XU46J2kPYG9ge0nvr9k1FhhddmBlcBeBmVln3c0+ujswGxgHvLdm+zrgkyXGVJrAcw2ZmTXqMhFExLXAtZIOjoi7BjCmUjkNmJnVK9JHsFrSLZKWAkh6q6Sv9HSSpNGS7k0L0y+T9NWcYyTpfEkrJD0kaf9elKEwNw2ZmXVWJBF8DzgdeBVeW3DmuALnvQIclham3xc4XNJBDcccAeyWXnOA7xYLuw9cJTAzq1MkEbwhIu5t2Laxp5Misz59bEqvxt/kxwCXpWPvBsZJmlggpl5xhcDMrLMiieA5SdNJ36OSjgWeKXJxSaMkLQFWATdHxD0Nh0wCnq753Ja2lSM815CZWaMiaxbPBRYAe0j6I/B74PgiF4+IvwL7ShoHXC1pRkQsrTkk71u50w93SXPImo6YMmVKkVt3yYOGzMzqdVsjkDQKOCUi3g00A3tExKER8dTm3CQiXgRuAw5v2NUG7FLzeTKwMuf8BRHREhEtzc3Nm3Pr+uu4ccjMrJNuE0H6RX9Aev/niFhX9MKSmlNNAEnbAO8GHmk47DrgY2n00EHAmogo1OzUW64QmJnVK9I09ICk64CfAX/u2BgRC3s4byJwaapVbAFcGRHXSzo5nT+fbFrrI4EVwAbgpM0vQnEePmpm1lmRRLAjsBo4rGZbAN0mgjTMdL+c7fNr3gdZH8SACNxHYGbWqMdEEBGl/kofaB41ZGZWr8dEIOn8nM1rgNY0DcWw4YVpzMw6K/IcwWiyJ4MfS6+3kjUXfULSeaVFVhI3DZmZ1SvSR/AWsqkiNgJI+i5wE/Ae4OESY+t3rg+YmXVWpEYwiWyZyg5jgJ3T0NJXSomqRK4QmJnVK1Ij+AawRNJtZN+j7wDOlTQG+FWJsfU7dxGYmXVWZNTQDyQtAmaSJYIvR0TH079fLDO4UriTwMysTo9NQ8qW9HoXsE9EXANsKWlm2YGVxWnAzKxekT6C7wAHAx9Kn9cB3y4topJ46KiZWb4ifQQHRsT+kh4AiIgXJG1VclylccuQmVm9IjWCV9N8QR3rETQDm0qNqgSuEJiZ5SuSCM4HrgbeKOkc4E7g3FKjKkFHHvAUE2Zm9bptGpK0BdlCNKeRdRgLeF9ELB+A2ErhpiEzs3rdJoKI2CTpPyPiYDqvJTCsuLPYzCxfkaahmyR9IA0jHfZGRCHMzPpRkVFDXyCbVmKjpJfJvksjIsaWGlk/c33AzCxfjzWCiNguIraIiK0iYmz63GMSkLSLpFslLZe0TNLnco6ZJWmNpCXpdUZvC9KTjpahkVGvMTPrP0VqBL21ETg1Iu6XtB2wWNLNEfG7huPuiIjZJcZRZ4S0cJmZ9ZsifQS9EhHPRMT96f06YDnZTKaDItw4ZGaWq7REUEvSNLL1i+/J2X2wpAcl3SBp7y7OnyOpVVJre3t7maGamVVOoUQg6VBJJ6X3zZJ2LXoDSdsCVwGfj4i1DbvvB6ZGxD7ABcA1edeIiAUR0RIRLc3NzUVv3XCNXp1mZjbiFZl99EzgfwKnp01NwI+KXFxSE1kSuDwiFjbuj4i1EbE+vV8ENEmaUDD2XnEXgZlZvSI1gn8Ejgb+DJDWItiup5PScwc/AJZHxDe7OGanjucT0tTWWwCri4VuZmb9ocioob9EREjqmHRuTE8nJIcAHwUelrQkbfsyMAUgIuYDxwKnSNoIvAQcFyU9Avza8FE/UmZmVqdIIrhS0kXAOEmfBD4OfK+nkyLiTnp4kDciLgQuLBJof3HTkJlZvSJLVc6T9B5gLbA7cEZE3Fx6ZP3Mw0fNzPL1mAgk/Qvws+H45Z/HFQIzs3pFOovHAr+UdIekuZLeVHZQZfDwUTOzfEXmGvpqROwNzAV2Bn4j6VelR9bPXluYxlUCM7M6m/Nk8SrgT2TDO99YTjjl86ghM7N6RR4oO0XSbcAtwATgkxHx1rID629emMbMLF+R4aNTyaaHWFJyLAPCTUNmZvW6TASSxqa5gb6RPu9Yuz8ini85tn7l+oCZWb7uagQ/BmYDi8m+R2t/Swfw5hLjMjOzAdJlIuhYLCYiCs80OpS5i8DMLF+RzuJbimwb8l5bqtKdBGZmtbrrIxgNvAGYIGkHXm8aGkv2PMGw5DRgZlavuz6CTwGfJ/vSX8zr36FrgW+XG1b/81xDZmb5uusj+BbwLUmfiYgLBjCmUrllyMysXpHZRy+QNAPYCxhds/2yMgPrb+4sNjPLV3SpygvS651kzxUcXeC8XSTdKmm5pGWSPpdzjCSdL2mFpIck7d+LMhTy2lxDZd3AzGyYKjLX0LHAu4A/RcRJwD7A1gXO2wicGhF7AgcBcyXt1XDMEcBu6TUH+G7RwHvLo4bMzOoVSQQvRcQmYKOksWSTz/X4MFlEPBMR96f364DlwKSGw44BLovM3WSroE3crBIU5LmGzMzyFUkErZLGkS1PuRi4H7h3c24iaRqwH3BPw65JwNM1n9vonCyQNEdSq6TW9vb2zbl1Tix9Ot3MbMQp0ln8z+ntfEk3AmMj4qGiN5C0LXAV2cR1axt3590yJ4YFwAKAlpaWXv20d33AzCxfdw+UddlxK2n/jmaf7khqIksCl0fEwpxD2oBdaj5PBlb2dN3e6GgZcoXAzKxedzWC/+xmXwCHdXdhZb2yPwCWR8Q3uzjsOuDTkq4ADgTWRMQz3V23z9w2ZGZWp7sHyt7Zx2sfAnwUeFjSkrTty8CUdP35wCLgSGAFsAE4qY/37JKfLDYzy9djH4Gkj+Vt7+mBsoi4kx5aYiIbyjO3pxj6k+sDZmb1iqxQ9raa96PJnim4HxhWTxa7QmBmlq/IqKHP1H6WtD3ww9IiKpm7CMzM6hV5jqDRBrIngYcVVwjMzPIV6SP4Oa9/j25BNvnclWUGVYbXh4+6SmBmVqtIH8G8mvcbgacioq2keErnpiEzs3pF+gh+A5DmGdoyvd8xIp4vObZ+5eGjZmb5ijQNzQG+BrwEbCIbgRkUmHhuKHKFwMysXpGmoS8Ce0fEc2UHUyZPPmpmlq/IqKHHyUYKDWuvLUzjKoGZWZ0iNYLTgf+SdA/wSsfGiPhsaVGVyKOGzMzqFUkEFwG/Bh4m6yMYlrwwjZlZviKJYGNEfKH0SAaKKwRmZnWK9BHcmlYImyhpx45X6ZH1M1cIzMzyFakRfDj9eXrNNg8fNTMbIYo8ULbrQAQyUORhQ2ZmdUpbj0DSxcBsYFVEzMjZPwu4Fvh92rQwIs7uKZ7ectOQmVm+MtcjuAS4sIfj7oiI2QVi6DeuD5iZ1SttPYKIuF3StN6H1r8815CZWb7BXo/gYEkPSrpB0t79dM1uuYvAzKzeYK5HcD8wNSLWSzoSuIYuEkya+G4OwJQpU3p1M/cRmJnlG7T1CCJibc37RZK+I2lC3uR2EbEAWADQ0tLSq690zzVkZpavy0Qg6S3AmzrWI6jZ/reSto6Ix/tyY0k7Ac9GREiaSVbbWN2Xaxa6r7uLzczqdNdHcB6wLmf7S2lftyT9BLgL2F1Sm6RPSDpZ0snpkGOBpZIeBM4HjosSJwTyXENmZvm6axqaFhEPNW6MiNYio4Ei4kM97L+QbHjpgHLTkJlZve5qBKO72bdNfwdSNtcHzMzydZcI7pP0ycaNkj4BLC4vpHK4ZcjMLF93TUOfB66WdDyvf/G3AFsB/1hyXKXxXENmZvW6TAQR8SzwdknvBDrmCvpFRPx6QCLrd64SmJnlKTLFxK3ArQMQy4BwfcDMrF5vppgYltxHYGaWrzqJIP3pLgIzs3qVSQQd/GSxmVm9yiQCNw2ZmeWrTCLo4KYhM7N6lUkEXpjGzCxfZRJBB1cIzMzqVSYRuI/AzCxf5RKB+wjMzOpVJhG8zpnAzKxWZRKBO4vNzPKVlggkXSxplaSlXeyXpPMlrZD0kKT9y4ql/r4DcRczs+GjzBrBJcDh3ew/AtgtveYA3y0xFncWm5l1obREEBG3A893c8gxwGWRuRsYJ2liWfF0cIXAzKzeYPYRTAKervnclrZ1ImmOpFZJre3t7X26qRemMTOrN5iJIO8bObcBJyIWRERLRLQ0Nzf36mZuGjIzyzeYiaAN2KXm82RgZdk3dX3AzKzeYCaC64CPpdFDBwFrIuKZsm7m4aNmZvl6XKqytyT9BJgFTJDUBpwJNAFExHxgEXAksALYAJxUVizZPTviKvMuZmbDT2mJICI+1MP+AOaWdf+uOBGYmdWr0JPFZmaWpzKJoIOXqjQzq1eZRBAeP2pmlqsyieA1rhCYmdWpTCJwfcDMLF91EkHH8NHBDcPMbMipTCLo4LmGzMzqVSgRuHHIzCxPhRJBxvUBM7N6lUkEHj1qZpavOokg/ekuAjOzepVJBB38ZLGZWb3KJAI3DZmZ5atMIujgpiEzs3qVSQSea8jMLF9p6xEASDoc+BYwCvh+RPzvhv2zgGuB36dNCyPi7FJjKvPiZjbkvfrqq7S1tfHyyy8PdiilGD16NJMnT6apqanwOWWuUDYK+DbwHrL1ie+TdF1E/K7h0DsiYnZZcXRwfcDMANra2thuu+2YNm3aiJtpICJYvXo1bW1t7LrrroXPK7NpaCawIiKeiIi/AFcAx5R4v2691jI0sv67m9lmevnllxk/fvyISwKQTaEzfvz4za7tlJkIJgFP13xuS9saHSzpQUk3SNo770KS5khqldTa3t7ep6A8fNTMRmIS6NCbspWZCPKiaWyhuR+YGhH7ABcA1+RdKCIWRERLRLQ0Nzf3Kphw45CZWa4yE0EbsEvN58nAytoDImJtRKxP7xcBTZImlBiTh4+a2aB78sknmTFjRt22s846i3nz5gEwb9489thjD2bMmME+++zDZZddBsCsWbPYfffd2Xfffdlzzz1ZsGBBv8RT5qih+4DdJO0K/BE4Dvhw7QGSdgKejYiQNJMsMa0uJRpXCMxsGJg/fz4333wz9957L2PHjmXNmjVcc801r+2//PLLaWlp4fnnn2f69OmceOKJbLXVVn26Z2mJICI2Svo08Euy4aMXR8QySSen/fOBY4FTJG0EXgKOi5IG/Luv2MwaffXny/jdyrX9es29dh7Lme/N7e4s5Nxzz+XWW29l7NixAGy//faccMIJnY5bv349Y8aMYdSoUb2+V4dSnyNIzT2LGrbNr3l/IXBhmTE0GsmdRGY2vL300kusW7eO6dOnd3nM8ccfz9Zbb81jjz3GeeedN/QTwVDiB4vNrFFffrn3RVc/SDdt2tTjj9WOpqH29nbe/va3c/jhhzN16tQ+xVOZKSY6uEJgZoNt/PjxvPDCC3Xbnn/+eaZNm8aYMWN44oknerxGc3Mz+++/P/fcc0+f46lMIvDwUTMbKrbddlsmTpzILbfcAmRJ4MYbb+TQQw/l9NNPZ+7cuaxdm/VdrF27Nnd00IYNG3jggQe6bUYqqjJNQw/84UXAncVmNjRcdtllzJ07l1NPPRWAM888k+nTp3PKKaewfv163va2t9HU1ERTU9Nrx0DWR7DNNtvwyiuvcOKJJ3LAAQf0ORYNt1k5W1paorW1dbPPW/zUC/z0vj9w5nv3ZszWlcl/ZtZg+fLl7LnnnoMdRqnyyihpcUS05B1fmW/EA6buwAFTdxjsMMzMhpzK9BGYmVk+JwIzq5zh1iS+OXpTNicCM6uU0aNHs3r16hGZDDrWIxg9evRmnVeZPgIzM4DJkyfT1tZGX6e0H6o6VijbHE4EZlYpTU1Nm7V6VxW4acjMrOKcCMzMKs6JwMys4obdk8WS2oGnenn6BOC5fgxnOHCZq8Flroa+lHlqROSu9TvsEkFfSGrt6hHrkcplrgaXuRrKKrObhszMKs6JwMys4qqWCDpP6j3yuczV4DJXQyllrlQfgZmZdVa1GoGZmTVwIjAzq7jKJAJJh0t6VNIKSV8a7Hj6QtLFklZJWlqzbUdJN0t6LP25Q82+01O5H5X0DzXbD5D0cNp3vqQhuZKnpF0k3SppuaRlkj6Xto/kMo+WdK+kB1OZv5q2j9gyd5A0StIDkq5Pn0d0mSU9mWJdIqk1bRvYMkfEiH8Bo4DHgTcDWwEPAnsNdlx9KM87gP2BpTXbvgF8Kb3/EvD19H6vVN6tgV3T38OotO9e4GCypZxvAI4Y7LJ1Ud6JwP7p/XbAf6dyjeQyC9g2vW8C7gEOGsllrin7F4AfA9eP9P+3U6xPAhMatg1omatSI5gJrIiIJyLiL8AVwDGDHFOvRcTtwPMNm48BLk3vLwXeV7P9ioh4JSJ+D6wAZkqaCIyNiLsi+7/osppzhpSIeCYi7k/v1wHLgUmM7DJHRKxPH5vSKxjBZQaQNBk4Cvh+zeYRXeYuDGiZq5IIJgFP13xuS9tGkjdFxDOQfXECb0zbuyr7pPS+cfuQJmkasB/ZL+QRXebURLIEWAXcHBEjvszAecBpwKaabSO9zAHcJGmxpDlp24CWuSrrEeS1lVVl3GxXZR92fyeStgWuAj4fEWu7aQIdEWWOiL8C+0oaB1wtaUY3hw/7MkuaDayKiMWSZhU5JWfbsCpzckhErJT0RuBmSY90c2wpZa5KjaAN2KXm82Rg5SDFUpZnU/WQ9OeqtL2rsrel943bhyRJTWRJ4PKIWJg2j+gyd4iIF4HbgMMZ2WU+BDha0pNkzbeHSfoRI7vMRMTK9Ocq4GqypuwBLXNVEsF9wG6SdpW0FXAccN0gx9TfrgNOSO9PAK6t2X6cpK0l7QrsBtybqpvrJB2URhd8rOacISXF9wNgeUR8s2bXSC5zc6oJIGkb4N3AI4zgMkfE6RExOSKmkf0b/XVEfIQRXGZJYyRt1/Ee+HtgKQNd5sHuMR+oF3Ak2WiTx4F/Hex4+liWnwDPAK+S/RL4BDAeuAV4LP25Y83x/5rK/Sg1IwmAlvQ/3ePAhaQnzYfaCziUrJr7ELAkvY4c4WV+K/BAKvNS4Iy0fcSWuaH8s3h91NCILTPZSMYH02tZx3fTQJfZU0yYmVVcVZqGzMysC04EZmYV50RgZlZxTgRmZhXnRGBmVnFOBDaiSRqfZnVcIulPkv6Y3q+X9J0BimFfSUcOxL3MeqMqU0xYRUXEamBfAElnAesjYt4Ah7Ev2RjvRQN8X7NCXCOwSpI0q2a++7MkXSrppjQ3/PslfSPN7X5jmt6iY77336TJwX7ZMQVAw3X/SdJSZesI3J6eZD8b+GCqiXwwPU16saT7lM27f0w690RJ16Z7PirpzLR9jKRfpGsulfTBgfubsipwjcAsMx14J9l873cBH4iI0yRdDRwl6RfABcAxEdGevozPAT7ecJ0zgH+IiD9KGhcRf5F0BtASEZ8GkHQu2fQJH0/TSNwr6Vfp/JnADGADcF+671RgZUQclc7fvrS/BaskJwKzzA0R8aqkh8kWMroxbX8YmAbsTvYFfXOa9XQU2TQfjX4LXCLpSmBhzn7I5pM5WtL/SJ9HA1PS+5tTcxaSFpJNr7EImCfp62TTLtzR61Ka5XAiMMu8AhARmyS9Gq/PvbKJ7N+JgGURcXB3F4mIkyUdSLa4yhJJ++YcJrIax6N1G7PzGud8iYj4b0kHkM2v9O+SboqIszezfGZdch+BWTGPAs2SDoZsWmxJezceJGl6RNwTEWcAz5FNGbyObInNDr8EPpNmiUTSfjX73qNsvdptyFaY+q2knYENEfEjYB7ZMqVm/cY1ArMCUlv/scD5qY1+S7LVtJY1HPofknYj+9V/C9mskn8AvqRstbF/B76Wzn0oJYMngdnp/DuBHwJvAX4cEa3KFij/D0mbyGacPaWkYlpFefZRsyFC0onUdCqbDRQ3DZmZVZxrBGZmFecagZlZxTkRmJlVnBOBmVnFORGYmVWcE4GZWcX9fx8fIfPIsX9bAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(1)\n",
    "coef = 0.1 # 控制不确定性比重的系数\n",
    "UCB_solver = UCB(bandit_10_arm, coef)\n",
    "UCB_solver.run(5000)\n",
    "print('上置信界算法的累积懊悔为：', UCB_solver.regret)\n",
    "plot_results([UCB_solver], [\"UCB\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FA2EA57FC904479F8FAEADBA28D7E46F",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 汤普森采样算法 (Thompson Sampling) \n",
    "\n",
    "MAB中还有一种经典算法为汤普森采样。我们先假设每个拉杆的奖励服从一个特定的分布，然后根据每个拉杆的期望奖励来进行选择。但是由于计算每个拉杆的期望奖励计算代价比较高，汤普森采样算法使用采样的方式，即根据当前每个动作 $a$ 的奖励分布进行一轮采样，得到一组各个拉杆的奖励样本，再选择样本中奖励最大的动作。我们可以看出，汤普森采样是一种计算每个拉杆产生最高奖励概率的蒙特卡罗采样方法。\n",
    "\n",
    "了解了汤普森采样的基础思路后，需要解决另一个问题：当前每个动作 $a$ 的奖励分布怎样得到并且在过程中进行更新？在实际情况中，我们通常对当前每个动作的奖励分布用Beta分布进行建模。具体来说，若某拉杆选择了$k$次，其中$m_1$次奖励为1，$m_2$次奖励为0，则该拉杆的奖励服从参数为 $(m_1+1, m_2+1)$ 的Beta分布。下图是汤普森采样的一个示例。\n",
    "\n",
    "<img src=\"https://staticcdn.boyuai.com/user-assets/358/5vUaGksi15BtPE8DyAJozj/qg2ueouwk0.png!png\" width = \"600\" height = \"600\" align=center />\n",
    " \n",
    "<center>汤普森采样示例 </center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "82E18B61B9524D68944470E70744AC45",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "我们下面将用代码实现汤普森采样算法，并且仍然使用上文定义的10臂老虎机来观察实验结果。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "2B1F199E2D8A44968730D4D7C7D2C086",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class ThompsonSampling(Solver):\n",
    "    \"\"\" 汤普森采样算法，继承Solver类 \"\"\"\n",
    "    def __init__(self, bandit):\n",
    "        super(ThompsonSampling, self).__init__(bandit)\n",
    "        self._a = np.ones(self.bandit.K)  # 列表，表示每个拉杆奖励为1的次数\n",
    "        self._b = np.ones(self.bandit.K)  # 列表，表示每个拉杆奖励为0的次数\n",
    "\n",
    "    def run_one_step(self):\n",
    "        samples = np.random.beta(self._a, self._b) # 按照Beta分布采样一组奖励\n",
    "        k = np.argmax(samples) # 选出采样数值最大的拉杆\n",
    "        r = self.bandit.step(k)\n",
    "        \n",
    "        self._a[k] += r # 更新Beta分布的第一个参数\n",
    "        self._b[k] += (1 - r) # 更新Beta分布的第二个参数\n",
    "        return k"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "A0A37B905FEA4C81937D2F215038586F",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "汤普森采样算法的累积懊悔为： 57.19161964443925\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEWCAYAAABhffzLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAutElEQVR4nO3deXxU5fXH8c9JCIQl7GHfF9lc2IoiVLG44FJxQat1wWIVt1q1arG/1mp/9We1dlGrVqxatFbFiorWWimIKKIsiuwUwYBIWGQJhJCQ5fz+uDcxBEgmMZNJZr7v1yuvzDwz997zQHLyzHPvPY+5OyIikjiSYh2AiIjULCV+EZEEo8QvIpJglPhFRBKMEr+ISIJR4hcRSTBK/CJRYGZ/NbNfH+a1K8zs/RqK44BjmVm2mfWoiWNL7aXELzFjZjeY2UIzyzOzvx7i9dFmtsrMcszsHTPrGoMw44q7N3H3dVD+HyeJb0r8EkubgF8DT5V9wcxaA9OAXwAtgYXAi9VxUAvoZ18Sln74JWbcfZq7vwpsP8TL5wHL3f0ld88F7gKOMbO+h9qXmbUwszfMbJuZ7Qwfdyr1+mwzu8fM5gI5QA8zczO7zszWmNkeM/tfM+tpZvPMbLeZTTWz+qX2cZaZLTazXWb2gZkdXeq1QWb2cbifF4HUCrpvZvawmWWFn2pGl3rhB2a2MtzXOjObWOq1UWa20cx+YmZbzSzTzH5Q6vVWZjY9jH8+0LPMQd3MepnZ1cAlwO3h9M/rFcQrcUSJX2qrAcCnxU/cfS+wNmw/lCTgaaAr0AXYB/ypzHsuA64G0oD1YdsYYAhwHHA7MJkgIXYGjgQuBjCzwQSfTCYCrYDHgelm1iD84/Aq8CzBp5OXgPMr6N+xwDqgNfBLYJqZtQxf2wqcBTQFfgD8ITx+sXZAM6AjcCXwiJm1CF97BMgF2gMTwq+DuPtk4Dng/nD657sVxCtxRIlfaqsmQFaZtiyCpH0Qd9/u7i+7e4677wHuAU4s87a/uvtydy9w9/yw7T533+3uy4FlwNvuvs7ds4B/AYPC910FPO7uH7l7obtPAfII/mAcB6QAf3T3fHf/B7Cggv5tLfX+F4HVwJlhX/7p7ms98C7wNvDtUtvmA78Kt30TyAb6mFkywR+cO919r7svA6ZUEIckICV+qa2yCUa8pTUF9phZl3B6ItvMsgHMrJGZPW5m681sNzAHaB4mw2JfHOI4W0o93neI503Cx12Bn4TTPLvMbBfBp4IO4deXfmDFw/WU71Dv7xD25XQz+9DMdoTHOYPgk0Gx7e5eUOp5ThhnOlCvTD8rikMSkBK/1FbLgWOKn5hZY4L56uXuviGcnmji7sWJ+SdAH+BYd28KnFC8aal9fpNStF8A97h781Jfjdz9eSAT6GhmpY/VpYL9Her9m8ysAfAy8ADQ1t2bA2+W6cfhbAMKCP4gRRKHSvMmKCV+iRkzq2dmqUAykGxmqWZWL3z5FeBIMzs/fM+dwBJ3X3WY3aURjNB3hXPlv6zmcJ8ArjGzY8Orghqb2ZlmlgbMI0i4N4Z9Og8YVsH+2oTvTzGzC4B+BAm+PtCAMImb2enAqZEE6O6FBFdC3RV+AuoPjC9nky2ArulPQEr8Eks/J0jWk4BLw8c/B3D3bQTz1fcAOwlOhl5Uzr7+CDQEvgI+BN6qzkDdfSHBPP+fwng+A64IX9tPcBXSFeFr3yNIwOX5COgdxnsPMC48T7EHuBGYGu7r+8D0SoR6A8G0z2bgrwQnvA/nSaB/OHX1aiWOIXWcaSEWEZHEohG/iEiCiWriN7PmZvaP8AaVlWY23MxamtmM8KaZGaWuPxYRkRoQ7RH/g8Bb7t6X4AqNlQTzuTPdvTcwM3wuIiI1JGpz/GbWlODOyx6lr1c2s9XAKHfPNLP2wGx37xOVIERE5CD1Kn5LlfUguCTtaTM7BlgE/Jjg2uRMgDD5tznUxmEtkasBGjduPKRv30OWaBERkcNYtGjRV+6eXrY9miP+oQSX1Y1w94/M7EFgN/Cj8KaU4vftdPdy5/mHDh3qCxcujEqcIiLxyswWufvQsu3RnOPfCGx094/C5/8ABgNbwikewu9boxiDiIiUEbXE7+6bgS/MrHj+fjSwguBmlOK7CccDr0UrBhEROVg05/gBfgQ8F5atXUdQYjYJmGpmVwIbgAuiHIOIiJQS1cTv7ouBg+aXCEb/30h+fj4bN24kNzf3m+5K4khqaiqdOnUiJSUl1qGI1FrRHvFHzcaNG0lLS6Nbt24cWORQEpW7s337djZu3Ej37t1jHY5IrVVnSzbk5ubSqlUrJX0pYWa0atVKnwJFKlBnEz+gpC8H0c+ESMXq7FSPiEi8WroxixkrNgNw7uBOdG/duFr3r8RfRdu3b2f06OAc9ebNm0lOTiY9PZ2MjAw6dOjAihUrYhxh5RQVFXHTTTcxa9YszIzU1FSmTp0a1bnybt26sXDhQlq3bs3xxx/PBx98ELVjidQm27PzeGf1NooOcwPtlA8yWL5pN2YwuGsLJf7aolWrVixevBiAu+66iyZNmnDrrbeSkZHBWWedFdvgquDFF19k06ZNLFmyhKSkJDZu3EjjxtX7w1YeJX1JJE++/zmPzl5b7nvGDenEAxccU+57qkqJPwoKCwu56qqr+OCDD+jYsSOvvfYaDRs2ZPHixVxzzTXk5OTQs2dPnnrqKVq0aMGoUaMYNGgQixYtYtu2bTzzzDPce++9LF26lO9973v8+te/JiMjgzFjxnDsscfyySefcMQRR/DMM8/QqFEjJk2axPTp06lXrx6nnnoqDzzwAOvXr2fChAls27aN9PR0nn76abp06cIVV1xB06ZNWbhwIZs3b+b+++9n3LhxZGZm0r59e5KSgtM+nTp1KunPtddey4IFC9i3bx/jxo3j7rvvBoIR+/e//33eeecd8vPzmTx5MnfccQefffYZt912G9dccw2zZ8/mzjvvpFWrVqxevZoTTjiBRx99tOQ4xZo0aUJ2djazZ8/mrrvuonXr1ixbtowhQ4bwt7/9DTPjzTff5JZbbqF169YMHjyYdevW8cYbb9Tcf6xIJa3M3M3rn246qH326m2kpzXgleuOP+y27ZqmRi2uuEj8d7++nBWbdlfrPvt3aMovvzugStuuWbOG559/nieeeIILL7yQl19+mUsvvZTLL7+chx9+mBNPPJE777yTu+++mz/+8Y8A1K9fnzlz5vDggw8yduxYFi1aRMuWLenZsyc333wzAKtXr+bJJ59kxIgRTJgwgUcffZQJEybwyiuvsGrVKsyMXbt2AXDDDTdw+eWXM378eJ566iluvPFGXn31VQAyMzN5//33WbVqFWeffTbjxo3jwgsvZOTIkbz33nuMHj2aSy+9lEGDBgFwzz330LJlSwoLCxk9ejRLlizh6KOPBqBz587MmzePm2++mSuuuIK5c+eSm5vLgAEDuOaaawCYP38+K1asoGvXrowZM4Zp06Yxbty4w/77ffLJJyxfvpwOHTowYsQI5s6dy9ChQ5k4cSJz5syhe/fuXHzxxVX6v5G6bfmmLN7977ZYhxGxfy7JZPmm3aQkH3zRwSn929KpRaMYRBUnib+26d69OwMHDgRgyJAhZGRkkJWVxa5duzjxxBMBGD9+PBdc8PVNy2effTYARx11FAMGDKB9+/YA9OjRgy+++ILmzZvTuXNnRowYAcCll17KQw89xE033URqaio//OEPOfPMM0ummebNm8e0acGyr5dddhm33357ybHOOecckpKS6N+/P1u2bAGCEf7q1auZNWsWs2bNYvTo0bz00kuMHj2aqVOnMnnyZAoKCsjMzGTFihUlib903NnZ2aSlpZGWlkZqamrJH6Fhw4bRo0ewpvfFF1/M+++/X27iHzZsWMknjoEDB5KRkUGTJk3o0aNHyTmHiy++mMmTJ1f6/0Zqh8KiqhWHvO+t1cypQ4kf4LzBHfn9hQNjHcYB4iLxV3VkHi0NGjQoeZycnMy+ffsi3iYpKemA7ZOSkigoKAAOvlTRzKhXrx7z589n5syZvPDCC/zpT39i1qxZB+2/9Lal91+6OmuDBg04/fTTOf3002nbti2vvvoqPXr04IEHHmDBggW0aNGCK6644oDr5KsadyT/FhD8+xUUFKC1oePHX+d+zl2vV/3ih9F92/DIJYOrMaLoalCv9l01HxeJvy5o1qwZLVq04L333uPb3/42zz77bMnoP1IbNmxg3rx5DB8+nOeff56RI0eSnZ1NTk4OZ5xxBscddxy9evUC4Pjjj+eFF17gsssu47nnnmPkyJHl7vvjjz+mXbt2dOjQgaKiopLpnN27d9O4cWOaNWvGli1b+Ne//sWoUaMqFff8+fP5/PPP6dq1Ky+++CJXX311pbYH6Nu3L+vWrSMjI4Nu3brx4osvVnofUnP25OYz97OvKHL49Rsr2JGzHyP4g7+/sIhWjesz/vhuVdr3yf3akpqSXI3RJh4l/ho0ZcqUkpO7PXr04Omnn67U9v369WPKlClMnDiR3r17c+2115KVlcXYsWPJzc3F3fnDH/4AwEMPPcSECRP47W9/W3Jytzxbt27lqquuIi8vDwimW2644QZSU1MZNGgQAwYMoEePHiVTTZUxfPhwJk2axNKlSznhhBM499xzK72Phg0b8uijjzJmzBhat27NsGHDKr2PRJebX0heQVHJ8+c+Ws/H63dG5Vj/WXlgtfV2TVM5e2CHkufHdm/J6H5to3JsqVjUFmKpTodaiGXlypX069cvRhHVvOLLRJctWxbrUCpl9uzZPPDAA9Vy9U12djZNmjTB3bn++uvp3bt3yYnv0hLtZyMSX+zI4Tu/m01+4cG/7wM6NI3KMbu2asSPRx9BchJ0b92E5CTdVV3TDrcQi0b8Umc88cQTTJkyhf379zNo0CAmTpwY65DqhH8v38zEZxcBMPHEHrRJCy4TNGDMke3o0LxhDKOTWNCIX+JOov9srN2WzV3Tl5NfWMTOvfms3rIHgLOObs9DFw0iSSPvhBGXI353V1EuOUBdGMhEwt3ZkxdcFVVU5PzfmyvZsTc/om037sxh1eY9fKtbC5o1SuHY7i35+Zn9OapTs2iGLHVInU38qampbN++XaWZpURxPf7U1Ojd8RgNh/pj9dOXlzB14caD2vu3r3g+PsmMs45uz8MXD9LvhhxSnU38nTp1YuPGjWzbVrdu5pDoKl6Bq67YnJXLyb9/l+xwdF9a//ZNOW9wRwAapCRzwZBOuoxRqkWdTfwpKSlaZUnqvM+/2kt2XgEXDOlExxYHnmQ9bUA7+kUwwheprDqb+EVqq925+fxnxZaIyhI8OHMNAD8Y0Z3+UbqsUqQsJX6RavbIrM94fM66iN9fPzmJvu3SohiRyIGU+EWqyY69+3nknc948v3PSU9rwLRrD19yt7TWTRroEkupUUr8ItXkjmlL+PfyoNrpz8/sR+eWsSm5K1IRJX6RavLOqm00rp/MkrtOU3kCqdVqX71QkTooKyef/YVFnNS3jZK+1HpK/CLf0Je79jH0nhlAcAmmSG0X1akeM8sA9gCFQIG7DzWzlsCLQDcgA7jQ3aNTG1YkyoqKnF+8uoz8QuekPumcrFLDUgfUxBz/Se7+Vannk4CZ7v4bM5sUPv9pDcQhUq3ufn05T8/NKHn++GVDqV8LV1sSKSsWJ3fHAqPCx1OA2SjxSx1RWORs2Z3LG0s2lST9W089gguGdlbSlzoj2onfgbfNzIHH3X0y0NbdMwHcPdPM2hxqQzO7GrgaoEuXLlEOU6Ri2/bkcfqD7/FVdl5J2yvXHc+gLi1iGJVI5UU78Y9w901hcp9hZqsi3TD8IzEZgnr80QpQpCI5+ws479EPWLU5qGvftVUjrhvVk6HdWtIzvUmMoxOpvKgmfnffFH7famavAMOALWbWPhzttwe2lrsTkRj6YkcO4/78AVt259GxeUMuOa4L157YU+WOpU6LWuI3s8ZAkrvvCR+fCvwKmA6MB34Tfn8tWjGIfBO5+YWc+oc57MsvpHvrxsy85USVVpC4EM0Rf1vglXBkVA/4u7u/ZWYLgKlmdiWwAbggijGIVNkvXl3GvvxCjurYjNd/NDLW4YhUm6glfndfBxxziPbtwOhoHVekOmTnFfDSomAFrH9cOzzG0YhUL11/JnII5z4yF4AbTupFg3pa9UriixK/SBnvrNrKmq3ZNG+Uwq2n9Yl1OCLVTtU5RYA/zVrDlHnrgeB6fYDp12teX+KTEr8I8N6arzBgdFhr55hOzejSSvX0JT4p8UtCWb99L+u35wAwec46Vm3eDRi7cvYzqk869553VGwDFKkBSvySUC6e/CGbsnJLnpvBxcOCkiBnH9MhVmGJ1Cglfol7m7Nyw5E9bN2Tx3mDO3LJsUGy79e+KY3q69dAEot+4iWurd++lxN/O/uAtoGdmzOka8vYBCRSCyjxS9y66YVPeHXxJgAuObYL5w/pRL0ko3/7pjGOTCS2lPglLn2xI6ck6T988SC+q/l7kRJK/BJ3XvlkIze/+CkAj10ymNOPah/jiERqFyV+iRubs3L53durS2rsTBjRnTFHavFzkbKU+CVunPvoXDLDSzXn3HaSbsASOQwlfqnTsvMKWPD5Dh6cuYbMrFwuGNKJ28f0JT2tQaxDE6m1lPilzvpkw07Of+wDisKFOZs0qMeNo3sr6YtUQIlf6qStu3M599EPADi5X1tuHN2LI9qmkZqiEsoiFVHilzpnf0ERI+9/B4BbTjmCG0f3jnFEInWL6vFLnfPL6cvZX1DEkR2bKumLVIFG/FIn5BUUcskTH5GZlcuXu/YB8NyVx8U4KpG6SYlf6oTnPtzAwvU7Ob5nK0b0asX1J/WiWaOUWIclUicp8Uut9ru3V/Pnd9eSXxhcuvO7C4+hfbOGMY5KpG5T4pdaqaCwiEnTlvKP8C7c60b15JT+bZX0RapBpRK/mSUBTdx9d5TikQTn7tzzz5Ws+2ovs1ZtpUfrxlx/Ui/OH9Ip1qGJxI0KE7+Z/R24BigEFgHNzOz37v7baAcnief6v3/Mm0s3A3BUx2Y8dulgOrVQ6QWR6hTJiL+/u+82s0uAN4GfEvwBUOKXarNxZw4/eHoBa7ZmA7D87tNo3EAzkSLREMlvVoqZpQDnAH9y93wzi25UklBy8wsZeV9wQ9YRbZvwl8u/paQvEkWR3MD1OJABNAbmmFlXICvSA5hZspl9YmZvhM9bmtkMM1sTfm9RlcAlfjw4cw0Ax/dsxds3n6iqmiJRFknif93dO7r7Ge7uwAZgQiWO8WNgZannk4CZ7t4bmBk+lwT16zdW8NjstQA8dsmQGEcjkhgiSfwvl34SJv8XItm5mXUCzgT+Uqp5LDAlfDyFYApJEtAz8zL4y/ufA/DcD4/VDVkiNeSwE6lm1hcYQHAVz3mlXmoKpEa4/z8CtwNppdraunsmgLtnmlmbwxz/auBqgC5dukR4OKmtcvML+Xj9Tn7+6jLCKsp8/tVeAD762WjaNo30R0pEvqnyzqD1Ac4CmgPfLdW+B7iqoh2b2VnAVndfZGajKhuYu08GJgMMHTrUK3i71HITn13Eu//dBkCvNk3o374pR3Vsxpgj2ynpi9SwwyZ+d38NeM3Mhrv7vCrsewRwtpmdQfAJoamZ/Q3YYmbtw9F+e2BrlSKXOuG+t1bx17kZ7MsvBGDqxOF8q1sLdGWYSOxEMse/3cxmmtkyADM72sx+XtFG7n6Hu3dy927ARcAsd78UmA6MD982HnitaqFLbZZXUMiPX/iEx2avZV9+IRNP6MF/bjmBYd1bKumLxFgkF0s/AdxGcFkn7r4kvJv311U85m+AqWZ2JcEVQhdUcT9SC+XmFzLlgwzu/deqkrbpN4zg6E7NYxeUiBwgksTfyN3nlxmlFVTmIO4+G5gdPt4OjK7M9lK7vbhgA899tAF3WLYpCw/PyFw4tBO/OKs/aam6WkekNokk8X9lZj0huBjDzMYBmVGNSmqlPbn5/OW9z8ktKCxp25yVy2uLNwHwnb5tOKlPG5o3TOHusQOU8EVqqUgS//UEV9f0NbMvgc+BS6IaldRKc/77FQ/OXEP95CSKPwA6UC/JmHz5EL7Tt21M4xORyJSb+M0sGbjW3U82s8ZAkrvvqZnQJNbcnafmZrB1dy4AqzYH//Xv/fQkXYIpUoeVm/jdvdDMhoSP99ZMSFJbbN2Tx/++sYKUZCM5KRji90hvTItG9WMcmYh8E5FM9XxiZtOBl4CS5O/u06IWldQKe3KDc/i/u3AgZx/TIcbRiEh1iSTxtwS2A98p1eaAEn+c+/tHGwBIS1WJZJF4UuFvtLv/oCYCkdona18+AMN7tIpxJCJSnSJZevGhQzRnAQvDsg4SB+7910pmLN9yQNuW3bn0bZdGakpyjKISkWiI5DN8KtCXYI4f4HxgOXClmZ3k7jdFKTapQTOWbyGvoIjBXb9eF2dAx2ac1Cc9hlGJSDREkvh7Ad9x9wIAM3sMeBs4BVgaxdikhnz6xS7WfbWXi77Vmd+cf3SswxGRKIukSFtHgmUXizUGOrh7IZAXlaikxrg7Fz/xIQDtmunafJFEEMmI/35gsZnNBgw4Afi/8Iau/0QxNqkB2XkF5OwvZOKJPfjRd3rHOhwRqQGRXNXzpJm9CQwjSPw/c/dN4cu3RTM4ia5n5mXw23+vBqBry8YlN2mJSHyL5KoeI6im2cPdf2VmXcxsmLvPj354Up227M7lsdlryS8sAuCNJZmkJCcxsldrzh/SMcbRiUhNiWSq51GgiOAGrl8RLL34MvCtKMYl1WzR+h08/u463l6xhVaN62MGKclJ3D6mDxcO7Rzr8ESkBkWS+I9198Fm9gmAu+80MxVrqUP25hVw/mPB6pldWzVi9q2jtAqWSAKLJPHnh1U6i+vxpxN8ApA6IDe/kCnzMgD44cju3HFGPyV9kQQXSeJ/CHgFaGNm9wDjgArX3JXYe3ZeBr94bXnJ80mn99UJXBGpsB5/EsHCK7cTnOA14Bx3X1kDsUklbNmdy4frtpc837Qrl/veWkWzhilcc2JP+rVPo15yJLdtiEi8q6gef5GZ/c7dhwOrynuv1Ly7X1/O2m1Bpew5/912yPf85NQjuHx4txqMSkRqu0imet42s/OBae7Fy2hLrO3Yu5+n52bQuWVDWjVuwMDOzflWtxZcNKxLyXvSGtSjjVbKEpEyIkn8txCUaSgws1yC6R5396ZRjUwAKCpy3liaSXa4KEqxn70SlEm667sDGN1Pa92KSOQiuXM3rSYCkYO5O4+9u7bk7tqyhnVrybd7q3qmiFSOllaqpb7YkcN1z33M0i+zAJj1kxNp3ODr/64kM9LTGsQqPBGpw5T4a5lrnl3E0i+z+HLXPgBSko2XrjmeHulNYhyZiMQLJf5axN15e8VmjmibxvmDOx10slZEpDpElPjNbCTQ292fDu/cbeLun1ewTSowB2gQHucf7v5LM2sJvAh0AzKAC919Z9W7ED8+3rCLIoezB3bgulG9Yh2OiMSpCu/oMbNfAj8F7gibUoC/RbDvPIKVu44BBgJjzOw4YBIw0917AzPD5wJMfHYhAB2bN4xxJCISzyK5lfNc4GxgL0BYi7/CK308kB0+TQm/HBgLTAnbpwDnVC7k+PTMvAy+yt7P2IEdGDtQJZJFJHoimerZ7+5uZsVF2hpXtEGxsLjbIoJ1ex9x94/MrK27ZwK4e6aZtTnMtlcDVwN06RJ/89yFRc41f1tEZlZwEnfZl7sBuO20PrEMS0QSQCQj/qlm9jjQ3MyuIlhu8YlIdu7uhe4+EOgEDDOzIyMNzN0nu/tQdx+anh5/16pv25PHjBVbKCh02qalMrpvGx75/mA6tWgU69BEJM5FcgPXA2Z2CrAb6APc6e4zKnMQd98Vrtk7BthiZu3D0X57YGsV4q7znv0wA4DrT+rFd4/pENtgRCShRHJy92Zgpbvf5u63Rpr0zSzdzJqHjxsCJxMUepsOjA/fNh54rSqB13XFUzvH9mgZ40hEJNFEMsffFPi3me0AXiC4LHNLBNu1B6aE8/xJwFR3f8PM5hFMH10JbAAuqGLsddq7/93GyF6taZOmImoiUrMimeq5G7jbzI4Gvge8a2Yb3f3kCrZbAgw6RPt2gtr+CWtZWIahUf3kGEciIomoMitzbAU2A9uBQ16JI5H5eENwv9qEkd1jHImIJKJI5vivDU/MzgRaA1e5+9HRDiyeLcwIEv/Azs1jG4iIJKRI5vi7Aje5++Iox5Iwlm/KonWTBqSmaKpHRGreYRO/mTV1993A/eHzAy4/cfcdUY4tbqUkJ9Gqcf1YhyEiCaq8Ef/fgbMI7rx1gpW3ijnQI4pxxYXt2Xnc+tKn7N1feED751/t5cyj2scoKhFJdIdN/O5+VvhdZyAryd15+eMvWZixg3dWb+OYTs1oWOoKnsFdWnDm0Ur8IhIbFc7xm9lMdx9dUZt8bf32HG596VMAmjSoxzMTjqVZo5QYRyUiEihvjj8VaAS0NrMWfD3V0xRQjYHDcHf+/O5aAB67ZDCj+7Wlfr3KXDUrIhJd5Y34JwI3EST5RXyd+HcDj0Q3rLpr8Re7eGHBFwD0aZempC8itU55c/wPAg+a2Y/c/eEajKlOu/fNVQBMu07r5IpI7RRJyYaHw3LK/YHUUu3PRDOwuiqvoJBmDVMYpJuzRKSWiuTk7i+BUQSJ/03gdOB9QIm/lI837OStZZvJ2J7DqD7pmFnFG4mIxEAkd+6OA44BPnH3H5hZW+Av0Q2rbtlfUMRFkz8kv7CIhinJDOnaItYhiYgcViSJf5+7F5lZgZk1JSjWppu3Svlw3Xb2FxRx4hHpTJkwLNbhiIiUK5LEvzBcUOUJgqt7soH50QyqLsnNL2TxF7sA+MVZ/WIbjIhIBCI5uXtd+PDPZvYW0DSstS/AQzPX8OjstSQZtGrcINbhiIhUqLwbuAaX95q7fxydkOqGbXvyOPfRuWzZnUuHZqk8ecW3aKHCayJSB5Q34v9dOa858J1qjqVOWbNlDxt37uP0I9sxdmBH+rVvGuuQREQiUt4NXCfVZCB1zX1vBTdq/fjk3vRtp6QvInVHJNfxX36o9kS+gcvd+XRjFs0aptCnbVqswxERqZRIrur5VqnHqQQLpX9Mgt7A9c8lmby1fDMA3z2mvW7UEpE6J5Kren5U+rmZNQOejVpEtcz+giKefP9z9uYVAPDSoi/Yk1tArzZNuHBo5xhHJyJSeZGM+MvKAXpXdyC11TPzMrjvrVUkGSWj+9tO68M1J/aMcWQiIlUTyRz/6wRX8QAkEdTsmRrNoGqT/6zcAsCC/zmZVk10nb6I1H2RjPgfKPW4AFjv7hujFE+tk7WvgFF90pX0RSRuRDLH/y5AWKenXvi4pbvvKG87M+tMcAK4HVAETHb3B82sJfAi0A3IAC50953foA/VJmd/AWc8+B5b9+SVaitkQIdOMYxKRKR6RTLVczXwv8A+ggRuBFM/FRVqKwB+4u4fm1kasMjMZgBXADPd/TdmNgmYBPy06l2oPht25JCxPYdT+relW6tGQDCvf87AjjGOTESk+kQy1XMbMMDdv6rMjt09E8gMH+8xs5VAR2AsQX1/gCnAbGpB4s8vLOLhmZ8BcNW3ezCse8sYRyQiEh2RJP61BFfyVJmZdQMGAR8BbcM/Crh7ppm1+Sb7rg7bs/N44r3P+efSTAC6t24c44hERKInksR/B/CBmX0ElEx+u/uNkRzAzJoALwM3ufvuSG94CqeYrgbo0qVLRNtU1W3/WMKsVVsB+PCO0aSn6USuiMSvSBL/48AsYCnBHH/EzCyFIOk/5+7TwuYtZtY+HO23J1jY5SDuPhmYDDB06FA/1HuqQ2GRM2vVVpo3SuHd206iWcOUaB1KRKRWiCTxF7j7LZXdsQVD+yeBle7++1IvTQfGA78Jv79W2X1Xpw/XbQfg8uHdlPRFJCFEkvjfCaddXufAqZ5yL+cERgCXAUvNbHHY9jOChD/VzK4ENgAXVDbo6rR1Ty4AZx/TPpZhiIjUmEgS//fD73eUaqvwck53f5/g0s9DGR3BcWvE9uz9AKSnpcY4EhGRmhHJDVzdayKQWHl6bgZm0DS1KmWLRETqnoSux+/ufLlrH+lpDVReWUQSRkLX4//38qAA28QTKroJWUQkfiR0Pf6XFn4BwJCuLWIciYhIzUmqwjZxU49/Z85+RvRqxaAuSvwikjgSuh7/rpx82jdvGOswRERqVELX49+Rs58WjXTTlogklsMmfjPrRVBQ7d0y7d82swbuvjbq0UVRYZGTtS+flo3qxzoUEZEaVd4c/x+BPYdo3xe+Vme9t2Ybw+75D+7QorESv4gklvISfzd3X1K20d0XEqyeVWdN+SCD7Xv3c9W3u3PGUSrVICKJpbw5/vJqGNTpM6JZ+/IB+J8z+8c4EhGRmlfeiH+BmV1VtjEsrrYoeiFF15bduSzI2Mkp/dvGOhQRkZgob8R/E/CKmV3C14l+KFAfODfKcUXN/725EoBebZrEOBIRkdg4bOJ39y3A8WZ2EnBk2PxPd59VI5FFQda+fF5bvIl2TVO5/bQ+sQ5HRCQmIinZ8A7wTg3EEnVXPD0fgCtHdldRNhFJWFUp2VAn/X7Gf/lkwy5aNErhsuFdYx2OiEjMJEziX7pxFwDTrhtBakpybIMREYmhhEn8H2/YxYherejeunGsQxERiamESPw79u4na18+7hW/V0Qk3iVE4i++Yeu8wZ1iHImISOwlROLfm1cAQJrW1RURSYzEvzNnPwAtVIlTRCQxEv+OvUHib6lKnCIiSvwiIokmIRL/2m3ZADRrqNW2REQSIvFv25MHQHKSyjSIiEQt8ZvZU2a21cyWlWpraWYzzGxN+L1FtI5f2v6CIvq0TauJQ4mI1HrRHPH/FRhTpm0SMNPdewMzw+dRtzu3gNZpmt8XEYEoJn53nwPsKNM8FpgSPp4CnBOt45e2e18+TVM1vy8iAjU/x9/W3TMBwu9tDvdGM7vazBaa2cJt27Z9o4PuyS1Q4hcRCdXak7vuPtndh7r70PT09G+0r925+TRtqLt2RUSg5hP/FjNrDxB+3xrtA+YXFpGzv1AjfhGRUE0n/unA+PDxeOC1aB9wT25Qp6eJ6vSIiADRvZzzeWAe0MfMNprZlcBvgFPMbA1wSvg8qrJziwu0acQvIgIRrLlbVe5+8WFeGh2tYx7KnrygJLMqc4qIBGrtyd3qsntfOOJvoMQvIgIJkPiXb8oCoGUT3cAlIgIJkPjzC4P1Fju3aBTjSEREaoe4T/zZefnUSzIa1U+OdSgiIrVC3Cf+FZt207B+MmaqzCkiAgmQ+OslJ5GzvzDWYYiI1Bpxn/hz8ws5smOzWIchIlJrxH3iz8svIrVe3HdTRCRicZ8RcwsKaagTuyIiJeI/8ecXklpPiV9EpFgCJP4iUlPivpsiIhGL+4y4L7+Q1BSN+EVEisV94s9V4hcROUBcJ353V+IXESkjrhP/vvxC8gudZg1Vi19EpFhcJ/7iO3YbN9CIX0SkWFwn/n1h4tdUj4jI1+I68efmB4lflTlFRL4W14l/X5j4G2rELyJSIq4Tf/EcvxK/iMjX4jrxF4/4Gyjxi4iUiOvEn7tfc/wiImXFdeLPUeIXETlIXCd+ndwVETlYfCf+4pO7GvGLiJSI78Sfrxu4RETKikniN7MxZrbazD4zs0nROs6+/EJSko2U5Lj++yYiUik1nhHNLBl4BDgd6A9cbGb9o3GsffsLNb8vIlJGLIbCw4DP3H2du+8HXgDGRuNAfdulMebIdtHYtYhInVUvBsfsCHxR6vlG4NiybzKzq4GrAbp06VKlA100rAsXDavatiIi8SoWI347RJsf1OA+2d2HuvvQ9PT0GghLRCQxxCLxbwQ6l3reCdgUgzhERBJSLBL/AqC3mXU3s/rARcD0GMQhIpKQanyO390LzOwG4N9AMvCUuy+v6ThERBJVLE7u4u5vAm/G4tgiIolOdzaJiCQYJX4RkQSjxC8ikmDM/aBL6GsdM9sGrK/i5q2Br6oxnLpAfU4M6nNi+CZ97uruB90IVScS/zdhZgvdfWis46hJ6nNiUJ8TQzT6rKkeEZEEo8QvIpJgEiHxT451ADGgPicG9TkxVHuf436OX0REDpQII34RESlFiV9EJMHEdeKvqbV9o83MnjKzrWa2rFRbSzObYWZrwu8tSr12R9jn1WZ2Wqn2IWa2NHztITM71NoItYKZdTazd8xspZktN7Mfh+1x228zSzWz+Wb2adjnu8P2uO0zBMuxmtknZvZG+Dyu+wtgZhlhvIvNbGHYVnP9dve4/CKo/LkW6AHUBz4F+sc6rir25QRgMLCsVNv9wKTw8STgvvBx/7CvDYDu4b9BcvjafGA4wWI4/wJOj3Xfyulze2Bw+DgN+G/Yt7jtdxhfk/BxCvARcFw89zmM9Rbg78AbifCzHcabAbQu01Zj/Y7nEX+Nre0bbe4+B9hRpnksMCV8PAU4p1T7C+6e5+6fA58Bw8ysPdDU3ed58BPzTKltah13z3T3j8PHe4CVBMt2xm2/PZAdPk0Jv5w47rOZdQLOBP5Sqjlu+1uBGut3PCf+Q63t2zFGsURDW3fPhCBJAm3C9sP1u2P4uGx7rWdm3YBBBCPguO53OO2xGNgKzHD3eO/zH4HbgaJSbfHc32IOvG1mi8L1xaEG+x2Tevw1JKK1fePQ4fpdJ/89zKwJ8DJwk7vvLmcKMy767e6FwEAzaw68YmZHlvP2Ot1nMzsL2Orui8xsVCSbHKKtzvS3jBHuvsnM2gAzzGxVOe+t9n7H84g/3tf23RJ+1CP8vjVsP1y/N4aPy7bXWmaWQpD0n3P3aWFz3PcbwN13AbOBMcRvn0cAZ5tZBsFU7HfM7G/Eb39LuPum8PtW4BWCqeka63c8J/54X9t3OjA+fDweeK1U+0Vm1sDMugO9gfnhR8c9ZnZceOb/8lLb1DphjE8CK93996Veitt+m1l6ONLHzBoCJwOriNM+u/sd7t7J3bsR/H7OcvdLidP+FjOzxmaWVvwYOBVYRk32O9Znt6P5BZxBcDXIWuB/Yh3PN+jH80AmkE/wV/5KoBUwE1gTfm9Z6v3/E/Z5NaXO8gNDwx+wtcCfCO/cro1fwEiCj61LgMXh1xnx3G/gaOCTsM/LgDvD9rjtc6l4R/H1VT1x3V+CKw0/Db+WF+emmuy3SjaIiCSYeJ7qERGRQ1DiFxFJMEr8IiIJRolfRCTBKPGLiCQYJX6JO2bWKqx6uNjMNpvZl+HjbDN7tIZiGGhmZ9TEsUQqK55LNkiCcvftwEAAM7sLyHb3B2o4jIEE11i/WcPHFamQRvySMMxsVKma73eZ2RQzezusjX6emd0f1jZ/KywXUVzv/N2wmNa/i2+pL7PfC8xsmQV19OeEd4r/Cvhe+Enje+Hdmk+Z2QILas+PDbe9wsxeC4+52sx+GbY3NrN/hvtcZmbfq7l/KYl3GvFLIusJnERQ73wecL67325mrwBnmtk/gYeBse6+LUy+9wATyuznTuA0d//SzJq7+34zuxMY6u43AJjZ/xGUJJgQlmWYb2b/CbcfBhwJ5AALwuN2BTa5+5nh9s2i9q8gCUeJXxLZv9w938yWEizc81bYvhToBvQhSMgzwqqgyQSlM8qaC/zVzKYC0w7xOgT1WM42s1vD56lAl/DxjHB6CjObRlCu4k3gATO7j6CUwXtV7qVIGUr8ksjyANy9yMzy/ev6JUUEvxsGLHf34eXtxN2vMbNjCRYUWWxmAw/xNiP4RLH6gMZgu7J1U9zd/2tmQwjqE91rZm+7+68q2T+RQ9Icv8jhrQbSzWw4BGWizWxA2TeZWU93/8jd7wS+Iiihu4dgychi/wZ+FFZRxMwGlXrtFAvWW21IsILSXDPrAOS4+9+ABwiW3hSpFhrxixxGOFc/DngonGOvR7Bi1PIyb/2tmfUmGNXPJKi6uAGYZMFqWvcC/xtuuyRM/hnAWeH27wPPAr2Av7v7QgsW1P6tmRURVGW9NkrdlASk6pwiMWRmV1DqJLBITdBUj4hIgtGIX0QkwWjELyKSYJT4RUQSjBK/iEiCUeIXEUkwSvwiIgnm/wGE4pZIYxyYdAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(1)\n",
    "thompson_sampling_solver = ThompsonSampling(bandit_10_arm)\n",
    "thompson_sampling_solver.run(5000)\n",
    "print('汤普森采样算法的累积懊悔为：', thompson_sampling_solver.regret)\n",
    "plot_results([thompson_sampling_solver], [\"ThompsonSampling\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "455744E26FDE4FA6B74962371C42AFDD",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "通过实验我们得到以下结论：$\\epsilon$-贪心策略的累积懊悔是随时间线性增长的，而另外三种算法（$\\epsilon$衰减贪心算法、上置信界算法、汤普森采样算法）的累积懊悔都是次线性增长的（具体为对数形式增长）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "59CB789087D4454D998B935764C3C798",
    "jupyter": {},
    "mdEditEnable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 总结\n",
    "\n",
    "探索与利用是与环境做交互性学习的重要问题，是强化学习 *试错法 (trial-and-error)* 中的必备技术，而 *多臂老虎机* 是研究探索与利用技术理论的最佳环境。了解多臂老虎机的探索与利用问题，对接下来我们学习强化学习的策环境探索有很重要的帮助。多臂老虎机各种算法存在理论上的累积懊悔分析，有兴趣的同学可以自行查阅相关资料。$\\epsilon$-贪心、UCB和汤普森采样方法在多臂老虎机任务中十分常用，其中UCB和汤普森采样方法均能保证对数的渐进最优累积懊悔。\n",
    "\n",
    "多臂老虎机和强化学习的一大区别在于与环境的交互并不会改变环境，即多臂老虎机的每次交互的结果和以往的行动无关，所以可看作是无状态的强化学习 (Stateless Reinforcement Learning)。接下来的章节，我们则开始在有状态的环境下，即马尔可夫决策过程，讨论强化学习算法。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}