Convolutional Neural Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we'll be walking through the Tensorflow code behind creating a convolutional neural network. If you'd like more of a conceptual view of how these networks work, check out my [blog post](https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/). A CNN tutorial from the Tensorflow docs can also be found [here](https://www.tensorflow.org/tutorials/deep_cnn)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import random\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import datetime\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inputs and Outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, in this next step, we're just going to create a session. Your x and y_ are just going to place placeholders that basically just indicate the type of input you want in your CNN and the type of output. For each of these placeholders, you have to specify the type and the shape. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph() \n",
    "sess = tf.InteractiveSession()\n",
    "x = tf.placeholder(\"float\", shape = [None, 28,28,1]) #shape in CNNs is always None x height x width x color channels\n",
    "y_ = tf.placeholder(\"float\", shape = [None, 10]) #shape is always None x number of classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Network Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our placeholders, we just have to specify the network architecture. Basically, the main point we have to remember is that all of the filters (weights) and biases are tensorflow variables. Let's create our filter and bias for the first layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "W_conv1 = tf.Variable(tf.truncated_normal([5, 5, 1, 32], stddev=0.1))#shape is filter x filter x input channels x output channels\n",
    "b_conv1 = tf.Variable(tf.constant(.1, shape = [32])) #shape of the bias just has to match output channels of the filter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our filter and our bias, we can call our first conv layer. The 4 arguments you have to specify are the input (which is where our placeholder comes into play), the filter (we just created the variable for that), the stride, and the padding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"Placeholder:0\", shape=(?, 28, 28, 1), dtype=float32)\n",
      "Tensor(\"Variable/read:0\", shape=(5, 5, 1, 32), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "print x\n",
    "print W_conv1\n",
    "h_conv1 = tf.nn.conv2d(input=x, filter=W_conv1, strides=[1, 1, 1, 1], padding='SAME') + b_conv1\n",
    "h_conv1 = tf.nn.relu(h_conv1)\n",
    "h_pool1 = tf.nn.max_pool(h_conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This just defines some methods to make the function calls a little nicer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def conv2d(x, W):\n",
    "  return tf.nn.conv2d(input=x, filter=W, strides=[1, 1, 1, 1], padding='SAME')\n",
    "\n",
    "def max_pool_2x2(x):\n",
    "  return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's just complete the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Second Conv and Pool Layers\n",
    "W_conv2 = tf.Variable(tf.truncated_normal([5, 5, 32, 64], stddev=0.1))\n",
    "b_conv2 = tf.Variable(tf.constant(.1, shape = [64]))\n",
    "h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)\n",
    "h_pool2 = max_pool_2x2(h_conv2)\n",
    "\n",
    "#First Fully Connected Layer\n",
    "W_fc1 = tf.Variable(tf.truncated_normal([7 * 7 * 64, 1024], stddev=0.1))\n",
    "b_fc1 = tf.Variable(tf.constant(.1, shape = [1024]))\n",
    "h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])\n",
    "h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)\n",
    "\n",
    "#Dropout Layer\n",
    "keep_prob = tf.placeholder(\"float\")\n",
    "h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)\n",
    "\n",
    "#Second Fully Connected Layer\n",
    "W_fc2 = tf.Variable(tf.truncated_normal([1024, 10], stddev=0.1))\n",
    "b_fc2 = tf.Variable(tf.constant(.1, shape = [10]))\n",
    "\n",
    "#Final Layer\n",
    "y = tf.matmul(h_fc1_drop, W_fc2) + b_fc2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's formulate our loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "crossEntropyLoss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels = y_, logits = y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we need to use an optimizer to minimize the function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "trainStep = tf.train.AdamOptimizer().minimize(crossEntropyLoss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following are the statements that help with calculating accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next line is the main statement that gets initializes all the variables we've declared earlier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is code to allow us to visualize our training with Tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tf.summary.scalar('Cross Enropy Loss', crossEntropyLoss)\n",
    "tf.summary.scalar('Accuracy', accuracy)\n",
    "merged = tf.summary.merge_all()\n",
    "logdir = \"tensorboard/\" + datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\") + \"/\"\n",
    "writer = tf.summary.FileWriter(logdir, sess.graph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 784)\n",
      "Tensor(\"Reshape_4:0\", shape=(1, 28, 28, 1), dtype=float32)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAFfCAYAAACfj30KAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJztnX+sdGtV37/PnDlzfr2XlxSaiykGUTS9/qHp+9paqhRa\nmmBpgvQfzamGojEGfzTmTarEhAgFE6MGL614G/+woEFOQqJGaoBrRfyBFqj3FRWhIVAQFe4VvHDe\n+54fc+bMefrHnDWsWbPWs/fMmTl7fnw/yZP97D17Zvaemf2dtdez1npSzhmEEEKaodX0ARBCyDpD\nESaEkAahCBNCSINQhAkhpEEowoQQ0iAUYUIIaRCKMCGENAhFmBBCGoQiTAghDdJu+gBSSs8A8BIA\nnwZw2uzREELITNgG8FUAHs05/31px7mJcErphwD8ZwDPAvBnAP5Tzvn/OLu+BMCvzus4CCGkQb4L\nwNtLO8xFhFNK3wngjQC+H8CHANwB8GhK6etyzl8wu38aAN72trfhoYceGnngzp07ePjhh+dxiI2z\nyucGrPb58dyWl+s6v4997GP47u/+buBS30rMyxK+A+AXc86/AgAppVcB+HcAvhfAz5h9TwHgoYce\nwq1bt0YeuHnz5ti2VWGVzw1Y7fPjuS0vDZxfpYt15gNzKaVNALcBvFe25UGptt8B8PxZvx8hhCwz\n84iOeCaADQBPmO1PYOAfJoQQcglD1AghpEHm4RP+AoA+gAfN9gcBPB496c6dO7h58+bItuc85zkz\nP7hFYX9/v+lDmCurfH48t+VlHud3cHCAg4ODkW2Hh4e1n5/mMbNGSukDAD6Yc/6Ry/UE4DMA/lvO\n+WfNvrcAPPbYY4+t9IAAIWR9uHv3Lm7fvg0At3POd0v7zis64ucAvDWl9Bi+HKK2C+Ctc3o/QghZ\nSuYiwjnnd6SUngng9Ri4IT4M4CU558/P4/0IIWRZmVvGXM75EQCPzOv1CSFkFWB0BCGENAhFmBBC\nGoQiTAghDUIRJoSQBqEIE0JIg1CECSGkQSjChBDSIBRhQghpEIowIYQ0CEWYEEIahCJMCCENQhEm\nhJAGoQgTQkiDUIQJIaRBKMKEENIgFGFCCGkQijAhhDQIRZgQQhqEIkwIIQ1CESaEkAahCBNCSINQ\nhAkhpEEowoQQ0iAUYUIIaRCKMCGENAhFmBBCGoQiTAghDUIRJoSQBqEIE0JIg1CECSGkQSjChBDS\nIBRhQghpEIowIYQ0CEWYEEIahCJMCCENQhEmhJAGoQgTQkiDUIQJIaRBKMKEENIgFGFCCGkQijAh\nhDQIRZgQQhqEIkwIIQ1CESaEkAaZuQinlF6bUrow7aOzfh9CCFkF2nN63Y8AeDGAdLl+Pqf3IYSQ\npWZeInyec/78nF6bEEJWhnn5hL82pfS3KaVPppTellL6yjm9DyGELDXzEOEPAHglgJcAeBWA5wL4\ng5TS3hzeixBClpqZuyNyzo+q1Y+klD4E4K8AfAeAt8z6/QghZJmZl094SM75MKX0cQDPK+13584d\n3Lx5c2Tb/v4+9vf353l4hBByJQ4ODnBwcDCy7fDwsPbzU8551sc0+gYp3QDwGQA/kXN+s/P4LQCP\nPfbYY7h169Zcj4UQQq6Du3fv4vbt2wBwO+d8t7TvPOKEfzal9C9TSs9JKf0LAL8BoAfgoOKphBCy\ndszDHfFsAG8H8AwAnwfwfgD/POf893N4L0IIWWrmMTBHJy4hhNSEtSMIIaRBKMKEENIgFGFCCGkQ\nijAhhDQIRZgQQhpk7hlzZD3xkoDstlklCqWUio9f5X2qXvuqjxNCESZzI+dcq10FT+T0Nnl9u/S2\npZSGz/WWVdui5xJSgiJM5kLOGRcXF7i4uBjpe9smRYtbqa9FVgu+ty7PqdNarVatfQipA0WYzAUR\nORHafr+Pfr8/7OvlJNZwyVK1y7qW+MXFxZjATrK0fTl/fTyERFCEyVzQAnx+fj4UYa9vRbiOn7WO\na0CLrO3bJYChiE7bhFarhZwzBZjUgiJM5kbOeURwbev1ejg/n2z6QSu0utntVmwj94gW4Y2NDXdZ\nemxjYyM8TkKqoAiTuWBdEVp4z87Ohv1erzexO6Ju83zR2g2im7gTNjY2ik3vk3MeE2BrhRNSBUWY\nzAV9+y/WsAjw2dnZSH8Sd8QkA2ZiiVvx9bYBwMbGBtrt9sjS62v3hhZbfRwUYFIXijCZG9YSFstX\nxLfb7boiXCISXL0ufe0O0X8G3npKaUSAdev3+yMC3G63xyxdfSwXFxeMjiC1oQiTuSAiZX3CWoBP\nT0/R7XZribDn840iFrQ1agcDowYAm5ubQ+HVfVnXPmRPgFut1oiPmdYwqQNFmMwFzyesLeFut4tu\nt4uTkxNXrCKXRJ0wMS2INhLDi844Pz9HSmkovJubmzg/P8fm5iY2NzfHfMieCGsBpj+YTMJainDV\nBTLJBVRn33W8ICP3w+npKU5OTkaWXsJGlQiX4nS1JSwWuLbGrQBrEY6a7NfpdMbEOLJ8J42QsPt7\nz5826oLRGovLWoowMJ5S622LUl6jFNg6r7cuVtLp6elYsyIsbRaWsCfCniUc9QEMBVes4VK/bmu3\n22PnVFpWJYLox6PPiYK7XKylCHuB+14Qv/XvlQS19Hre+qoL8dnZ2VB4tf/X60efRSQwdfzBdmDO\na1qMZWDO+oM933DVUvc3NjYq45l1qwqR06Fyco76c5Jt+vOjKC82aynCAEbEMQpb8sTZy8KyYhul\n5+rlOoiwRD+I4Oq+Xi99Ft4teikioiTCNiJCbwMwFhWhmxc14Ym03S4i7NWb8CI8Sn8C0s85Y3Nz\ncyjAXnae3s7svcVmLUXYi2GNWsmS9fpVt73SXwcRrtsmiY4AUFvQJvlTFCvUxgmXlpFA63VttVa5\nUVqtFjqdDjY3N9HpdIZN1uV3K8cqn0tJgMnis5YiDGBMhHU6rR3E8bKuokwsScWtamJ9rSoyKCcD\nc6V+3T+kyHcategOxVsH4CZnRK2UzGG3eXUmrPhKJt7W1tZY297eHnExbGxsDLP8AAwF17olyHKw\nliJsXQi2loFe1rGibFZY1HTEwKqLsD7XqC/r1522bJutHeGlKEf9uv7bqA6FXt/Y2MDOzg62t7ex\nvb09UuBIhFZcFiLCniXsfWZkcVlbEdbWUCSeZ2dnRTeFN9DjpeV6/XUQ4bptWhG263Y7gBGhLfUB\nVIpknUI+0bKOsLfbbXS7Xezu7o7chcn5iIUt20WUvc+ILA9rKcLAqDvCFpfRLfLrett6vd7IgJNe\n2m2TVg9bNqLPyEuamFSEZemJse57A6l6QNUmXpRcBXWb536I3Biei0P/5jwBlphlSY22YuxFR5DF\nZi1F2BuY8zK6xHKNgv2toNh0XC8kS9ZXXYT1Z1tVt2ESrKVbV4Rt85It6gygaYGu0xcRjgb4dF8E\ntiTAW1tbY49LYgowWkyeg3PLwVqKMDAaouZZwiKY1k9cahIbW2qSqLAOIlwawNTrk1ISYr0EylMb\necLltWgA0IvIsH1xM3ghbnabTpGW49Fha1tbWyPjFMCXC8iLZcywtOVjZUR4kltauQCtK8LWNTg9\nPR2pfWvF2G6TWggitrrZbb1eb46fRvPUcQNYd0AdvAiAqqgAK8B2m/d6VW6POtu1iEZNxwFLGBqA\nEQtaBFjfmdk/LxFgPYhn+1WfK90YzbBSIlx1yynbzs7OxlJno/WrWMLapaHdGuuQMRd9/tYKndV7\n6XAtAGPrdd7Pvo73mnWPWz9fxyJ70Qx6cFAXNpIYYRFp7e4A4PqUPV+z/jws9hzJ9bNSIlyV/SZ9\nbbF6BWX0YxLJUPIHW5+wzQrTAmxjU1eVuuI76cVfRyjt65as32ner/R8G7OrBViW9li0CMsfuRZg\nEVP9ujnnyjTrzc3N4WcS3UHQ+m2elRNhr0qWXVq3gXYV2G3igytFSci6HdTTYWn6NnJdLOHSHclV\nrOE64lvHEo7e3/pV7X6REHuCLU0LsX2O/oy63S46nQ5OTk5CC1j2l0w6vez3+8OUZsEW/LH+YvqQ\nm2XlRFhnrdkEAVl64lvy3dZNb7alG3VWmJeBt8rM0/0gr28H4Ox6VX+S96nrW/WORcLIdOKFvXOT\nLLhutztizXoWsIxn6Ky6Xq+Hra2tkd+W+KR1YkckthTg5lhJEY6SJqR54huJshVhL1tOv6+XFbaO\n7gigurznLF7fE946Ylz1mpEboq4gW7eE3i5NBFLXueh2u2M1J/Rz9W9NMut6vR62t7dH7jhEgHWK\ns3c80R0EuT5WUoS9UDPdPAGWdnx8PLIuLoSqGhJeDYqoZsQ6uSOq+npZB0/U7PY6rodpB+rqujDs\n8/V+WoRFLPv9/jBZo64I28QO/Vl42XU2akND8W2OlRJhG3ImYWbeAFwkvFaExRLWt4+6r9dL2XR6\nfZ0sYbu8igDr/SMxto9N4gvWj5ciI+pawYJ819oN4SWAXFxcjGTceS4IG1Kp766sBaznxrPCK8fg\nfWbkelkpEbaWsJ7JwYqrrFctxRL2BplsvxSR4bkw1gHPHTArVwRQjguuuz3aN3J1VFnDdn9tIdt0\nY90iAY6SivRAr7yXDlfrdDpDkdZpzdK350QhboaVFGFrCR8fHw/b0dHRUGQ94bV9KTAT+TXtQEtk\nMdv+qrsjgLILYFbn771OlSU86WtP8nr2OdY3LEJst+kICivANrVeXG267oaIq078sNl1AEb80Pr3\nTAFujpUTYf1D1Zbw0dERjo6OcP/+fVdso22SXlz31rqOYOvnrBPXdc6zfp9pXi9yj2jhtdu0e8Aa\nFeID3traGv6ubRSEzbCT55Wy66zLg2J8/aycCOtBC+uOEBE+OjoaE99IiHWNh2n8i2S1qTvAVwfr\nR7Y+4E6nM/xddzqd4XO0AHc6nZEUZz0wF93BUXybZWVEGMCIH9a7fdMVzXQIm41c0AMd+gfqWS+W\nZfsx23MrWW32Ftrbtmznr4kGDavugrx129fPL6174wvn5+djkRIARmqc2BRnHV1hfcWldGdgshmc\nl/n7XhRWRoT1D7iqMI9NKfbieCMBrhIkj0X+odrBobqtqoLYslGyEu0grDcGUFpGYq7f2zsO+S23\nWi2cn5+P/cmJAOsZnr0EDwBherNET7TbAynwfteWRf49LyMrI8LA6EhyHUu4SoAFK0DeNs8KrFpf\nBGyolA2b0ttsi7YvswiXBlWtlerFj3vx5CVrWb+/7msB9gbsgHERjmpM5JxHJg/Vac5ehp2N6NAw\nkmL2rIwIR5awHaib1BLWeJaftQhlP/s8r78IRCLqCWzV9D1627JRR2Ctu6tO075YL0mnyiXhCbA0\n+U2LAEfxxRcXF9ja2hr6i6UwvBddoSMnKMDXw0qJcOQTlnA1nbxhpzCyiRSRAJdmUrDWcJ1+01RN\nRKm3VfkT9bZlI+c8VguktG6r6Nl1+S3oEMeU0lhsuT0Gzx1hH5ffqE1xjgS43++PTB7qzd4h36/e\nJq/j/XYpxrNj+a6WAlXuCG0NexXOoow2zx8aWY+TDHQtAt7MwKU50LwZIrzty0bOOcxw9PpeUf9e\nr+emGlsr2BM6jU3k0K+lLXJvEE7vp4/XllS1FrBOcZbtpYHWRfoNLzsTXy0ppRcA+FEAtwF8BYCX\n55zfafZ5PYDvA/B0AH8E4Adyzp+4+uHG1B2YE59wqbiO9eFdnlPl7bnOSqqzXAS8uc480dUxqHWW\ni3SOddAZaZ5lq7fZGbk3NjYqBVjW9fvp34P1F+vH9TYpyCO1JjwBtteB/Z1rAZbvtdfrDX3EwGjM\nsmbZvtdlYBqTZQ/AhwH8EoBftw+mlF4N4IcBvALApwH8JIBHU0oP5ZzPpj/Uajx3hK6opv3B3owZ\nkTvCs4Kj6cxlf/08u00vm6ZkzdrtMgAkgzpek8eWDZ3oY38Xtq/vrvR371mtctsurgh5TA9+abQB\noEtf2nRn/Ru0FrBNcda1rD0L2KY4y7HKPoI+3kX5/a4CE4twzvk9AN4DAMn/Jn4EwBtyzr91uc8r\nADwB4OUA3jH9oVYeV+iO8CzhqMJZKUTNE2JtPdri2fp5Xn8R0GFLNozJmwNNNz3ibrctyvnVRVu4\n9vbdliWVSmd1qp15fl9du8G75deWsPUne2MT+jlelp1OcZbXkt+sfF86sUMLsBZd+oHnw0yddyml\n5wJ4FoD3yrac872U0gcBPB/XJMJ14oQjn19dS9hawTIDQkl0F1WEdZxpaV1G10Vw9Yi7fWxRzq8u\nuvBTaSmDvHUEWJpGC6H9PdgwNm1Bl35XpbEQSXPWvl59xyPfm/39iwDrvj4uCvLsmPUIyrMAZAws\nX80Tl4/NFW9Awg7M6Xnj9L66r28J9Q+tJMBWhD3x9S6ipvFcC3qb7uuZHHTb3t4e27ZsXFxcjA3g\n2skApFUJsHYHSBaa3se7y7L76KWHfo4nwDbFWd7PqzFhZ3HW1rutM+FdF+RqLN8wdoD2cWkB2d7e\nxs7ODnZ3d7G3tzd0R+jbRSvEept+zaqJFXXaZ0l8F+UHnFJy/brWArYirONNrSUs/UU5x7rId62b\ndcvoPyXvMc89o0MgS4X+U0ojdUqA6sw6bwzE/r50nLA3wYHXxPqN4sI9XzaZnlmL8OMAEoAHMWoN\nPwjgT0tPvHPnDm7evDmybX9/H/v7+7XeWN9mafHd29sbGYjr9wezIlfFgeqMpbpRAdYSjqzhRRGo\nlJJ7LtH5WYEptUU5x7ropAh7t6OjB2QZia++O9je3h6LxLG+ZekL1gq2UTr6Mf24/hPxxDiaaeb0\n9HTsLqbf74+cu0RlCIv0G14EDg4OcHBwMLLt8PCw9vNnKsI550+llB4H8GIAfw4AKaWnAfhmAL9Q\neu7DDz+MW7duTf3e2nqRi2FnZ2d4W6nDc8Qd4Ymvba1WK7QSbd+mjHriu0g/YBHhqoiIyBqM+lLh\na5mQ786zhEV85VbffvedTgfdbndobYr46ph0u5TICju4Jkuv6X30fjabT//GZB97DFqAdV+iJOTc\nJR7ZC9ekW2KAZyzevXsXt2/frvX8aeKE9wA8DwOLFwC+OqX0jQCezDn/NYA3AXhNSukTGISovQHA\n3wD4zUnfa8LjGhntFUvYhueklEZEuKpZES7dsnuW8KKLcBQP7C2r/oj0+rIhUQva139+fo52u43z\n83Nsbm4O3QfW6hch84ROxiF0nYdutzuW3m3rTNh49Si7ro4Aiwh7x6ersEm7uLgYxgxH4kt3xOyY\nxhL+JgDvw2AALgN44+X2XwbwvTnnn0kp7QL4RQySNf4QwL+dd4ywdUeIJWwD1Fut1sjcXDZKoq4I\ne01bwsskwnVbZDV7IW3LhvhBtRDb9GT5TYgrQdfujQbzdE1rfcekfaviTtCCKlERNkYYiK1hEWG9\njzzmZY16AiyDePpPQNCfD5kd08QJ/z6AYoWWnPPrALxuukOaDi3Cm5ubwzx5bQHLLaaIcJ001Var\nFd5+223LKMJVhXn0MrKQvW3LhgihiK+k8XouKx1HrNPfbV/C2Y6Pj0MBtiGVNixNhFfHDetj1jHF\nMrhnLWgZB4lmH7dCLO+nY4T1H7ZXW4VMz/KZLAEiFiKKIqLAuJVsU5U9a8cT4cj/WccS9sLXFgGb\ndl1qpfoSdtuyYa1Rr6+XelCtNOB2dnbmCjAwOntGr9cbCr8WYjk2+3uylm5kAcsfiwiwjZIQAdaF\n4fXr6N+vfLd0R8yWlRFhLbRbW1sjOfA6wmFra2vMEi4tRYRthpgV4k6nM5ElvChCLKLgLaNtdUR7\nUc6vLta/atf1NnFJ2LRg2z8/Px+WmtRx5ABGrGttRaf05Qk/9bHppd6u3Rk2BllHS2jftQwkio/a\n3t3ZgWUdW2x91+TqrIwIa0vYuiC0hSyDdfoWsNTXIlyVqqtFuM4g3SJQ13K326uWy0hpYExvs3Um\nbNPbJQpCfybWAtZCrTPsbLSE97vxLGHtnpDvRCduaDEWK9iKsH6uDE5q9wwFeHasjAhrS1gPwlkL\n2NYPrqqctbGxUTs21rOES2K8CGirR5ZV2+w5LOq5TYIXHqbXdd/6cUv9brc74kIQq1XXdeh0OqEv\n3XNFWNeDPrboO7MDc9qA8KJ8bH0U+YPw0vrJ1VgpEZZIBv0PLlaqJ7hR9pJeb7VaIzURbN0E3drt\n9kRW5aJSOrbosUU+n2koiYwXW+5F2IjQymej3Q+6nonOwtPvL2Fz1tq1x+kdq923JMBWjG1EjMRG\n60gjivDsWDkR1imX9jGv5kOVNaMtYeuKsOuRT3jZRJiUKblw5DcmA2IppbEsOkki0n2d1aldGhJS\nCXw5ltnGDHtYkYx80NoVYuuf2DoTusgPRXh2rIwIC/bCKI3UexdQq9UaSdvUSQraYtHhWFX+X7K6\n6N+Pl9mmRcymNe/s7IyEtV1cXIyUyjw7OxtJ6ChZxFVoN4oWYq8inHVF2IgjCvFsWSkRtqIajSpH\nwqvjRCVWVLs1dFaY/uHqiACK8OoT+cK1YMrjWsi0EG9vb49UL5P4XhHFbrc7ltIsd3lXFWFtbXvp\n054A67BOCvBsWRkR1heGCLC9KORxHYtpxVhE2Bbw8bLCdIxsSXwpyKtHSXw1JUvYm3aolFEnv8dZ\niLAVYP263jFThOfHyogwUH1h6Mf17AFSJcoL0LcjxDZTLLKE9fuR1aVkDYuI2ip0ImqesHkCrMVz\nliLsFRCSa0ELsJ0klCI8W1ZKhIFYiO12nY8vTVIydfOyw7xsMf1+WoT10vbJciNCqb97m2kmIqyF\n2Jt6XkdDWAHWg8SzFGFPgMUNF7lNGKY2e1ZKhD2XhKxby1eWOjPKBuTbhA9vaa1gfRz2mMjq4H2v\ndhzCs4RlKiFPgPXrWQH2LNdJ8CzqSIS1AEtRIpvqTwGeHSsjwpG1KReGti70SHYpM8qmf9rR41Kq\nLq3g9cB+v/a3pkXYRhlYi1ILoh1A0y6wq4iwFmI5Tmshe/W49QAireDZsjIirNEiW5UJVbVduzCs\nS8PrC/ZCoQCvFp74WgEWbNKQFWA7VqGTPXQR+atawiLunvWr30usYKmDrAsS0Sc8e1ZKhCPhs6Fq\nkyy9gbYoRKnqeMhqUfV7EzwRtn/y2lWms+2iZIppkNeWY7SCL9XWxAq2Ihy5UMjVWCkRBmIfbNUP\npu4PihYu0dT5vemMM2+2Cu3i0uFjUnpS16u+qk9Y9yUuXvuIW60Wtre3sbu7OyLCeoowseLJbFgZ\nEa76YVIsyayo81uy4xIppZHBXDtbiYi0necvCn+cFG/cwz4mA9ZeHQw7dkJmR3GGDEIIIfOFIkwI\nIQ1CESaEkAahCBNCSINQhAkhpEEowoQQ0iAUYUIIaRCKMCGENAhFmBBCGoQiTAghDbIyacuEXBfe\n3IVRESiZvFPPoGybnenbVlqbdcqwFA7S63qbV2FQl8LUTQoC1ampwtIBPhRhQqbA1p22Qinr/X5/\nOKW9FMSxzRbK0WLtlb68ihB7AhyJrxVb/WchfyreZAac4msyKMKETIFnFdr5CaXvCW/UdO1eT4hn\nVTzHq39sp1Wy5S5tnWM9513UBApxDEWYkAnRQmUFyhMsKUuphdb2tRXsWcN22q1ZnostRB+5HyJL\nWM84Y5cABbgKijAhU+AVRY+aJ7albXYm5llNrqkFt2oZuSLs+eo6xHrqMBFgPb0Y8aEIEzIFWqjE\nKtQDcLqJyFqxjZp+rueOmOXgnF7Kdmvpl3zCWoQ3NjaGAmzn2SMxFGFCJsSKlBZe61KoEttInK9j\nYM4KsBVOawl7QqxnH5fXlNfSSxJDESZkCqwIi+/XczFosfUE2oqxNzA3DytY+gDGrGHtg7ZCbC1h\nEWBBD8xpUacg+1CECZkCO1ilLWEb8eBZyF7fE+FoiqGr4oWq6XOzAmz/cOT4rAADo/Pm0RVRDUWY\nkAnxoiNsFMTp6ak7ZXzUPOvYWsKznmDTCqS2jr3ICM8S1iJsY4NlzjoKcRmKMCFTUPIJiwifnJzg\n9PR0bKDNE2HtzvBihK9qCddxB0QhalaArV9YI+LLQbn6UITJXJjk4rP7eunAV+3Pkn6/PxRZacfH\nx+766empGz0RLb3MuVmGqNXdz7N6rZV/fHyMnDM2Nzexubk5cnx0SdSHIkzmTh2RtdZTVYvShPX6\nvDg/Px+xdEV4vX632x3zo0at1+u5LoxZCrGHfT0RYBFeEVwR242NjaFV3e120el0sLW1NWxyrMBA\njDc2NtBuU2oi+MmQuaJvcaOl9UF6PsnIT1naNm9LuE7T4WZesR7d1+6MyBqe5cCcoCMjAIzEPp+d\nneH09BSdTmdMgHPO6Ha72N7exvb29shxAhhGTszzD3EVoAiTueElAOh16dsR+Kqll8HltXld/BcX\nFyPFeKK+tOh4vXUbJaHFelb1I6pC1LQIyzm12+0xAe73+zg7O8Pu7i56vZ4rwO12e65/iKsARZjM\nlcidoB+bRKSiFGHvVn+elrBX/8Hrn52djf0xRH8YXtKHZwlfhShBQyPHqC1hScqQ17DH2+/3R15T\nBPj8/JyWcAUUYTIXJvHtVtVesGJbt83TEi5lwllr1rPovb6tUKb7s3CzeBESWpRlqa3ydrs9koZs\nrWRxt8hxiQBvbm6i0+nM7M9jlZlYhFNKLwDwowBuA/gKAC/POb9TPf4WAP/RPO09OeeXXuVAyfJS\nGlTzkgCqQrrqpgLPS4TFSoySLnRfBKpui/zF0mbhE7bWsMUKrQi3tpDlMX3HIS4IEeCtra0RgSY+\n01jCewA+DOCXAPx6sM+7AbwSgPztdqd4H7ICeANwdqlH46P6C1pco7KQdnu/35/LOV1cXIz9QUR9\na8GWlvqz8NwWswpTq+OO0GnJ8jxtqXe73aHIyuuIC0IEWD4LuiPKTCzCOef3AHgPAKQ4+rubc/78\nVQ6MLD8lAbYirC3hUtUxGfjSGWk6GkGvz0uEc8613SdiKVaF2Nk7g8hSnlVkRNXAnPwp6nUtwBKu\n1u/30Wq10G63sbm5ia2tLWxvbw//OOmOqGZePuEXpZSeAPBFAL8L4DU55yfn9F5kgfFExgs70xe5\nFVw7FZBP50evAAAgAElEQVQXo6vXZTlPS9gbLIwGFOtGiUTuGq9/VbT46nVg4G6xURA6Tbndbg+j\nJXLOQwHudDrY3t7Gzs7OsGYGLeFq5iHC7wbwawA+BeBrAPwUgHellJ6f+Xe4VtQdlLMCbLOzrPjq\njDS9tNtsSu0sz8tzG3h9zwrU616/7nJW52IRd4Scp64XbPsARlwQOzs72NvbGxNhXvoxMxfhnPM7\n1OpfppT+AsAnAbwIwPtm/X5keqILo3TBVF1M+pa2TtyvxNyWmli30hfBjZrsM08RbjphZJ7I+QGD\n71GiI8RC1tMXbW1tVWb5LeNncJ3MPUQt5/yplNIXADwPBRG+c+cObt68ObJtf38f+/v7cz5CEt0W\nR49Jv7T0YmGj5Apr8UY+3263O+J+EHHWAmArj83r89Iia9sqYGfM8PqtVmuYLdfpdIZZddpdIYK9\nyhwcHODg4GBk2+HhYe3nz12EU0rPBvAMAJ8r7ffwww/j1q1b8z4cYqjji/T8kVWuhlKKrl3XlcdK\nU8LbQTivXKQV+nlhfdurJMS6+I74gDc2Nty2vb09rBkhIizpzSLUqz7tvWcs3r17F7dv3671/Gni\nhPcwsGrlU/3qlNI3Anjysr0WA5/w45f7/TSAjwN4dNL3IvPHGzCzg2ZWbGx4ldePstls5bDz89GJ\nMOv0I7HWQizHPO/PzPvDkn2WFS3A0sS6tW1nZ2coxOtqCV+VaSzhb8LArZAv2xsvt/8ygB8E8A0A\nXgHg6QA+i4H4/kTOuXfloyVzIfJnev7bOoKdc54os61qIkxvYkxv23W6IyIhlseXGW0Ji6CKhWv7\nWoC1CIt4U4SrmSZO+PcBtAq7fNv0h0OuG88SLhXEKcWx6u3RbBGludbqrpey1GyCxDw/s8h1o/db\nVrQPWAuvhKLpkDQRYi3Oku5MEa6GtSPIUDyi2FftZ43CsOxjXkqxnQDTE+Q6IlvHxTFPS1h/ZqW2\nrHg+YZ2MIVaviHDkE9aWMIU4hiK85lhLWAuxbZGFHEU8VKUW64kwvToRXt2I6E/C+8OYpxBGUSTL\nLL6C9QnrZAxbwH1nZ8d1R9AnXB+K8JpTEuHIwvSyw+zSCzvzQs90UL9n3XrvX2WRX0d8alW43rKL\ncWQJiwiLG0K7I6xPmFZwPSjCxHVHeFXNIuvY224nuywlYEQ1F7zX9gYDo/48P6+q/jIjAqytYWsJ\nWwGuio6gEMdQhNccL9ohKqgThZ1566XaDnabuBmqrGxr4Va5BK5TFFdFgIEv+4StO0J8wtoV4SVr\neD5hEkMRXgKmucC959ht2vot1erVxbut6yBaWrGNZiWWGg91fc2LghUWWZ+14NT5HueBFmIbpmb9\nwiUBpghXQxFeMqpuheskVOhlVeytbp7FGzWv5oNMfOnF9NqwtzoDXZEQeo9F2+o8Fu2vb7V1RIHd\nNsnr1/3uIot/Vu4Re+z6/KyrwmbGWRcERbgMRXiJ8G69vX7dCl9eKFlJiD1fbdT3qqB50RBR1bEq\nIfYu8mhZ6nvrVVifaVRXQSbGnET8qwYb9bLqtyDrk+KJry3cE7Vp/3zWGYrwkhDFoXqJA5MMoFXF\n8+rmFeGJ1r10Yy+7zbOCozoVQmSledv0c0rLSUgpDQeedHZY1Pee723LebzmRvT9yf514pRnLcQl\n8fWEmJShCC8ZpWwtna0WJTp4UwiVmq7JUMqks9vqZL7ZmF4vC6/KHRG5AKos5auIcUppJIW3tNTT\nxOvne/2cq9O9tY81+g3o17sKJfG1rgjPEqYQ14MivCRE4mtDs7QAelZoZJnWafaWOMqYs38EXtKF\nzWzzzidyR0Q+WNu3++p177G6tFqtkdTd0rLdbtd2j+Sci/UzrADLZ5VSGvZbrdbw8xNreVqiP7Y6\nFjFdEvWhCC8RkQDb8DItwlF9XllOIsLe+0XN3k6X4oxLA1F28KnqNrnKMi65MeoicbM6OsBbSj0F\noSTGIpieD92Gesn3LMeurV8RZXm9aYXYHmMkvHZgjoNyk0MRXiKsJewVzxHB0wNjXmyu9CcRYc/y\n9rZ57gkv/tfG/tbxbwLjQmxFoHRrXCXOddjY2BhJWrBL3e90OmPHrJe6LyJ8cnIyVpdX+4D1Zxcd\n/ywEeFILOLKE7fmSUSjCS4IXgua5BrQlXGc6oElFOPJHe1Z6HbeFzm4rDTpaSgJhM7U8d8WsRFiS\nFiRxQfq6zq4cr13abTlnHB8fj1UiEzHViTS9Xg8ppZHYaf25TWPh289XH6d19XjRIKXPmcRQhJcI\nT/C8mg8S9SCW8PHxMY6OjnB0dIT79+8P+0dHR0XRFYta+lXRGXq9ylrW2+R5dZZCZJ3VEQYrEuI/\nroue2FKEd3d3F7u7uyP93d3dERGuEuKLi4ux2gty7laA2+32WPKKfPazcgfUtYIjvzCpB0V4CSiJ\nr73116nGek42EeCnnnoKTz31FO7fv4/79++7wmsHhGRbFPbkCWVdsbbiWrUOjIpYFLuqrWFPgL1t\ndRER1lO87+7uYm9vb7iUtr29HYqu7WsR9gbh9HerQ9889434hq/CJO6IKEaYYlwNRbghJvHV1bF+\ndV9bweKO0CJ87969oRhH4uuJ86ypc4HafaLBIRFdPSWPFWhPhKex3HRBc3E9aPG9ceMGbty4gb29\nPezs7LjC5AlVztkVXxvyJ5ay7LOxsTEmvvNwR0wSpkaLuD4U4QapGozSF1YpRdhawJ7Fq6eD9+Zk\n86IVdERC6WLyHiv5Xj1/oX2N6DWt0JaaJwwlYa5Lu90eEV3b393dHRmgm8QS1qFtdqoge06e4E7r\nhtD7W5Et1Y6wdSO0K4VCXA+KcINEYWY2WUEsoarCOVqExfLVPuDj4+ORUDUbt2uz1zT6QqoSzEl8\niJ7VVhIUO/lkad2zhktRFHURERYL2C6lbwfm6viEraDZspBVkR/e9+ER/ckB4yIcCbEOydMCzDKW\nk0ERbohSOJfNQvOSHUoTZmrR1YNwOkTN1oPwEic8SzgSY/u45x6IttURKP26+kIvLavCqey2umxs\nbIxEQ0RLsRa9z8dbik/Yq8vrFcspWcTed1T63uyxaCGuEmDPEqY1XB+KcIPomM+qOg/Wbxv1ZTBO\nh6HpdeuO8OJ3IyGeZGkF0fathSfPLVl3KaVar2utsbpiXJdWq+XGB+uljROu85lZS9haw/a4rfhG\nn52m9GcaWcJ1rGCvmDsFuB4U4Qax/l5vIkvZVncaeB0RESVqiDvCpiJ7yROAP5ikt3t97+KN+tp9\nUSUqnpB7AlwS4WgwqS4SJ+xlytltk6QtiyXsuSI8F0uV4Fom+SOdtRCTGIpwQ1h3hPXrRuFitnnb\n7VRC0baqVGQdwRFZplUirC9YW1tB+qWwMbtuRdgKlideVSLsFdkp0Wq1xqZ/j/rtdnv4mejP0ut7\nPmHr6y7506cVZM8SriO+duZl70+DAlwNRbhBrDsiKrxTqs0b9es0nYrsLYU6bgLb5ILUF6mtqyD9\nqvAxvZS6DZ4I235U69frTyrC3h+At67jeSPxlfV+v++6I6oE2BPf6HzsPt73OKkVXIqOoCVcDUW4\nIXT4mUQ/aBH2hLduq0rA0E0fi5dQIVS5CSK3gZ2h11uKxVSKXtDC4JWN9LbZwUArvHo5iVDo87Ou\nAm+bfa73esCgoLvnD45cEdNGSdT5I51EiEvRHJNY5+sKRbhBPHeEiKM3M3Fpqf29dSIpxOfsJY2U\najXU8duWRFianiSyagBNi7EWAivCnhhHomu3yXF76dH2zwhA8ZjtYxFWmMQS1rf2JddKSXjrit5V\nBbiO5U4BLkMRbggvEcOWoIwmy4wmzDw+Pka3261VQlIiIYDySLm+wCM/rbdNLFYb3K8L3Eg8reej\njQTOE13rg/Vujb0QOd2f1BKu07QA13l9awl7olYamCsJ8CR3MZOIsAw+6j8+z2onMRThBin5hG3a\ncRRuZquinZ6ehjNg2BoTdUOcrODW6esL01rCusDN7u7umMUaDZ6JCFvrKxoUs24BK7xW2JrG+oRL\nlnDJpaCpEsCrCLFYwlVRKaQMRbhBbGqyjRkuTVFkrdyqEDNrlbVareFMDKXb/6rtdh9Z1xavtnq9\nLLOS+Foh1hZ25JLQolB6TfvHMa/vuNTX27w7ldKUUl4dZ1sUKapREgm2J8T2j8uLx47cJfK6JIYi\n3CB1SlJGvtzSTBWeAOsLS4t/yTqMBraqxLLVarm+X5tVJn1r4ZVet050ROm4mxKHOmVAS/VBIlH2\nxDiqUCdE4qv73ndRNQgZ+atJGYpwg1gRrhLgyM9rL0yNHnASa1hfnNai8W4tI2ErLXXWmO170RF1\nLWx9bPZYpxnIus7v2RNJ3eydjf1+vTseWyC/SoAt0XhAlRDrz9r62Zv+vJcNinBDWGtoUku4ZCl5\ngzQiwPZC8yzJaJtnJUcWdDTvWhQnbN0D0TKyxEq3x56gX7dAlOKxZVmqkFflmoiE3VISXd2vcklU\n3SlRfOtDEW6QWQmwvSC1aAEoCpsd1CplgUW3od66zY7zMud0xpy9cK0LRfcjMYhEwbOwr1MgonrQ\n1o1QVaq05IqILOEql0RJlO0AnecbrvLnU4iroQg3iHdxloTY1peIbls9QfMuFLFYbfZT1CJXhefK\nKNWKsOue2JZG7b0LPxKD0usC1+cXLhXml7432Bq5nLx5+my2Y12XBOAncHj+eO0S0oOf9o+O4lsf\ninDDlAblSn5hzxLWsb/a/WBv4/VFJK6BOs3zvUatju9WX8T6WCOxjAbt5Py8x6tE/TqI/mz1d35x\nceF+v15o4SSWsMb+8dRxR5RcEl5mXFPunmWGItwQJctIxw1PKsTiE7bREVFkgTdNe7Qe+Ys9f3LJ\nX2xbSQy8fl3/cUnMr1scPFeEJ7DTRkhEPuGruCM8S9iKcLvdXog/umWGItwg+kKJLkhPeCMx1u4I\nmRlD38J7AfdRQXKvX8e14BVxiSxY/bgcq7e02yZp3nOj95gX2j0Q/eFGIhyJ7jSWcER0x2H/wG10\nhP4TrbKqSQxFuGG8C7PkjqgKT/MuRCvENovNS6qwy52dHXcQLxrQKwlj5Df0Lla7bRJhjZ4bvdc8\n8GKCPddTv993v9tZ+4TtZ1Dnj67KEo5e224jPhThBokuTGl1BDiKkKgjwKVUYjuF++7ubhjx4C3r\nCOq0LNuFHUVGlFwR9rudRIirLGHvz8kT4OjOxVrD5GpQhBtEC6QtjCLCeH5+Psxsi8Kx5DVSGtSk\nLSVK2GSJKutXz5XmWb/WN2z9vGRAKUzNuiQiofWEt+QLtndDupVCy6K6FdbFJK9LrgZFuCEiC3V7\ne3to8cqFllKqrJcgcbwiwt7gmjfwZv2/1g9sBTjKSqPvLyZySXh3P5P6gCMx9tC/uWigVETYft9W\ngPldzw6KcINYK3hrawu9Xm/Mt6vjeUsTQW5sbOD8/HysPkNVKwm1TFZpa9xay4gXZ5lSsoa1hiMh\njsTYpkPL+2msmyHKONS/taoKbmQ2UIQbwrOEt7a2Rixg2U+E2rOCPRGOCuV4S5tK7K2LJewlaFhL\nmBenT1WaepXwettK9Sj0ewreIJsXux1N3Ekhng8U4QbRItzpdIYXmh1Uk8cjP6wnwl6YmbfNphF7\nqcXSqvzSvCh9IgGu65Ko64IouSSiSIco5puuiOtjIhFOKf04gH8P4B8DOAHwxwBenXP+uNnv9QC+\nD8DTAfwRgB/IOX9iJke8IniWsFxw+nEv9beUpaZF2A6w2e069reqWWsockXwAvXxRHISIfYEue6g\nnFBHiG2oIa3g+TOpJfwCAD8P4E8un/tTAH47pfRQzvkEAFJKrwbwwwBeAeDTAH4SwKOX+5zN6sCX\nHfkhy4/fCyuLai9EAqxFWMLMqqIfJklFjmoJ0BKupkp4S0LsPcfGA0fCa6MjqsRX3wlxEPZ6mEiE\nc84v1esppVcC+DsAtwG8/3LzjwB4Q875ty73eQWAJwC8HMA7rni8K4W2hCML2JtSPBJgEWEd66sF\n2TYpqF63eRlwpeQLMqAUHWETdKLICM8toV/Tex/LpFZwZAlLrQ8yG67qE346gAzgSQBIKT0XwLMA\nvFd2yDnfSyl9EMDzQREeYt0Relu73Uav10On0xkuSy6IkghLooXX39nZKQbkVxXD8bYRnyo3hBch\n4T3uibBeRtv092VFuK4lTHfEfJhahNPgW3gTgPfnnD96uflZGIjyE2b3Jy4fIwq5GHLOIwLc7/eH\nA3WSulwKoNfrvV4Pe3t7I9luUX93d7dWWrFsA8qVt/Tj5MuUfMF1oyQ8Idavb9/Pw2bBTWoJ6zoR\n/J5nx1Us4UcAfD2Ab5nRsawVVtg2NjbCUfDz83PXAvZuFc/Pz4diK+3GjRtj28QS9lJYS+tV20mM\nDierEt9IeK0lXJcoTliL8CSWsLwmuTpTiXBK6c0AXgrgBTnnz6mHHgeQADyIUWv4QQB/WnrNO3fu\n4ObNmyPb9vf3sb+/P80hLg2epSnrcsGmlEZC2OxFqJ8j7gixdG08sL7ANjc3mzrttUG7BiLrN6qM\nV4qIqIsIpQioZ/nqBB2bLWkFWQ/OkQEHBwc4ODgY2XZ4eFj7+ROL8KUAfzuAF+acP6Mfyzl/KqX0\nOIAXA/jzy/2fBuCbAfxC6XUffvhh3Lp1a9LDWWpSSkORlaVsBzAU4iiUzasN0O/3ixcSL6Lrx7oi\nvKI9XpGmSUW4dBej/cDWB6wFWP9xS7akTovXFjEZ4BmLd+/exe3bt2s9f9I44UcA7AN4GYCjlNKD\nlw8d5pxPL/tvAvCalNInMAhRewOAvwHwm5O817qhLxh78eScx4RYhyjJc1qtFvr9/lhKslfnlyJ8\nfdjMNs8KjqY2siFpUdRDVd/zAdtqejahx7t74u9n9kxqCb8Kg4G33zPbvwfArwBAzvlnUkq7AH4R\ng+iJPwTwbxkj7OP9mMUC1vvoUDZPgMVKubi4GKuWpi8ixvQ2g2cJa2s4soSrrGBPdL2l/ROPXBE6\nhlynsdMdMT8mjROudQ+Sc34dgNdNcTxrj3ZF6NtYHQlhXRD6MamiZifvpCXTHJE/WFu8djJXzwqu\nsoY98dW/J/070WVTdTW9kjvC/onzNzQbWDuiYewPWfuGBRtPrLfrwZZer4eLiwt3ynptyfACun4m\ntYS9aIm6Auyte8kZVT5hbQlbnzB/P7ODItwgMiAn/dLjIraCtmzOz8+xubk5rMBmi7F4oUa8iK6P\nUpKGN6FrNDAXpSYLXuKMF5Zm3RFVPmFGR8wXivACYKMjLHIB6fV+vz/igtAV2HQMsY4l5kXUDJNE\nR9hpjkqxwV6iTdRKA3ORT5jRNdcDRbhhIgG24WuyTQbtouQOHUnhpSBLn1wPUZzwJO6IupawUBJh\nL0TN8wlbV4QnwhTi2UARXgDkx2zFWAuxTeCISiPK8/SFEvXJ9RClLJciI6LJW+u4Iuy6l6Zs5zPU\nlrCeTSWKjuAf+eygCC8QkV9YEFeD9PX2qtdjqmmzlFKVRXTPzs5Cn3ApOsITXvunqwdxq3zCUmfa\nzqDNgbn5QBFuiGl+xPzhLwalOxG77fz8HPfv38fR0dGwHR8fj7STkxOcnJzg9PQU3W4X3W7XFeSS\nX9grM6qbV6jfzpxSckEwW25+UIQJmRAv1CwqzN7r9Ybie//+/RFB1n1PjD0h9lLVS75/6du4cRvG\nGFVPiwq60yCYHRRhQqYgSjm2y263i+Pj46HQ6qXddnx8jNPT0zGL2FrCdqBWi25UkN8KcDR7d2lG\nDa+uNLk6FGFCJkQsYfHj2ibiKX3reojWT05OhuKrLWEdMeGlrNsQNG8ZCXBJiG1YIwd15wNFmJAJ\nyTmPWMLdbndowXpN3Axes49bQbcCbC1haw17dYJ1NEQkuvaxqumtKMSzgyJMyBRoS1iENLJuS+Ks\n17vdLnq93ogFbC1hGx0RhaDZCWKtyJYEeXNzc8zFQQGeHxRhQiZE3BEimNrv6w24icBqX6+33u12\nx0paesV8BC8Rw6YmR5EQniDbQj3ePIMU4tlDESZkQrQ7QlvCR0dHeOqpp0ba0dHR0L8bLXXfVlfT\nscTRwJyXlqwt4EiEvTnltAh7ccYU4NlDESZkCsQdIT5hEeH79+/j3r17ODw8xOHhIe7fvz90L2g/\nr+2L26E0x1wdd4RnCev4X0+A7bLdbodJH4yMmD0UYUImxEZHWEv43r17+NKXvoQvfelLuHfv3liV\nNOvr1X1t7UZ9b2DOuiM8X3CVS0Jbwvr1S31ydSjChEyIdUdon/BTTz2Fw8NDfPGLX8STTz6Je/fu\njVVGi5q2dHXhH29d8AbmIl9wZA3bxI0oLZnCOx8owoRMgc6a07UfdLiaREnofTxfr17q17fvJ4gY\n6sGyqlmUbZF2aV6taV02lcwfijAhU2Iro0XlKqO54qpmy/CEV1NVI1gXabezZbBQ++JAESZkCjx3\ngZ1NuSTEXrEfS5UoTjJ5p5So5LyDiwdFmJApieoEa0vY2x4JcNVUV3a9VCPYK9Remi2Dg23NQREm\nZApK5SyjCmvWbeEJsMWrCS39OpawWMFV0xXpOGByvVCECbkCkQDbfpUQl9wR3jKl5GbJeUXa7TT2\nnDdusWB1ZkImpMofXBqY82J+q6xhwJ89w1rC1h3hTd5ZNW8cuX5oCRMyBXZ6qUkEOHJleFTNHRcl\nZ0SWsA5TY3TEYkARJmRKqgbmdKpxHUtYv240OBdN3lnlExYB1pN4ct64xYAiTMiUePG+1hL2rODI\nFWHTkauanca+FB2hLWCJE6Y7YjGgCBMyBZFf2HNJRG6IqmnsNdEMyqVkDe0T9lKVPXcEhfj6oQgT\ngnEfr9eXdZty7E32WYoJrvIBe/V7vbq+Yt3qptOUdYvmkbMF28n1QxEm5JJILG3TsyHbeeBsJbQq\n0QVGIx+iiTq99sADD+DGjRu4ceMG9vb2wgE4b+44zpaxOFCECbmkVAdCr3sirAv0ePPByetrRPj0\nQJwdbNMFdmzB9gceeAAPPPAA9vb2xkTYDsBVzaJMIW4OijAhGA8ziwqraxGOZkTWsyJHMcBSoMcK\nnxftYEtTylIsYRHgKktYz55MAV4cKMKEXCKiaf281v9bR4CjQTdbGU0Lsfh+7SCbrQUs/cgdYVOU\nO52OO3MyhXgxoAgTcom1hKPi63p25MglIa+jX9vD1oOwIWd6wM0utQB7PmEbD6xD2jhx5+JAESYE\no+4Ibf16UxCVBuYid4SHFT5doF2LsAirtnJ3dnawt7c3FGLrjrA1IkozJ1OIm4UiTMgl2hLWlq8W\nWplTzrokPCvYzoys8aqjiSWsS1OK1SsCK2IrTQR4b29vpIC7Lt6+ubkZhrxx8s7moQgTcon1CWsL\nWKxemcKoFKJmLWHP9yvYmhDiMvBmyRC3gzTZpgU6soS14FrxZYxws1CECQHGQtK0CIvw6jnkrBDb\nEDU7Pb0XDWEL83hV0bQlrN0PN27ccIu2e1MZdTqdsAiQt06uF4owIZdon7CIqraE9SSepQgJzx0B\nlAu0a6vU8wmL2N64cWMYHywWss6Miwbmqt6fNAdFmKwsdWoy6H3twFwkwCcnJyODc547whuU8wrw\n6PW6c8WJRexFTURzyFFoFxeKMCFALQE+OTkZTmOvhbjb7Y5ZwtoNUar/oLe32223HoRXH6IkuDoS\ngiw+FGGy9ohgWleEZwEfHx8Pm2cNezHC2t9bWm5uboYFeHTz6gJHpSlpAS8+FGGylniZbNoajixh\nEWCxhE9PT0fC1PSgHDAe9VBqeo64qqpoUhlNW8HWEqYILwcUYbJ2eL5iOyinw9KsCB8dHY35hSVC\nInJH2JkwdDywrMu0RNbq9URZxwDb4jycPXm5oAiTtSKq5RBFRnj+YLGET05O3IQNGyPsWcIinFpA\n7WBc5IqQfbzqapyuaPmYyHOfUvrxlNKHUkr3UkpPpJR+I6X0dWaft6SULkx712wPm5DJKRXTsXUj\nPHeE9gdXDcx5PmFvJgwvEqKuS8IbmGOh9uVjUkv4BQB+HsCfXD73pwD8dkrpoZzzidrv3QBeCUB+\nAd0rHichV6JKgD1LWGfHWZ9wFCtcckdEAiwiGglu5I6wFdFsnz7h5WAiEc45v1Svp5ReCeDvANwG\n8H71UDfn/PkrHx0hc8ArtK5Tlm2ShhcdoTPoJh2Y0+4H3SLBjdwRXsibbWTxuapP+OkAMoAnzfYX\npZSeAPBFAL8L4DU5Z7sPIddOSYBtBTVvYO7o6AhHR0cjIi1Lb2AO8K1hawl7olvljoiSPlgZbbmY\nWoTT4Nt9E4D355w/qh56N4BfA/ApAF+DgcviXSml5+dJUpgImRFRNIRFV1ATkRWXgx2Y02Uubd8O\nzGkB9lwSVRN2RmIMjBYAipYU4sXmKpbwIwC+HsC36I0553eo1b9MKf0FgE8CeBGA913h/Qi5EiUx\n9qY38upH2HA0W6zH1oDIOYeWrxVWW4AnGoSTRlaDqUQ4pfRmAC8F8IKc8+dK++acP5VS+gKA56Eg\nwnfu3MHNmzdHtu3v72N/f3+aQyRkYmwlM11IR7sMxNr1prq36wDCjDe7bWdnZzhxp1RG0+UoWQdi\nMTk4OMDBwcHItsPDw9rPn1iELwX42wG8MOf8mRr7PxvAMwAUxfrhhx/GrVu3Jj0cQmoj5SSjx0q+\nW2252kpptunHU0pjwhv1t7e3h8V5dnd3x2ZN1okYZHHwjMW7d+/i9u3btZ4/kQinlB4BsA/gZQCO\nUkoPXj50mHM+TSntAXgtBj7hxzGwfn8awMcBPDrJexFy3XhRDJ4LQdwP2ver+3o9peSKbrTUs2bQ\nEl4PJrWEX4VBNMTvme3fA+BXAPQBfAOAV2AQOfFZDMT3J3LOvSsdKSFzQBdb11XNdBaaTaiQRAzr\nT/a2tVqtcLJOr2/nkdMpysyEW00mjRMu3gflnE8BfNuVjoiQhqhrCU86eacnuKXylNZNQUt4tWHt\nCNTkwxwAAAbiSURBVELg+4R1gRxtCev9vSmC9LaNjY1aNYJtfWAbDSEiTJ/w6kERJmuJN0hXio7Y\n2trC2dnZWJKEnTbeLksirK1r7fuNGi3h1YQiTFaeUlSE3c9zR2gh7vV6Y0Xabd8WbI+E1/YlAsKW\nutR9+oRXD4owWXtsuq+1hrWLoNfrjZWlLBVslymLPPG1Sz34Fgk8LeHVgyJMyCWRO0LihCWDzivM\nHvWtCHs+X+2KiGpBeD5nshpQhMlaEbkmohoP4orQMynbQup2Zgu9bqMrbOoxU5EJRZisLVqQbclJ\nLb62Mtr5+fnYbBbeLBfanRHNA8dqZ4QiTNYOK77S91wREhes9xcRtm6JaGmjGyjAREMRJmuBdUOU\nrGCxhLUFrLPq+v1+5czJWnC9+eQoxESgCJO1xQqxiGK73Ua/30en0xl7fGNjYyjCpSgJr36wdUdw\nCiICUITJGhElaOg6wNp6LQl0KYTM9rXwWiuYljChCJO1IhJiK7TWVaHF+eLiYkxES/O8laxlJl8Q\nijBZO0T0rBiLQHoCLINx4ieuSlfWAhtl09l4YLKeUITJ2qLFWAuuPCbrEhusi7VHCRXRpJueONMS\nJgBFmJChCOrqZGKx2lkzdKSEPLeqmlqdDDgK8fpCESYrS93CPbKvtlD1LBm6ARgRYXmu7dttdrsV\nbbK+LHRhUjt53iqxyucGLM751bFAbZywN5vGzs7OcNqhd77znWPTENlZkqMSlV7W3CIJ8aJ8b/Ni\nEc+PItwQq3xuwGqfH89teVnE81toESaEkFWHIkwIIQ1CESaEkAZZhOiIbQD42Mc+NvbA4eEh7t69\ne+0HdB2s8rkBq31+PLfl5brOT+nZdtW+qW4Iz7xIKf0HAL/a6EEQQsh8+K6c89tLOyyCCD8DwEsA\nfBrAaaMHQwghs2EbwFcBeDTn/PelHRsXYUIIWWc4MEcIIQ1CESaEkAahCBNCSINQhAkhpEEWUoRT\nSj+UUvpUSukkpfSBlNI/bfqYZkFK6bUppQvTPtr0cU1DSukFKaV3ppT+9vI8Xubs8/qU0mdTSscp\npf+VUnpeE8c6DVXnl1J6i/Ndvqup461LSunHU0ofSindSyk9kVL6jZTS1zn7LeV3V+f8Fu27WzgR\nTil9J4A3AngtgH8C4M8APJpSemajBzY7PgLgQQDPumzf2uzhTM0egA8D+EEAYyE2KaVXA/hhAN8P\n4J8BOMLge+xc50FegeL5XfJujH6X+9dzaFfiBQB+HsA3A/g3ADYB/HZKaUd2WPLvrvL8Llmc7y6q\nm9pUA/ABAP9VrScAfwPgx5o+thmc22sB3G36OOZwXhcAXma2fRbAHbX+NAAnAL6j6eOd0fm9BcCv\nN31sMzi3Z16e37eu6Hfnnd9CfXcLZQmnlDYB3AbwXtmWB5/a7wB4flPHNWO+9vIW95MppbellL6y\n6QOaNSml52JgXejv8R6AD2J1vkcAeNHlLe//TSk9klL6B00f0BQ8HQNL/0lgJb+7kfNTLMx3t1Ai\njMG/1gaAJ8z2JzD4YSw7HwDwSgwyBF8F4LkA/iCltNfkQc2BZ2Hww1/V7xEY3M6+AsC/BvBjAF4I\n4F1pUaqz1+DyWN8E4P05ZxmbWJnvLjg/YMG+u0Uo4LM25JwfVasfSSl9CMBfAfgODG6RyJKQc36H\nWv3LlNJfAPgkgBcBeF8jBzU5jwD4egDf0vSBzAn3/Bbtu1s0S/gLAPoYOMw1DwJ4/PoPZ77knA8B\nfBzAUow8T8DjGPjy1+J7BICc86cw+P0uxXeZUnozgJcCeFHO+XPqoZX47grnN0bT391CiXDOuQfg\nMQAvlm2XtwgvBvDHTR3XvEgp3cDgiy/+SJaNyx/14xj9Hp+GwYj1yn2PAJBSejaAZ2AJvstLgfp2\nAP8q5/wZ/dgqfHel8wv2b/S7W0R3xM8BeGtK6TEAHwJwB8AugLc2eVCzIKX0swD+JwYuiH8E4L8A\n6AFYvImvKrj0Yz8PA6sJAL46pfSNAJ7MOf81Br6416SUPoFBhbw3YBDl8psNHO7ElM7vsr0WwK9h\nIFjPA/DTGNzVPDr+aotDSukRDMKxXgbgKKUkFu9hzlmqGC7td1d1fpff62J9d02HZwRhJT+IwZd/\nAuB/A/impo9pRud1gMGP+QTAZwC8HcBzmz6uKc/lhRiE/vRN+x9qn9dhEO50jMEP/HlNH/cszg+D\nMoXvweAiPgXw/wD8dwD/sOnjrnFe3jn1AbzC7LeU313V+S3id8dSloQQ0iAL5RMmhJB1gyJMCCEN\nQhEmhJAGoQgTQkiDUIQJIaRBKMKEENIgFGFCCGkQijAhhDQIRZgQQhqEIkwIIQ1CESaEkAahCBNC\nSIP8f+zIP5HvZio8AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11f59f290>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "b = mnist.train.next_batch(1)\n",
    "print b[0].shape #b[0] contains the image\n",
    "image = tf.reshape(b[0], [-1,28,28,1])\n",
    "print image\n",
    "my_img = image.eval() #here is your image Tensor\n",
    "my_i = my_img.squeeze()\n",
    "plt.imshow(my_i, cmap='gray_r')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 0, training accuracy 0.04\n",
      "step 100, training accuracy 0.94\n",
      "step 200, training accuracy 0.98\n",
      "step 300, training accuracy 0.9\n",
      "step 400, training accuracy 0.96\n",
      "step 500, training accuracy 0.98\n",
      "step 600, training accuracy 1\n",
      "step 700, training accuracy 0.98\n",
      "step 800, training accuracy 1\n",
      "step 900, training accuracy 1\n"
     ]
    }
   ],
   "source": [
    "batchSize = 50\n",
    "for i in range(1000):\n",
    "    batch = mnist.train.next_batch(batchSize)\n",
    "    trainingInputs = batch[0].reshape([batchSize,28,28,1])\n",
    "    trainingLabels = batch[1]\n",
    "    if i%10 == 0:\n",
    "        summary = sess.run(merged, {x: trainingInputs, y_: trainingLabels, keep_prob: 1.0})\n",
    "        writer.add_summary(summary, i)\n",
    "    if i%100 == 0:\n",
    "        trainAccuracy = accuracy.eval(session=sess, feed_dict={x:trainingInputs, y_: trainingLabels, keep_prob: 1.0})\n",
    "        print \"step %d, training accuracy %g\"%(i, trainAccuracy)\n",
    "    trainStep.run(session=sess, feed_dict={x: trainingInputs, y_: trainingLabels, keep_prob: 0.5})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "testing accuracy: 0.984700024128\n"
     ]
    }
   ],
   "source": [
    "testInputs = mnist.test.images.reshape([-1, 28, 28, 1])\n",
    "testLabels = mnist.test.labels\n",
    "acc = accuracy.eval(feed_dict = {x: testInputs, y_: testLabels, keep_prob: 1.0})\n",
    "print(\"testing accuracy: {}\".format(acc))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}