05. Decision Trees.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Decision Trees\n",
    "Let's start with a large dataset and find out how we can use decision trees when our data size is large-sh. We will use the kaggle dataset that attempts to classify plankton. We will use some example code to get us started from the tutorial here:\n",
    "http://www.kaggle.com/c/datasciencebowl/details/tutorial\n",
    "\n",
    "*For questions on this notebook, please contact Professor Eric Larson at eclarson@smu.edu*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of classes: 121\n"
     ]
    }
   ],
   "source": [
    "# load another dataset (large) and train using various methods of gradient (and mini-batch)\n",
    "import glob\n",
    "import os\n",
    "\n",
    "# change this to point to the dataset on your machine/cluster!!\n",
    "directory_of_dataset = \"D:/SMU/Larson/2019 Data Mining 7331 Updates/Class Github/kaggle_plank/\"\n",
    "\n",
    "# get the classnames from the directory structure\n",
    "directory_names = list(set(glob.glob(os.path.join(directory_of_dataset,\"train\", \"*\"))\n",
    " ).difference(set(glob.glob(os.path.join(directory_of_dataset,\"train\",\"*.*\")))))\n",
    "\n",
    "print ('number of classes:', len(directory_names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading images\n",
      "5.0 % done\n",
      "10.0 % done\n",
      "15.0 % done\n",
      "20.0 % done\n",
      "25.0 % done\n",
      "30.0 % done\n",
      "35.0 % done\n",
      "40.0 % done\n",
      "45.0 % done\n",
      "50.0 % done\n",
      "55.0 % done\n",
      "60.0 % done\n",
      "65.0 % done\n",
      "70.0 % done\n",
      "75.0 % done\n",
      "80.0 % done\n",
      "85.0 % done\n",
      "90.0 % done\n",
      "95.0 % done\n",
      "100.0 % done\n"
     ]
    }
   ],
   "source": [
    "# Rescale the images and create the combined metrics and training labels\n",
    "from skimage.transform import resize\n",
    "from skimage.io import imread\n",
    "import numpy as np\n",
    "\n",
    "#get the total training images\n",
    "numberofImages = 0\n",
    "for folder in directory_names:\n",
    "    for fileNameDir in os.walk(folder):   \n",
    "        for fileName in fileNameDir[2]:\n",
    "             # Only read in the images\n",
    "            if fileName[-4:] != \".jpg\":\n",
    "              continue\n",
    "            numberofImages += 1\n",
    "\n",
    "# We'll rescale the images to be 40x40\n",
    "maxPixel = 25\n",
    "imageSize = maxPixel * maxPixel\n",
    "num_rows = numberofImages # one row for each image in the training dataset\n",
    "num_features = imageSize # for our ratio\n",
    "\n",
    "# X is the feature vector with one row of features per image\n",
    "# consisting of the pixel values and our metric\n",
    "X = np.zeros((num_rows, num_features), dtype=float)\n",
    "# y is the numeric class label \n",
    "y = np.zeros((num_rows))\n",
    "\n",
    "files = []\n",
    "# Generate training data\n",
    "i = 0    \n",
    "label = 0\n",
    "# List of string of class names\n",
    "namesClasses = list()\n",
    "\n",
    "print (\"Reading images\")\n",
    "# Navigate through the list of directories\n",
    "for folder in directory_names:\n",
    "    # Append the string class name for each class\n",
    "    currentClass = folder.split(os.pathsep)[-1]\n",
    "    namesClasses.append(currentClass)\n",
    "    for fileNameDir in os.walk(folder):   \n",
    "        for fileName in fileNameDir[2]:\n",
    "            # Only read in the images\n",
    "            if fileName[-4:] != \".jpg\":\n",
    "              continue\n",
    "            \n",
    "            # Read in the images and create the features\n",
    "            nameFileImage = \"{0}{1}{2}\".format(fileNameDir[0], os.sep, fileName)            \n",
    "            image = imread(nameFileImage, as_gray=True)\n",
    "            files.append(nameFileImage)\n",
    "            #axisratio = getMinorMajorRatio(image)\n",
    "            image = resize(image, (maxPixel, maxPixel))\n",
    "            \n",
    "            # Store the rescaled image pixels and the axis ratio\n",
    "            X[i, 0:imageSize] = np.reshape(image, (1, imageSize))\n",
    "            #X[i, imageSize] = axisratio\n",
    "            \n",
    "            # Store the classlabel\n",
    "            y[i] = label\n",
    "            i += 1\n",
    "            # report progress for each 5% done  \n",
    "            report = [int((j+1)*num_rows/20.) for j in range(20)]\n",
    "            if i in report: print (np.ceil(i *100.0 / num_rows), \"% done\")\n",
    "    label += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num features: (30336, 625)\n",
      "num labels: (30336,)\n"
     ]
    }
   ],
   "source": [
    "# here is where the online tutorial code stops and my code starts\n",
    "print ('num features:', X.shape)\n",
    "print ('num labels:', y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "StratifiedShuffleSplit(n_splits=1, random_state=None, test_size=None,\n",
      "            train_size=0.5)\n"
     ]
    }
   ],
   "source": [
    "# now divide the data into test and train using scikit learn built-ins\n",
    "from sklearn.model_selection import StratifiedShuffleSplit \n",
    "\n",
    "cv = StratifiedShuffleSplit(n_splits=1,train_size=0.5)\n",
    "print (cv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Once:\n",
      "Wall time: 17 s\n",
      "accuracy: 0.2579113924050633\n"
     ]
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn import metrics as mt\n",
    "\n",
    "dt_clf = DecisionTreeClassifier()\n",
    "\n",
    "# now get the training and testing\n",
    "for train, test in cv.split(X,y):\n",
    "    print ('Training Once:')\n",
    "    # train the decision tree algorithm\n",
    "    %time dt_clf.fit(X[train],y[train])\n",
    "    yhat = dt_clf.predict(X[test])\n",
    "    print ('accuracy:', mt.accuracy_score(y[test],yhat))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from the tutorial: http://www.kaggle.com/c/datasciencebowl/details/tutorial \n",
    "def multiclass_log_loss(y_true, y_pred, eps=1e-15):\n",
    "    \"\"\"Multi class version of Logarithmic Loss metric.\n",
    "    https://www.kaggle.com/wiki/MultiClassLogLoss\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    y_true : array, shape = [n_samples]\n",
    "            true class, integers in [0, n_classes - 1)\n",
    "    y_pred : array, shape = [n_samples, n_classes]\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    loss : float\n",
    "    \"\"\"\n",
    "    predictions = np.clip(y_pred, eps, 1 - eps)\n",
    "\n",
    "    # normalize row sums to 1\n",
    "    predictions /= predictions.sum(axis=1)[:, np.newaxis]\n",
    "\n",
    "    actual = np.zeros(y_pred.shape)\n",
    "    n_samples = actual.shape[0]\n",
    "    actual[np.arange(n_samples), y_true.astype(int)] = 1\n",
    "    vectsum = np.sum(actual * np.log(predictions))\n",
    "    loss = -1.0 / n_samples * vectsum\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.2579113924050633\n",
      "Log Loss: 25.63083248293235\n"
     ]
    }
   ],
   "source": [
    "yhat = dt_clf.predict(X[test])\n",
    "class_probabilities = dt_clf.predict_proba(X[test])\n",
    "print ('Accuracy:', mt.accuracy_score(y[test],yhat))\n",
    "print ('Log Loss:', multiclass_log_loss(y[test], class_probabilities))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Once:\n",
      "Accuracy: 0.3024789029535865\n",
      "Log Loss: 24.091525201619085\n"
     ]
    }
   ],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "n_components = 50\n",
    "pca = PCA(n_components=n_components, svd_solver='randomized')\n",
    "\n",
    "for train, test in cv.split(X,y):\n",
    "    print ('Training Once:')\n",
    "    \n",
    "    # transform the data using pca\n",
    "    pca.fit(X[train])\n",
    "    X_train = pca.transform(X[train])\n",
    "    X_test = pca.transform(X[test])\n",
    "    \n",
    "    # train the decision tree algorithm\n",
    "    dt_clf.fit(X_train,y[train])\n",
    "    yhat = dt_clf.predict(X_test)\n",
    "    class_probabilities = dt_clf.predict_proba(X_test)\n",
    "    \n",
    "    print ('Accuracy:', mt.accuracy_score(y[test],yhat))\n",
    "    print ('Log Loss:', multiclass_log_loss(y[test], class_probabilities))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_losses = []\n",
    "accuracies = []\n",
    "params = []\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Once:\n",
      "Accuracy: 0.22303533755274263\n",
      "Log Loss: 3.5747380928751866\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAEWCAYAAACkFdnuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHZ9JREFUeJzt3X+cXXV95/HXm/CzirKSUREioZquRVZBU5S1rmyl20C74KO1blgVsdjstlBqxbZou7Slj0frj1ZXC5ZFZaWtFSn66EYNi6LQaivIgAENiEZEE6Ey8lMKgsHP/nFP2Oswk7mTfOfOzJ3X8/G4j9xzznfO+ZwJ+fC+554fqSokSZK063ab7wIkSZJGhcFKkiSpEYOVJElSIwYrSZKkRgxWkiRJjRisJEmSGjFYSZI0JEnuT/Lj812H5o7BaglKcmuSY+Z4G3+Y5G/mchuSlrYkVya5O8le813LoKrq8VV1y3zXobljsJIkLTpJVgIvBgo4fojb3X1Y29LiZLDSj0jyq0k2J7kryfokT+tb9p+S3Jzk3iTvSfIPSV63E9v4ye6T5j1JNiU5vm/ZcUluTPK9JN9O8sZu/vIkH+9+5q4kn02yW7fsaUk+kmQiyTeSnN63viOTjCe5L8l3krxj135DkhaIk4CrgA8Ar9k+M8k+Sf48yTe7XvW5JPt0y346yT93fWRLkpO7+Vf297IkJyf5XN90JTk1ydeAr3Xz3tWt474k1yZ5cd/4ZUnenOTrXS+7NsmKvnU9s3u/V5I/S/Ktrj+d11frtD1PC5t/SXpUkp8B/hR4BXAA8E3gom7ZcuAS4E3A/sDNwL/fiW3sAXwM+CTwZOA3gA8m+bfdkPcD/62q9gUOAz7TzT8D2AqMAU8B3gxU12g+BlwPHAi8FHh9kp/rfu5dwLuq6gnAM4CLZ1uzpAXpJOCD3evnkjylm/9nwPPp9acnAb8D/DDJ04FLgb+g10cOBzbOYnsvA14AHNpNX9Ot40nA3wJ/l2TvbtkbgBOB44AnAL8CPDDFOt8K/ES3nmfS62Fndcum7HmzqFfzxGClfq8ELqiq66rqIXoh6qjukPtxwKaq+mhVbQPeDfzLTmzjhcDjgbdU1cNV9Rng4/SaEMAPgEOTPKGq7q6q6/rmHwAcXFU/qKrPVu9Blz8FjFXV2d36bgHeC6zt+7lnJlleVfdX1VU7UbOkBSTJTwMHAxdX1bXA14H/2n3Q+hXgN6vq21X1SFX9c9fPXglcXlUf6nrInVU1m2D1p1V1V1U9CFBVf9OtY1tV/TmwF7D9A+LrgN+vqpur5/qqunPSPgT4VeC3uvV+D/gTfrR3TdXztMAZrNTvafSOUgFQVfcDd9L7FPU0YEvfsqL3aWpntrGlqn7YN++b3TYAfoleiPtm91XjUd38twObgU8muSXJmd38g4GndYfL70lyD71Pdts/vZ5C7xPhV5Jck+QXdqJmSQvLa4BPVtV3u+m/7eYtB/amF7QmWzHN/EFt6Z9IckaSm7qvG+8Bnthtf9BtjQE/Blzb17v+bzcfpu95WuA8CU/9bqMXVABI8jh6X/t9G7gdOKhvWfqnZ7mNFUl26wtXTwe+ClBV1wAndF8Znkbvq7sV3ae5M4AzkjwbuCLJNfSa3TeqatVUG6uqrwEndp9kfxG4JMn+VfWvO1G7pHnWnYP0CmBZku1HzfcC9qN3hOf79L72v37Sj24Bjpxmtf9KL+Rs99Qpxjx6tKg7n+p36Z16sKmqfpjkbiB923oG8OUd7Mp3gQeBZ1fVtx+zsWl6XlV9egfr1ALgEaula48ke/e9dqf3qe+1SQ5P7/LlPwGurqpbgU8A/y7Jy7qxpzJ18+m326Rt7AVcTa+J/U6SPZIcDfxn4KIkeyZ5ZZInVtUPgPuARwCS/EKSZ3aBbvv8R4AvAPcl+d3upNVlSQ5L8lPdz70qyVgX4u7p6nqkyW9Q0nx4Gb1/w4fSOzfpcOAngc/SO+/qAuAd6V3UsizJUV3v+SBwTJJXJNk9yf5JDu/WuRH4xSQ/1p1YfsoMNewLbAMmgN2TnEXvXKrt3gf8cZJV6XlOkv37V9D1pPcC70zyZIAkB24/P3QHPU8LnMFq6dpA79PS9tcfdp+E/gfwEXpHqJ5B931/d8j9l4G30ft68FBgHHhoB9s4cdI2vl5VD9O7NPpYep/Y3gOcVFVf6X7m1cCtSe4D/jvwqm7+KuBy4H7g88B7qurKqnqEXjA7HPhGt8730TssD7AG2JTkfnonsq+tqu/P9pclacF4DfC/q+pbVfUv21/AOfTOozoT+BK9k8vvoneC+G5V9S16pxmc0c3fCDy3W+c7gYeB7wAX0gthO3IZvRPhv0rvVIbv86NfFb6D3tH2T9ILRe8H9pliPb9L7+u+q7qedzn//zytKXveDHVpAYjnwmlndF+tbQVeWVVXzHc9kiQtBB6x0sCS/FyS/brD6m+mdz6BV9lJktQxWGk2jqJ3pct36X399rLtlx5LC1WSC5LckWTKE4m7c2Dend6NcW9I8rxh1yhpdPhVoKSRluQ/0DtP5a+q6rAplh9H70a1x9G7AeS7quoFw61S0qjwiJWkkVZV/0jvZOXpnEAvdFV3A9n9khwwnOokjZp5u4/V8uXLa+XKlfO1eUnz4Nprr/1uVY3NPHKoDuRHr+ja2s27ffLAJOuAdQCPe9zjnv+sZz1rKAVKmn+D9q95C1YrV65kfHx8vjYvaR4k+ebMo4YuU8yb8hyJqjofOB9g9erVZQ+Tlo5B+5dfBUpa6rbSewTJdgfRe0KAJM2awUrSUrceOKm7OvCFwL1V9ZivASVpED4rUNJIS/Ih4GhgeZKtwB8AewBU1Xn0nkJwHL07YD8AvHZ+KpU0CgxWkkZaVZ04w/Ki9+xLSdplfhUoSZLUiMFKkiSpEYOVJElSIwYrSZKkRgxWkiRJjSyaqwJXnvmJ+S5B0iS3vuXn57sESVpQZjxilWTvJF9Icn2STUn+aIoxJyeZSLKxe71ubsqVJElauAY5YvUQ8DNVdX+SPYDPJbm0ewp8vw9X1WntS5QkSVocZgxW3c3z7u8m9+heUz6gVJIkaSkb6OT1JMuSbATuAD5VVVdPMeyXktyQ5JIkK6ZYTpJ1ScaTjE9MTOxC2ZIkSQvPQMGqqh6pqsPpPfX9yCSHTRryMWBlVT0HuBy4cJr1nF9Vq6tq9djY2K7ULUmStODM6nYLVXUPcCWwZtL8O6vqoW7yvcDzm1QnSZK0iAxyVeBYkv269/sAxwBfmTTmgL7J44GbWhYpSZK0GAxyVeABwIVJltELYhdX1ceTnA2MV9V64PQkxwPbgLuAk+eqYEmSpIVqkKsCbwCOmGL+WX3v3wS8qW1pkiRJi4uPtJEkSWrEYCVJktSIwUqSJKkRg5UkSVIjBitJkqRGDFaSJEmNGKwkSZIaMVhJkiQ1YrCSJElqxGAlSZLUiMFKkiSpEYOVJElSIwYrSZKkRgxWkiRJjRisJEmSGjFYSZIkNWKwkiRJasRgJUmS1IjBSpIkqRGDlSRJUiMzBqskeyf5QpLrk2xK8kdTjNkryYeTbE5ydZKVc1GsJM1WkjVJbu7605lTLH96kiuSfDHJDUmOm486JY2GQY5YPQT8TFU9FzgcWJPkhZPGnALcXVXPBN4JvLVtmZI0e0mWAecCxwKHAicmOXTSsN8HLq6qI4C1wHuGW6WkUTJjsKqe+7vJPbpXTRp2AnBh9/4S4KVJ0qxKSdo5RwKbq+qWqnoYuIhev+pXwBO6908EbhtifZJGzEDnWCVZlmQjcAfwqaq6etKQA4EtAFW1DbgX2H+K9axLMp5kfGJiYtcql6SZPdqbOlu7ef3+EHhVkq3ABuA3pluZPUzSTAYKVlX1SFUdDhwEHJnksElDpjo6NfmoFlV1flWtrqrVY2Njs69WkmZnkN50IvCBqjoIOA746yRT9kZ7mKSZzOqqwKq6B7gSWDNp0VZgBUCS3ekdTr+rQX2StCse7U2dg3jsV32nABcDVNXngb2B5UOpTtLIGeSqwLEk+3Xv9wGOAb4yadh64DXd+5cDn6mqxxyxkqQhuwZYleSQJHvSOzl9/aQx3wJeCpDkJ+kFK7/nk7RTdh9gzAHAhd3VNbvRu3rm40nOBsaraj3wfnqHzzfTO1K1ds4qlqQBVdW2JKcBlwHLgAuqatOk/nUG8N4kv0Xva8KT/WAoaWfNGKyq6gbgiCnmn9X3/vvAL7ctTZJ2XVVtoHdSev+8/v51I/CiYdclaTR553VJkqRGDFaSJEmNGKwkSZIaMVhJkiQ1YrCSJElqxGAlSZLUiMFKkiSpEYOVJElSIwYrSZKkRgxWkiRJjRisJEmSGjFYSZIkNWKwkiRJasRgJUmS1IjBSpIkqRGDlSRJUiMGK0mSpEYMVpIkSY0YrCRJkhoxWEmSJDUyY7BKsiLJFUluSrIpyW9OMeboJPcm2di9zpqbciVJkhau3QcYsw04o6quS7IvcG2ST1XVjZPGfbaqfqF9iZIkSYvDjEesqur2qrque/894CbgwLkuTJIkabGZ1TlWSVYCRwBXT7H4qCTXJ7k0ybOn+fl1ScaTjE9MTMy6WEmSpIVs4GCV5PHAR4DXV9V9kxZfBxxcVc8F/gL4+6nWUVXnV9Xqqlo9Nja2szVLkiQtSAMFqyR70AtVH6yqj05eXlX3VdX93fsNwB5JljetVJIkaYEb5KrAAO8Hbqqqd0wz5qndOJIc2a33zpaFSpIkLXSDXBX4IuDVwJeSbOzmvRl4OkBVnQe8HPi1JNuAB4G1VVVzUK8kSdKCNWOwqqrPAZlhzDnAOa2KkiRJWoy887okSVIjBitJkqRGDFaSJEmNGKwkSZIaMVhJkiQ1YrCSNNKSrElyc5LNSc6cZswrktyYZFOSvx12jZJGxyD3sZKkRSnJMuBc4GeBrcA1SdZX1Y19Y1YBbwJeVFV3J3ny/FQraRR4xErSKDsS2FxVt1TVw8BFwAmTxvwqcG5V3Q1QVXcMuUZJI8RgJWmUHQhs6Zve2s3r9xPATyT5pyRXJVkz3cqSrEsynmR8YmJiDsqVtNgZrCSNsqmeGjH5cVu7A6uAo4ETgfcl2W+qlVXV+VW1uqpWj42NNS1U0mgwWEkaZVuBFX3TBwG3TTHm/1TVD6rqG8DN9IKWJM2awUrSKLsGWJXkkCR7AmuB9ZPG/D3wHwGSLKf31eAtQ61S0sgwWEkaWVW1DTgNuAy4Cbi4qjYlOTvJ8d2wy4A7k9wIXAH8dlXdOT8VS1rsvN2CpJFWVRuADZPmndX3voA3dC9J2iUesZIkSWrEYCVJktSIwUqSJKkRg5UkSVIjBitJkqRGDFaSJEmNzBiskqxIckWSm5JsSvKbU4xJkncn2ZzkhiTPm5tyJUmSFq5B7mO1DTijqq5Lsi9wbZJPVdWNfWOOpfcIiFXAC4C/7P6UJElaMmY8YlVVt1fVdd3779G7e/Hkp8OfAPxV9VwF7JfkgObVSpIkLWCzOscqyUrgCODqSYsOBLb0TW/lseGLJOuSjCcZn5iYmF2lkiRJC9zAwSrJ44GPAK+vqvsmL57iR+oxM6rOr6rVVbV6bGxsdpVKkiQtcAMFqyR70AtVH6yqj04xZCuwom/6IOC2XS9PkiRp8RjkqsAA7wduqqp3TDNsPXBSd3XgC4F7q+r2hnVKkiQteINcFfgi4NXAl5Js7Oa9GXg6QFWdR+/J8ccBm4EHgNe2L1WSJGlhmzFYVdXnmPocqv4xBZzaqihJkqTFyDuvS5IkNWKwkiRJasRgJUmS1IjBSpIkqRGDlSRJUiMGK0mSpEYMVpIkSY0YrCRJkhoxWEmSJDVisJIkSWrEYCVJktSIwUqSJKkRg5UkSVIjBitJkqRGDFaSJEmNGKwkSZIaMVhJkiQ1YrCSJElqxGAlSZLUiMFKkiSpkRmDVZILktyR5MvTLD86yb1JNnavs9qXKUk7J8maJDcn2ZzkzB2Me3mSSrJ6mPVJGi2DHLH6ALBmhjGfrarDu9fZu16WJO26JMuAc4FjgUOBE5McOsW4fYHTgauHW6GkUTNjsKqqfwTuGkItktTakcDmqrqlqh4GLgJOmGLcHwNvA74/zOIkjZ5W51gdleT6JJcmefZ0g5KsSzKeZHxiYqLRpiVpWgcCW/qmt3bzHpXkCGBFVX18ppXZwyTNpEWwug44uKqeC/wF8PfTDayq86tqdVWtHhsba7BpSdqhTDGvHl2Y7Aa8EzhjkJXZwyTNZJeDVVXdV1X3d+83AHskWb7LlUnSrtsKrOibPgi4rW96X+Aw4MoktwIvBNZ7AruknbXLwSrJU5Oke39kt847d3W9ktTANcCqJIck2RNYC6zfvrCq7q2q5VW1sqpWAlcBx1fV+PyUK2mx232mAUk+BBwNLE+yFfgDYA+AqjoPeDnwa0m2AQ8Ca6uqplmdJA1NVW1LchpwGbAMuKCqNiU5GxivqvU7XoMkzc6MwaqqTpxh+TnAOc0qkqSGulMUNkyaN+X99qrq6GHUJGl0eed1SZKkRgxWkiRJjRisJEmSGjFYSZIkNWKwkiRJasRgJUmS1IjBSpIkqRGDlSRJUiMGK0mSpEYMVpIkSY0YrCRJkhoxWEmSJDVisJIkSWrEYCVJktSIwUqSJKkRg5UkSVIjBitJkqRGDFaSJEmNGKwkSZIaMVhJkiQ1MmOwSnJBkjuSfHma5Uny7iSbk9yQ5Hnty5QkSVr4Bjli9QFgzQ6WHwus6l7rgL/c9bIkSZIWnxmDVVX9I3DXDoacAPxV9VwF7JfkgFYFSpIkLRYtzrE6ENjSN721m/cYSdYlGU8yPjEx0WDTkiRJC0eLYJUp5tVUA6vq/KpaXVWrx8bGGmxakiRp4WgRrLYCK/qmDwJua7BeSZKkRaVFsFoPnNRdHfhC4N6qur3BeiVJkhaV3WcakORDwNHA8iRbgT8A9gCoqvOADcBxwGbgAeC1c1WsJEnSQjZjsKqqE2dYXsCpzSqSJElapLzzuiRJUiMGK0mSpEYMVpIkSY3MeI6VJGnXrDzzE/NdgqQp3PqWn2++To9YSZIkNWKwkjTSkqxJcnOSzUnOnGL5G5LcmOSGJJ9OcvB81ClpNBisJI2sJMuAc4FjgUOBE5McOmnYF4HVVfUc4BLgbcOtUtIoMVhJGmVHApur6paqehi4CDihf0BVXVFVD3STV9F7LJck7RSDlaRRdiCwpW96azdvOqcAl063MMm6JONJxicmJhqVKGmUGKwkjbJMMa+mHJi8ClgNvH26lVXV+VW1uqpWj42NNSpR0ijxdguSRtlWYEXf9EHAbZMHJTkG+D3gJVX10JBqkzSCPGIlaZRdA6xKckiSPYG1wPr+AUmOAP4XcHxV3TEPNUoaIQYrSSOrqrYBpwGXATcBF1fVpiRnJzm+G/Z24PHA3yXZmGT9NKuTpBn5VaCkkVZVG4ANk+ad1ff+mKEXJWlkecRKkiSpEYOVJElSIwYrSZKkRgxWkiRJjRisJEmSGjFYSZIkNTJQsEqyJsnNSTYnOXOK5ScnmejuAbMxyevalypJkrSwzXgfqyTLgHOBn6X3eIhrkqyvqhsnDf1wVZ02BzVKkiQtCoMcsToS2FxVt1TVw8BFwAlzW5YkSdLiM0iwOhDY0je9tZs32S8luSHJJUlWTLGcJOuSjCcZn5iY2IlyJUmSFq5BglWmmFeTpj8GrKyq5wCXAxdOtaKqOr+qVlfV6rGxsdlVKkmStMANEqy2Av1HoA4CbusfUFV3VtVD3eR7gee3KU+SJGnxGCRYXQOsSnJIkj2BtcCPPP09yQF9k8fTe4q8JEnSkjLjVYFVtS3JacBlwDLggqralORsYLyq1gOnJzke2AbcBZw8hzVLkiQtSDMGK4Cq2gBsmDTvrL73bwLe1LY0SZKkxcU7r0uSJDVisJIkSWrEYCVJktSIwUqSJKkRg5UkSVIjBitJkqRGDFaSJEmNGKwkSZIaMVhJkiQ1YrCSJElqxGAlSZLUiMFKkiSpEYOVJElSIwYrSZKkRgxWkiRJjRisJEmSGjFYSZIkNWKwkiRJasRgJUmS1IjBSpIkqZGBglWSNUluTrI5yZlTLN8ryYe75VcnWdm6UEnaGfYvScM0Y7BKsgw4FzgWOBQ4Mcmhk4adAtxdVc8E3gm8tXWhkjRb9i9JwzbIEasjgc1VdUtVPQxcBJwwacwJwIXd+0uAlyZJuzIlaafYvyQN1e4DjDkQ2NI3vRV4wXRjqmpbknuB/YHv9g9Ksg5Y103en+TmWdS6fPL6RtBS2EdwP0dG3jrrfTx4rmqZRrP+BbvUw0b+v4XOUtjPpbCPsET2c5Y9bKD+NUiwmuqTW+3EGKrqfOD8Abb52CKS8apavTM/u1gshX0E93OULIJ9bNa/YOd72CL4PTWxFPZzKewjuJ+7YpCvArcCK/qmDwJum25Mkt2BJwJ3tShQknaB/UvSUA0SrK4BViU5JMmewFpg/aQx64HXdO9fDnymqqb8xCdJQ2T/kjRUM34V2J1zcBpwGbAMuKCqNiU5GxivqvXA+4G/TrKZ3ie9tXNQ6059hbjILIV9BPdzlCzofbR/Dd1S2M+lsI/gfu60+MFMkiSpDe+8LkmS1IjBSpIkqZEFG6ySPCnJp5J8rfvz3+xg7BOSfDvJOcOscVcNso9JDk/y+SSbktyQ5L/MR607Yyk8SmSAfXxDkhu7v7tPJxn2fZyamGk/+8a9PEklGfnLtHdkKfQvGO0ethT6FyyNHjbs/rVggxVwJvDpqloFfLqbns4fA/8wlKraGmQfHwBOqqpnA2uA/5lkvyHWuFOWwqNEBtzHLwKrq+o59O7q/bbhVrnrBtxPkuwLnA5cPdwKF6Sl0L9gRHvYUuhfsDR62Hz0r4UcrPofM3Eh8LKpBiV5PvAU4JNDqqulGfexqr5aVV/r3t8G3AGMDa3CnbcUHiUy4z5W1RVV9UA3eRW9+ygtNoP8XUIvILwN+P4wi1uglkL/gtHtYUuhf8HS6GFD718LOVg9papuB+j+fPLkAUl2A/4c+O0h19bKjPvYL8mRwJ7A14dQ266a6lEiB043pqq2AdsfJbJYDLKP/U4BLp3TiubGjPuZ5AhgRVV9fJiFLWBLoX/B6PawpdC/YGn0sKH3r0EeaTNnklwOPHWKRb834Cp+HdhQVVsW6geFBvu4fT0HAH8NvKaqftiitjnW9FEiC9TA9Sd5FbAaeMmcVjQ3drifXUB4J3DysApaCJZC/4Il28OWQv+CpdHDht6/5jVYVdUx0y1L8p0kB1TV7d0/yDumGHYU8OIkvw48Htgzyf1VtaPzGYaqwT6S5AnAJ4Dfr6qr5qjU1mbzKJGtWZyPEhlkH0lyDL3/Cb2kqh4aUm0tzbSf+wKHAVd2AeGpwPokx1fV+NCqHLKl0L9gyfawpdC/YGn0sOH3r6pakC/g7cCZ3fszgbfNMP5k4Jz5rrv1PtI7bP5p4PXzXe8s92134BbgkG4frgeePWnMqcB53fu1wMXzXfcc7OMR9L72WDXf9c7lfk4afyW9k13nvfZ5/J2NfP8adD8XYw9bCv1rFvu5qHvYfPSved/pHezc/t0/xq91fz6pm78aeN8U4xddYxpkH4FXAT8ANva9Dp/v2gfcv+OAr3b/KH+vm3c2cHz3fm/g74DNwBeAH5/vmudgHy8HvtP3d7d+vmuei/2cNNZgtQT616D7uVh72FLoXwPu56LvYcPuXz7SRpIkqZGFfFWgJEnSomKwkiRJasRgJUmS1IjBSpIkqRGDlSRJUiPzeoNQjY4kjwBfovff1DeAV1fVPfNblSTNzP6lljxipVYerKrDq+owencfPnW+C5KkAdm/1IzBSnPh83QPuUxydJJHH2yZ5JwkJ3fvb03yR0muS/KlJM/q5r8kycbu9cUk+87HTkhakuxf2iUGKzWVZBnwUmD9gD/y3ap6HvCXwBu7eW8ETq2qw4EXAw82L1SSJrF/qQWDlVrZJ8lG4E7gScCnBvy5j3Z/Xgus7N7/E/COJKcD+1XVtpaFStIk9i81Y7BSKw92n9AOpvegy+3nKGzjR/8723vSz21/UvojdBdTVNVbgNcB+wBXbT/ELklzxP6lZgxWaqqq7gVOB96YZA/gm8ChSfZK8kR6h9l3KMkzqupLVfVWYBywMUmac/YvtWCwUnNV9UXgemBtVW0BLgZuAD4IfHGAVbw+yZeTXE/v/IRL56xYSepj/9KuSlXNdw2SJEkjwSNWkiRJjRisJEmSGjFYSZIkNWKwkiRJasRgJUmS1IjBSpIkqRGDlSRJUiP/Dx6rEzNK6mNkAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# how to make it generalize a bit better? \n",
    "\n",
    "# defaults\n",
    "dt_clf = DecisionTreeClassifier(criterion='gini', max_depth=None, \n",
    "                                min_samples_split=2, min_samples_leaf=1, \n",
    "                                max_features=None,\n",
    "                                max_leaf_nodes=None)\n",
    "\n",
    "# overwrite with prepruning methods\n",
    "dt_clf = DecisionTreeClassifier(criterion='entropy', max_depth=5, \n",
    "                                min_samples_split=1000, min_samples_leaf=1, \n",
    "                                max_leaf_nodes=None)\n",
    "\n",
    "for train, test in cv.split(X,y):\n",
    "    print ('Training Once:')\n",
    "    \n",
    "    # transform the data using pca\n",
    "    pca.fit(X[train])\n",
    "    X_train = pca.transform(X[train])\n",
    "    X_test = pca.transform(X[test])\n",
    "    \n",
    "    # train the decision tree algorithm\n",
    "    dt_clf.fit(X_train,y[train])\n",
    "    yhat = dt_clf.predict(X_test)\n",
    "    class_probabilities = dt_clf.predict_proba(X_test)\n",
    "    \n",
    "    # get accuracy and log loss of this training\n",
    "    acc = mt.accuracy_score(y[test],yhat)\n",
    "    ll = multiclass_log_loss(y[test], class_probabilities)\n",
    "    print ('Accuracy:', acc)\n",
    "    print ('Log Loss:', ll)\n",
    "    \n",
    "    # save accuracy, log loss, and params of run\n",
    "    log_losses.append(ll)\n",
    "    accuracies.append(acc)\n",
    "    params.append(dt_clf.get_params())\n",
    "    \n",
    "plt.figure(figsize=(10,4))\n",
    "plt.subplot(1,2,1)\n",
    "plt.bar(range(len(log_losses)),log_losses)\n",
    "plt.title('Log Losses')\n",
    "plt.xlabel('Runs')\n",
    "\n",
    "plt.subplot(1,2,2)\n",
    "plt.bar(range(len(accuracies)),accuracies)\n",
    "plt.title('Accuracies')\n",
    "plt.xlabel('Runs')\n",
    "plt.ylim([0,1])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log Loss of \"Chance\": 4.795790545596759\n"
     ]
    }
   ],
   "source": [
    "# a telling example for log loss minimization\n",
    "class_probabilities = np.ones((len(y[test]),len(np.unique(y))))\n",
    "print ('Log Loss of \"Chance\":', multiclass_log_loss(y[test], class_probabilities))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Importances\n",
    "The feature importances can be calculated using the Mean decrease in the gini index. Essentially, every time a feature is tested for splitting, the decrease in the gini (or entropy) is saved. The importance of the feature can then be approximated by taking the mean decrease in the gini for all the nodes where the feature is tested (usually the feature is tested at every node):\n",
    "$$ Imp_{A} = \\frac{1}{|T|}\\sum_{t \\in T} gini_A(p_t) - gini_A(t) $$\n",
    "where $|T|$ is the total number of nodes in the tree, $T$ is the set of all nodes, $gini_A(t)$ is the gini for the $A^{th}$ attribute at node $t$, and $gini_A(p_t)$ is the gini for the $A^{th}$ attribute at the **parent** node for $t$.\n",
    "\n",
    "In this way, we have the mean decrease in the gini for attribute $A$. We can make a vector of these importance for all attributes in the dataset: $[Imp_1, Imp_2, ... Imp_{A_{tot}}]$.\n",
    "\n",
    "`scikit-learn` computes this value for you and it is stored inside the tree object as `dt_clf.feature_importances_`. For the current dataset, the images are feature vectors of size 25x25=625 pixels. Therefore we can look at the importances as a bar graph with 625 instances or we can reshape the instances to see the importances as a 25x25 image. Both are done below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Once:\n",
      "accuracy: 0.26173523206751054\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFXhJREFUeJzt3X+MZeV93/H3pztmSd0AyTKOCIuya7FxA7gm9naD5SRqTLCXxPG6CsiLkOEPqk1TqFKlVbWoArXIkcI/JbWMrJCAQ2jdxSWhHplNNrGx/0hkrXco2LDQrQdMxHjdsghCHVdA1vn2j/usc5ncmTnz8947835JV/ec5zznuc9z997zmfPjnk1VIUnS3xt2ByRJo8FAkCQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkZmLYHViK888/v3bs2DHsbkjSWHnsscdeqqrJxeqNVSDs2LGD6enpYXdDksZKkr/oUs9DRpIkwECQJDUGgiQJMBAkSU2nQEiyN8mJJDNJDg5YvjXJg2350SQ7WvmeJE+0x9eS/NOubUqS1teigZBkC3A3cDVwCXBdkkvmVLsJeKWqLgbuAu5s5U8Bu6vqcmAv8NtJJjq2KUlaR132EPYAM1X1XFW9ARwC9s2psw+4v00/BFyZJFX1/6rqdCs/Gzjz37N1aVOStI66BMKFwAt987OtbGCdFgCvAtsAkvxUkuPAk8A/b8u7tClJWkddAiEDyub+R8zz1qmqo1V1KfCPgVuTnN2xzV7DyYEk00mmT5061aG7kqTl6BIIs8BFffPbgZPz1UkyAZwLvNxfoaqeAb4LXNaxzTPr3VNVu6tq9+Tkor+8llbFjoOPDLsL0rrrEgjHgF1JdiY5C9gPTM2pMwXc2KavAR6tqmrrTAAk+THgHcDzHduUJK2jRe9lVFWnk9wCHAG2APdV1fEkdwDTVTUF3As8kGSG3p7B/rb6TwMHk/w18DfAv6iqlwAGtbnKY5MkLUGnm9tV1WHg8Jyy2/umXwOuHbDeA8ADXduUJA2Pv1SWJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKkxECRJgIEgSWoMBEkSYCBIkhoDQZIEGAiSpMZAkCQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSp6RQISfYmOZFkJsnBAcu3JnmwLT+aZEcrvyrJY0mebM/v71vny63NJ9rjbas1KEnS0k0sViHJFuBu4CpgFjiWZKqqnu6rdhPwSlVdnGQ/cCfwUeAl4Jeq6mSSy4AjwIV9611fVdOrNBZJ0gp02UPYA8xU1XNV9QZwCNg3p84+4P42/RBwZZJU1eNVdbKVHwfOTrJ1NTouSVpdXQLhQuCFvvlZ3vxX/pvqVNVp4FVg25w6vww8XlWv95V9uh0uui1JltRzSdKq6hIIgzbUtZQ6SS6ldxjpV/qWX19V7wR+pj0+NvDFkwNJppNMnzp1qkN3JUnL0SUQZoGL+ua3Ayfnq5NkAjgXeLnNbwceBm6oqmfPrFBV32rP3wE+Q+/Q1N9RVfdU1e6q2j05OdllTJKkZegSCMeAXUl2JjkL2A9MzakzBdzYpq8BHq2qSnIe8Ahwa1X9+ZnKSSaSnN+m3wJ8CHhqZUORJK3EooHQzgncQu8KoWeAz1bV8SR3JPlwq3YvsC3JDPDrwJlLU28BLgZum3N56VbgSJKvA08A3wJ+ZzUHJklamkUvOwWoqsPA4Tllt/dNvwZcO2C9jwMfn6fZ93TvpiRprflLZUkSYCBIkhoDQZIEGAiSpMZAkCQBBoIkqTEQJEmAgSBJagwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKkxECRJgIEgSWoMBEkSYCBIkppOgZBkb5ITSWaSHBywfGuSB9vyo0l2tPKrkjyW5Mn2/P6+dd7TymeSfCJJVmtQkqSlWzQQkmwB7gauBi4BrktyyZxqNwGvVNXFwF3Ana38JeCXquqdwI3AA33rfAo4AOxqj70rGIckaYW67CHsAWaq6rmqegM4BOybU2cfcH+bfgi4Mkmq6vGqOtnKjwNnt72JC4BzquorVVXA7wMfWfFoJEnL1iUQLgRe6JufbWUD61TVaeBVYNucOr8MPF5Vr7f6s4u0CUCSA0mmk0yfOnWqQ3clScvRJRAGHduvpdRJcim9w0i/soQ2e4VV91TV7qraPTk52aG7kqTl6BIIs8BFffPbgZPz1UkyAZwLvNzmtwMPAzdU1bN99bcv0qYkaR11CYRjwK4kO5OcBewHpubUmaJ30hjgGuDRqqok5wGPALdW1Z+fqVxV3wa+k+SKdnXRDcDnVjgWSdIKLBoI7ZzALcAR4Bngs1V1PMkdST7cqt0LbEsyA/w6cObS1FuAi4HbkjzRHm9ry34V+F1gBngW+KPVGpQkaekmulSqqsPA4Tllt/dNvwZcO2C9jwMfn6fNaeCypXRWkrR2/KWyJAkwECRJjYEgSQIMBElSYyBIkgADQZLUGAiSJMBAkCQ1BoIkCTAQJEmNgSBJAgwESVJjIEiSAANBktQYCJIkwECQJDUGgiQJMBAkSY2BIEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAnoGAhJ9iY5kWQmycEBy7cmebAtP5pkRyvfluRLSf4qySfnrPPl1uYT7fG21RiQJGl5JharkGQLcDdwFTALHEsyVVVP91W7CXilqi5Osh+4E/go8BpwG3BZe8x1fVVNr3AMkqRV0GUPYQ8wU1XPVdUbwCFg35w6+4D72/RDwJVJUlXfrao/oxcMkqQR1iUQLgRe6JufbWUD61TVaeBVYFuHtj/dDhfdliQd6kuS1kiXQBi0oa5l1Jnr+qp6J/Az7fGxgS+eHEgynWT61KlTi3ZWkrQ8XQJhFriob347cHK+OkkmgHOBlxdqtKq+1Z6/A3yG3qGpQfXuqardVbV7cnKyQ3clScvRJRCOAbuS7ExyFrAfmJpTZwq4sU1fAzxaVfPuISSZSHJ+m34L8CHgqaV2XpK0eha9yqiqTie5BTgCbAHuq6rjSe4ApqtqCrgXeCDJDL09g/1n1k/yPHAOcFaSjwAfAP4CONLCYAvwBeB3VnVkkqQlWTQQAKrqMHB4TtntfdOvAdfOs+6OeZp9T7cuSpLWg79UliQBBoIkqTEQpBXacfCRYXdBWhUGgiQJMBAkSY2BIEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAkwECRJjYEgSQIMBElSYyBIWjJv6LcxGQiSJMBAkCQ1BoIkCTAQJEmNgSBJAgwESVJjIEiSAANBktQYCJIkwEBQB/4qVdocOgVCkr1JTiSZSXJwwPKtSR5sy48m2dHKtyX5UpK/SvLJOeu8J8mTbZ1PJMlqDEiStDyLBkKSLcDdwNXAJcB1SS6ZU+0m4JWquhi4C7izlb8G3Ab8mwFNfwo4AOxqj73LGYAkaXV02UPYA8xU1XNV9QZwCNg3p84+4P42/RBwZZJU1Xer6s/oBcP3JbkAOKeqvlJVBfw+8JGVDESStDJdAuFC4IW++dlWNrBOVZ0GXgW2LdLm7CJtSpLWUZdAGHRsv5ZRZ1n1kxxIMp1k+tSpUws0KUlaiS6BMAtc1De/HTg5X50kE8C5wMuLtLl9kTYBqKp7qmp3Ve2enJzs0F1J0nJ0CYRjwK4kO5OcBewHpubUmQJubNPXAI+2cwMDVdW3ge8kuaJdXXQD8Lkl916StGomFqtQVaeT3AIcAbYA91XV8SR3ANNVNQXcCzyQZIbensH+M+sneR44BzgryUeAD1TV08CvAr8H/ADwR+0hSRqSRQMBoKoOA4fnlN3eN/0acO086+6Yp3wauKxrRyVJa8tfKkuSAANBktQYCJIkwECQJDUGgiQJMBAkSY2BIEkCDARJUmMgSJIAA0HSOvC/YR0PBoIkCTAQJEmNgSCtAg+JaCMwECRJgIEgSWoMBEkSYCBIkhoDQdqAPMmt5TAQJEmAgSBJagwEaQEeetFmYiBIkgADQZLUGAiSJMBAkCQ1BoLG0iie7B3FPklL0SkQkuxNciLJTJKDA5ZvTfJgW340yY6+Zbe28hNJPthX/nySJ5M8kWR6NQYjSVq+icUqJNkC3A1cBcwCx5JMVdXTfdVuAl6pqouT7AfuBD6a5BJgP3Ap8KPAF5L8eFV9r633c1X10iqOR5K0TF32EPYAM1X1XFW9ARwC9s2psw+4v00/BFyZJK38UFW9XlXfBGZae5KkEdMlEC4EXuibn21lA+tU1WngVWDbIusW8CdJHktyYL4XT3IgyXSS6VOnTnXorqRh8lzK+OoSCBlQVh3rLLTu+6rq3cDVwM1JfnbQi1fVPVW1u6p2T05OduiuJGk5ugTCLHBR3/x24OR8dZJMAOcCLy+0blWdeX4ReBgPJUnSUHUJhGPAriQ7k5xF7yTx1Jw6U8CNbfoa4NGqqla+v12FtBPYBXw1yVuT/CBAkrcCHwCeWvlwpOHycInG2aJXGVXV6SS3AEeALcB9VXU8yR3AdFVNAfcCDySZobdnsL+tezzJZ4GngdPAzVX1vSQ/AjzcO+/MBPCZqvrjNRifJKmjRQMBoKoOA4fnlN3eN/0acO086/4G8Btzyp4D3rXUzkoajh0HH+H53/zFYXdDa8xfKktjxsNSWisGgjasjbLh3Cjj0OgzEKQ53ABrszIQtGY2w4Z1o4xxo4xDK2MgaKy44ZLWjoEg9VlK4Cw1nAyzteX7u3IGgkbKWm6QJS3MQJAkAQaCxsiZPQL3DBY26u/PqPdvMzMQtCSDvsx+wbXa/EwNh4GgDWXUNySj3j9tbgbCJrOSDdJabsw2+oZyuePzMNnCfF9Wl4GgDaHLhmGzbTzWeryb7f3cDAyETcgv8mBz35dh/nW+nq+51Et9/fxsXAbCBrNWX9a1+hGWG5fufK/m53uzOjZ1IPghGq75/iJfi7bX6nVWw7D6M4y/9kftvT9jVPu13jZ1IIyq9bglwrj9BT9fP1arf6MyzsWs9OT0WljNQ2uj/O8wyn1bLQbCiBvm8evN8AVYrnF5b9Zrr2sj2Ojj68JA0Jpa67/sN4q13NNZj/faq7w2BgNhiJby5V3JhnXYX0Qvf1x9m3HM8xmVwNsIDIQNYrU/sOv5I7TFXmstD3uM8hd9Lfo2quPdbOeCRpWBMMd6X3nR/1rj8qVYiz4P07j+AnsU3vuNtvc3Cu/pMBkII2o5G93FLrUc5od9VH9otVavP8pXAy3lddb6dy2D+tT1ooZh/zsvZtT7N4iBsESLfamWe0hiVO4iOuwNxUpeZy0v193Mx6lHtV+L6Xp+bbWDZ6l/EIzS+2sgNKv58/1R+gceNaPy3oxKPwYZ5b6Niq7fv/69jdXaSG/kf59OgZBkb5ITSWaSHBywfGuSB9vyo0l29C27tZWfSPLBrm2ulVHYlV/Pwzcb+cM76sb9vR/n8wNr9SO5lf5GZ9R/4zOxWIUkW4C7gauAWeBYkqmqerqv2k3AK1V1cZL9wJ3AR5NcAuwHLgV+FPhCkh9v6yzW5qpa6FDOSo5Z9pc9/5u/uGDdpbQ7X73+11hta3nIZTn1N4rNNO7ljHXUPkcruWhiocvDB313R+2zsWggAHuAmap6DiDJIWAf0L/x3gf8+zb9EPDJJGnlh6rqdeCbSWZae3Roc82s1VUyo/aPO2r90WjYbJ+LYe6NL/ek+Zlla/kH4CBdDhldCLzQNz/bygbWqarTwKvAtgXW7dKmVmCzfemljWi9rw5MVS1cIbkW+GBV/bM2/zFgT1X9y746x1ud2Tb/LL09gTuAr1TVf27l9wKH6QXRgm32tX0AONBm3wGcWOZYzwdeWua6o8IxDN+49x8cwyhY7/7/WFVNLlapyyGjWeCivvntwMl56swmmQDOBV5eZN3F2gSgqu4B7unQzwUlma6q3SttZ5gcw/CNe//BMYyCUe1/l0NGx4BdSXYmOYveSeKpOXWmgBvb9DXAo9Xb9ZgC9rerkHYCu4CvdmxTkrSOFt1DqKrTSW4BjgBbgPuq6niSO4DpqpoC7gUeaCeNX6a3gafV+yy9k8WngZur6nsAg9pc/eFJkrpa9BzCRpHkQDv8NLYcw/CNe//BMYyCUe3/pgkESdLCvHWFJAnYJIEwrNtkLFWS+5K8mOSpvrIfTvKnSb7Rnn+olSfJJ9qYvp7k3cPr+ff7elGSLyV5JsnxJL/WysdpDGcn+WqSr7Ux/IdWvrPdluUb7TYtZ7XyeW/bMkxJtiR5PMnn2/y49f/5JE8meSLJdCsbm88RQJLzkjyU5H+278R7R30MGz4Q8re33rgauAS4Lr1baoyi3wP2zik7CHyxqnYBX2zz0BvPrvY4AHxqnfq4kNPAv66qnwCuAG5u7/U4jeF14P1V9S7gcmBvkivo3Y7lrjaGV+jdrgX6btsC3NXqjYJfA57pmx+3/gP8XFVd3nd55jh9jgD+E/DHVfUPgXfR+/cY7TFU1YZ+AO8FjvTN3wrcOux+LdDfHcBTffMngAva9AXAiTb928B1g+qNygP4HL37VY3lGIC/D/wP4Kfo/YhoYu5nit6Vcu9t0xOtXobc7+30NjbvBz4PZJz63/ryPHD+nLKx+RwB5wDfnPtejvoYNvweAuN/m4wfqapvA7Tnt7XykR5XO/Twk8BRxmwM7XDLE8CLwJ8CzwJ/Wb3bssCb+znfbVuG6beAfwv8TZvfxnj1H6CAP0nyWHp3K4Dx+hy9HTgFfLoduvvdJG9lxMewGQIhA8o2wqVVIzuuJP8A+APgX1XV/12o6oCyoY+hqr5XVZfT+0t7D/ATg6q155EaQ5IPAS9W1WP9xQOqjmT/+7yvqt5N71DKzUl+doG6oziGCeDdwKeq6ieB7/K3h4cGGYkxbIZA6HLrjVH2f5JcANCeX2zlIzmuJG+hFwb/par+sBWP1RjOqKq/BL5M73zIeendlgXe3M/vjyFvvm3LsLwP+HCS54FD9A4b/Rbj038Aqupke34ReJheMI/T52gWmK2qo23+IXoBMdJj2AyBMO63yei/LciN9I7Lnym/oV2dcAXw6pld0WFJEnq/Wn+mqv5j36JxGsNkkvPa9A8AP0/vZOCX6N2WBf7uGAbdtmUoqurWqtpeVTvofdYfrarrGZP+AyR5a5IfPDMNfAB4ijH6HFXV/wZeSPKOVnQlvTs2jPYYhnniZb0ewC8A/4veseB/N+z+LNDP/wp8G/hren8x3ETveO4XgW+05x9udUPv6qlngSeB3SPQ/5+mt5v7deCJ9viFMRvDPwIeb2N4Cri9lb+d3n24ZoD/Bmxt5We3+Zm2/O3DHkPfWP4J8Plx63/r69fa4/iZ7+w4fY5avy4Hpttn6b8DPzTqY/CXypIkYHMcMpIkdWAgSJIAA0GS1BgIkiTAQJAkNQaCJAkwECRJjYEgSQLg/wOXNNbIh64ERAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAEXpJREFUeJzt3UuIXPeVx/Hf0VuWWtZr3MiKZ5QRYog2owyNEXgYnEWCko2chSBZDFoEOgsbEsjGZJNsAt4kmVmEgDIR1iKxCCQeaxE8MiLgAQ8hSjCxjGYsYRS91QqW1Gq9Wq0+WXT10JHVdY66bj3a5/sB0d11/7r3X7fqp6rqe3T+5u4CUM+Sfk8AQH8QfqAowg8URfiBogg/UBThB4oi/EBRhB8oivADRS3r5cHMrONywiVL4n+vpqenOz1MyiDNJWPp0qXhmGi+vaoIzZzbpuYS7cfMenKczLEy+3D31IQ7Cr+Z7ZH075KWSvoPd38l+juZJ2A7q1evDsfcuXOno2PMih6IJ554ItzHxMREx8dp4kkjSUNDQ+GY27dvt91+//79cB+Zxzi6T6tWrQr3MTk5GY7JePDgQdvtmfuzbFkcpbt374Zjovsd3efovsy14Lf9ZrZU0o8kfVHSTklfNbOdC90fgN7q5DP/s5JOu/uH7j4p6bCkvc1MC0C3dRL+rZLOzfn5fOs2AItAJ5/5H/Uh82Mf5MxsVNJoB8cB0AWdhP+8pGfm/PwpSRcfHuTuByQdkJr5bT+AZnTytv93knaY2afNbIWkr0g60sy0AHTbgl/53X3KzF6S9F+audR30N3fb2xmALrKetnGy8w8uh7axHyaKhCJxmSOk7nuumLFirbbp6amwn1kZO5zVOSzfPnycB+ZmoPoOJniqMz5z8wlGpOpbcgcJ1MvED3W0WPo7ukiH8p7gaIIP1AU4QeKIvxAUYQfKIrwA0URfqConjbzkOLrlNF18cy13SaaVkjxtdsm9iHF15Ez9ydTT7By5cqO95O5P5m6hMzjGMnMJXNeorlkekjcu3cvHJMRzSXanqlJ+P99pUcC+EQh/EBRhB8oivADRRF+oCjCDxRF+IGiCD9QVM+LfKLCjKiIIbNQRmYxh04XD5FyBSSZQpToPjfREETKFd9E+8kUNq1duzYcE92nTLFKUwuIRPc5s49M0VITTUGiZiqP0/iFV36gKMIPFEX4gaIIP1AU4QeKIvxAUYQfKIrwA0X1fMWeTotrMvPNFFysWrUqHBMVC2UKeDLFN73q5JNZbSc6v5nz1lSBTqSpjkFRUVKmsOnu3bvhmMxzNzov0ePMij0AQoQfKIrwA0URfqAowg8URfiBogg/UBThB4rqaScfM9OyZe0PGRU5NLHMU3Y/UYFOprCjie4tmQKeTCFKpkBnYmKi7fZMoUoTHYMyMscZHh4Ox1y7dq3t9qaKrJrqttSUjsJvZmck3ZT0QNKUu480MSkA3dfEK//n3P3PDewHQA/xmR8oqtPwu6SjZvZ7Mxt91AAzGzWz42Z2vJf/iQhAe52+7X/O3S+a2VOS3jKz/3X3t+cOcPcDkg5I0pIlS0g/MCA6euV394utr2OSXpf0bBOTAtB9Cw6/ma0xs6HZ7yV9QdKJpiYGoLs6eds/LOn11jXqZZJ+7u5vtvsL7h5eD+10RZ+szPXUqCZhaGgo3Me9e/c6nkvm+nzmWnSm5iD6vcyePXvCfRw9ejQcEz0PMo/z+vXrwzHRNXwpXgUqU8+RmW8TY5pY3WnWgsPv7h9K+seF/n0A/cWlPqAowg8URfiBogg/UBThB4oi/EBRhB8oauBW7ImKfDKrzmSKWTZv3hyOuXHjRtvtTRX5RCsDZRp1ZOaSOXe3b99uuz0qiMmKCmcyjS8yxU+Z+3z58uW22zOP4cqVK8MxmaxFRTrRXFixB0CI8ANFEX6gKMIPFEX4gaIIP1AU4QeKIvxAUT1dsScjKvLJdODJdLUZHx8Px0RdU+7cuRPuIyMqVsl0gMkUAmXGROc3UzTTROebpgprMo9R9Hxp6vw3sdpRk518eOUHiiL8QFGEHyiK8ANFEX6gKMIPFEX4gaIG7jp/dM21iYYImeNI8TXVzD4y17zXrl3bdnu0cpAU10dIufOybt26jueyZcuWcMzVq1fbbs9cW79582Y4JvN8iWoBMrUNmeOsWbMmHDMxMdHRcTL1BrN45QeKIvxAUYQfKIrwA0URfqAowg8URfiBogg/UFRPV+xZsmSJRyuxrF+/vu32jz76KDxOpvgmUwwRFWVkGkU8/fTT4ZhoP5ninMxKOpkGGVFxTaaBRqYoJtrPxYsXw31knruZIqvMfCNbt24Nx5w6dSocE53/6D5PTk5qenq6mRV7zOygmY2Z2Yk5t200s7fM7FTr64bMwQAMjszb/lcl7XnotpclHXP3HZKOtX4GsIiE4Xf3tyU9/F57r6RDre8PSXqh4XkB6LKF/sJv2N0vSVLr61PNTQlAL3T9f/WZ2aik0W4fB8DjWegr/xUz2yJJra9j8w109wPuPuLuI5n/dgqgNxYa/iOS9re+3y/pjWamA6BXMpf6XpP0P5L+wczOm9nXJL0i6fNmdkrS51s/A1hEel7kE3WCiQouMoU10Qo4Uq7IZ/Xq1R3vI1NAEnW+uXHjRriPpop87t+/33b7+fPnw32MjIyEY86cOdN2e6ZoJvM4nzt3LhwzOTnZdnumUCizklQma1E+osfwwYMHcvdminwAfDIRfqAowg8URfiBogg/UBThB4oi/EBRhB8oqudFPlHRS1Q409RSXNESWZljRV2JpFzxTRNLlGUKePbv3x+OefPNN9tu37Ah7tuSKQSKuiRdvnw53EfU9UmShoaGwjEnTpxouz1awkySxsfHwzGZ/WQ6VbUzNTVFkQ+A9gg/UBThB4oi/EBRhB8oivADRRF+oKieX+ePGjBEzSQyMk0eMtffo3OTOc6tW7fCMVHtQ7SKi5RbSSdT/9BEM5UmejVev349HJNZDSlTLxA9zplGHRmZ53Y0l6j2hOv8AEKEHyiK8ANFEX6gKMIPFEX4gaIIP1AU4QeK6voqvQ+LmnVERQ6ZgpdMw4/MmGj1lEzjhU2bNoVjohVhohVlpHh1oex+oqYUmQYmmUKgjRs3hmMiZ8+eDccMDw+HY6L7nDlvmWYqmdWbonxEBVSPU2DFKz9QFOEHiiL8QFGEHyiK8ANFEX6gKMIPFEX4gaJ6XuQTFSFExTeZbjSZQpRMsVDUweXJJ58M93H79u1wTFTkkymIyRTWZOa7devWttsvXLgQ7iNzbqOuNlGxi5RbjWdsbCwcEz3nMsU5TdznzFyi7lGZ4rVZ4YzN7KCZjZnZiTm3fdfMLpjZu60/X0ofEcBAyLztf1XSnkfc/kN339X68+tmpwWg28Lwu/vbkjpbPRDAwOnkF34vmdkfWx8L5l261cxGzey4mR3vZadgAO0tNPw/lrRd0i5JlyR9f76B7n7A3UfcfaSJls4AmrGg8Lv7FXd/4O7Tkn4i6dlmpwWg2xYUfjPbMufHL0s6Md9YAIMpvM5vZq9Jel7SZjM7L+k7kp43s12SXNIZSV/v4hwBdEHPl+uKuuNERQpr1qwJj5Mppli/fn04Jiq+yRyniQKRzO9KMkuHRedekq5du9Z2e2aZs0xh0+bNm9tuzxTnZO5zpvgmWhosc/53794djnnnnXfCMZ126pmcnNT09DTLdQGYH+EHiiL8QFGEHyiK8ANFEX6gKMIPFNXT6/xmFl7nj5p1ZJoVbNu2LRxz5cqVcEwTq6Nkrq1HTUMy+8g8jvv27QvHHD58uO32TN1CZoWbqEYic30+c14yjV2aWDEpU3Nw8+bNcEynzW4kyd25zg9gfoQfKIrwA0URfqAowg8URfiBogg/UBThB4rq6Yo9q1at0vbt29uO+eCDDzo+zrlz51JziUSFKJnVgzIFIqtXr267feXKleE+rl69Go45ffp0OCYqIsmspJNZGejWrVttt2capWQai2RWMmpi9aBMA5PM4xgVa0VFQFHB2Fy88gNFEX6gKMIPFEX4gaIIP1AU4QeKIvxAUYQfKKrnnXyiDi3R9kyhRKawJlOgE43JdJLJFIhERSZREZCUOy+ZLjDj4+MdHydToBMVozS1onMTqx1NTEyE+8g8nzJz6bTIanp6mk4+ANoj/EBRhB8oivADRRF+oCjCDxRF+IGien6dP7oeGl1HzlxDHhoaCsdkmi9E52bdunXhPjI1B9G13Uw9QWaFm2hlGknatGlT2+3Xr18P95GpbYieB5kmHJm6hcx5yYyJZK7zZ+abeX634+7NXec3s2fM7DdmdtLM3jezb7Ru32hmb5nZqdbXDR3NGkBPZf7Jm5L0LXf/jKTdkl40s52SXpZ0zN13SDrW+hnAIhGG390vufsfWt/flHRS0lZJeyUdag07JOmFbk0SQPMe68OOmW2T9FlJv5U07O6XpJl/ICQ91fTkAHRPunuvma2V9EtJ33T38ex/vDCzUUmjC5segG5JvfKb2XLNBP9n7v6r1s1XzGxLa/sWSWOP+rvufsDdR9x9pIkJA2hG5rf9Jumnkk66+w/mbDoiaX/r+/2S3mh+egC6JfO2/zlJ/yrpPTN7t3XbtyW9IukXZvY1SWcl7evOFAF0w8AV+WSKJSKZ+5Q5TlT8kSlmyfxuJFp5JlOck2n4kSkyiQqKomYfWdFxMivPZB7nzKo+0fnNNDCJViCSpOXLl3c8l+j5RDMPACHCDxRF+IGiCD9QFOEHiiL8QFGEHyiK8ANFDdyKPZmuNZGminyiQpMmCpKkeL6Z4pBMx6BMwUvUQSdT2JSZb1Rw1MTqNlJuvk0UHGW6AWXmG93v6Dk3NTWl6elpinwAzI/wA0URfqAowg8URfiBogg/UBThB4oi/EBRnVfUPKaoGCLanilmyRT5ZIo/IpkOL5kuPNFcmupqc+/evXBMdKzMcTKFWtFxst2hI00USDU1l8x5iQqBou2PU7THKz9QFOEHiiL8QFGEHyiK8ANFEX6gKMIPFNXT6/xmFl4zbeI6c69kruFn5huNyTSByDQWydRIRI9PpmlFRlTb0NRxeiVTi9FEvUBTNQcSr/xAWYQfKIrwA0URfqAowg8URfiBogg/UBThB4rq9Yo9VyX9ac5NmyX9uWcT6Nximu9imqu0uOY7yHP9O3f/m8zAnob/Ywc3O+7uI32bwGNaTPNdTHOVFtd8F9Nc2+FtP1AU4QeK6nf4D/T5+I9rMc13Mc1VWlzzXUxznVdfP/MD6J9+v/ID6JO+hd/M9pjZ/5nZaTN7uV/zyDCzM2b2npm9a2bH+z2fh5nZQTMbM7MTc27baGZvmdmp1tcN/ZzjXPPM97tmdqF1jt81sy/1c46zzOwZM/uNmZ00s/fN7But2wf2/Gb1JfxmtlTSjyR9UdJOSV81s539mMtj+Jy77xrQSzyvStrz0G0vSzrm7jskHWv9PChe1cfnK0k/bJ3jXe7+6x7PaT5Tkr7l7p+RtFvSi63n6iCf35R+vfI/K+m0u3/o7pOSDkva26e5LHru/rakjx66ea+kQ63vD0l6oaeTamOe+Q4kd7/k7n9ofX9T0klJWzXA5zerX+HfKuncnJ/Pt24bVC7pqJn93sxG+z2ZpGF3vyTNPIElPdXn+WS8ZGZ/bH0sGLi30Wa2TdJnJf1Wi/P8/pV+hf9RjcgG+bLDc+7+T5r5mPKimf1Lvyf0CfRjSdsl7ZJ0SdL3+zudv2ZmayX9UtI33X283/NpQr/Cf17SM3N+/pSki32aS8jdL7a+jkl6XTMfWwbdFTPbIkmtr2N9nk9b7n7F3R+4+7Skn2iAzrGZLddM8H/m7r9q3byozu+j9Cv8v5O0w8w+bWYrJH1F0pE+zaUtM1tjZkOz30v6gqQT7f/WQDgiaX/r+/2S3ujjXEKzQWr5sgbkHNtMu9yfSjrp7j+Ys2lRnd9H6VuRT+tSzr9JWirpoLt/ry8TCZjZ32vm1V6aaXX+80Gbq5m9Jul5zfxvsyuSviPpPyX9QtLfSjoraZ+7D8Qv2eaZ7/Oaecvvks5I+vrsZ+p+MrN/lvTfkt6TNNtv/Nua+dw/kOc3iwo/oCgq/ICiCD9QFOEHiiL8QFGEHyiK8ANFEX6gKMIPFPUXG+4wvQL+5hIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "dt_clf = DecisionTreeClassifier()\n",
    "\n",
    "# now get the training and testing\n",
    "for train, test in cv.split(X,y):\n",
    "    print ('Training Once:')\n",
    "    # train the decision tree algorithm\n",
    "    dt_clf.fit(X[train],y[train])\n",
    "    yhat = dt_clf.predict(X[test])\n",
    "    print ('accuracy:', mt.accuracy_score(y[test],yhat))\n",
    "    \n",
    "# get the importances\n",
    "imp = dt_clf.feature_importances_\n",
    "\n",
    "#plot the importances as bar chart\n",
    "plt.bar(range(len(imp)),imp)\n",
    "\n",
    "# reshape the importances to be size of the original images\n",
    "# and show it\n",
    "imp_as_image = np.reshape(imp,(25,25))\n",
    "plt.figure()\n",
    "plt.imshow((imp_as_image), cmap=plt.cm.gray)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Jake\\Anaconda3\\envs\\ML7331\\lib\\site-packages\\ipykernel_launcher.py:5: RuntimeWarning: divide by zero encountered in log\n",
      "  \"\"\"\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2oAAADlCAYAAAAiJQSbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xtw3eV95/HP15Jly/f7/RZsLrYZMNjAJpCGBJYh2eySZhICGyhpupDOkJlemFKSbLftbneXzWySadIOGVgcQ9KGkLaZOiTchkuIoXFiEy42NrYxFhaW77bkmyzJevYPHbeK0fd7pHN0dB5Z79eMx7I++v2eR0e/7znn8Tn6PpZSEgAAAAAgH8OqPQEAAAAAwG9ioQYAAAAAmWGhBgAAAACZYaEGAAAAAJlhoQYAAAAAmWGhBgAAAACZYaEGAAAAAJlhodYPzGyHmV2bwTyeN7P/Uu15AAAAAChPbbUngPKZmUmyas8DAAAAQP/gFbV+ZGafM7MXzewbZnbYzLab2QcKn99pZnvN7LZuX7/KzL5tZk+b2REz+5mZze+Wf8DMfmVmzYW/P9Ate97M/qeZvSjpuKTvSvqgpL8xs6Nm9jeFr/vrwtgtZrbezD7Y7Rx/YWaPmtnDhfE3mtmKbvlcM/snM9tnZgdOn7OQfd7MNpnZITN78vS8rcs3Ct9rs5m9ZmYXVuYWBwAAAM5OLNT63xWSXpM0WdLfS3pE0mWSFkm6RV0LqTHdvv6zkv6HpCmSXpH0d5JkZpMk/UTSNwvn+rqkn5jZ5G7H3irpDkljJX1O0s8lfTGlNCal9MXC1/xK0jJJkwrz+aGZjex2jv9UmOMESaslnV7g1Uh6TFKDpAWSZhe+Tmb2CUlflvRJSVML436/cL7rJP2WpPMK5/yMpAO9vfEAAAAAsFCrhLdTSt9JKZ2S9ANJcyX995TSyZTSU5La1LVoO+0nKaUXUkonJX1F0vvNbK6k/yBpa0rpuymljpTS9yVtlvQfux27KqW0sZC39zSZlNL3UkoHCl/zNUkjJJ3f7UvWpJR+WpjvdyVdXPj85ZJmSfqTlNKxlFJrSmlNIfuCpP+dUtqUUuqQ9L8kLSu8qtauroXjBZKs8DVNJdyOAAAAwJDFQq3/7en28QlJSimd+bnur6jtPP1BSumopIPqWiDNUterWd01qOuVrfcc6zGzuwpvUWw2s8OSxqvr1bvTdnf7+LikkWZWq64FZkNhIXam+ZL+uvD2zsOFOZuk2SmlZ9X1qtzfStpjZveb2bhi8wQAAADwb1ioVd/c0x8U3hI5SdKuwp/5Z3ztPEnvdvt3OiP/jX8Xfh/tTyXdKGliSmmCpGb1rvHITknzCou2nrIvpJQmdPtTn1J6SZJSSt9MKS2XtFRdb4H8k16MBwAAAKCAhVr1fczMrjKzOnX9rtralNJOST+VdJ6Z/WczqzWzz0haoq7fG/PskXROt3+PldQhaZ+kWjP7b5J6++rWLyU1SbrXzEab2Ugzu7KQfVvSl8xsqSSZ2Xgz+3Th48vM7AozGy7pmKRWSad6OSYAAAAAsVDLwd9L+nN1vX1wubqaiyildEDSxyXdpa5mHHdL+nhKaX9wrr+W9KlCJ8ZvSnpS0uOStqjrbZOt6sXbJQvjn1LX78MtkvSOpEZ1NQZRSulHkv6PpEfMrEXSBkkfLRw6TtIDkg4Vxjwg6f/2ZkwAAAAAXSylM989h4FiZqskNaaU/mu15wIAAAAgH7yiBgAAAACZYaEGAAAAAJnhrY8AAAAAkBleUQMAAACAzLBQAwAAAIDM9LSZca+Z2fXqaglfI+n/pZTujb6+rq4u1dfXe+cKxxo9erSbHTt2rNhUezRsWLxOHTVqlJs1Nze7WW1tfLOeOuVvKxbdDuPG+VugnThxIhzz+PHjbjZy5Eg3i+Ya/Uwkqa2tzc2it9y2t7eH562rq3Oz6LZvaWlxs+i2leLbt6WlZX9KaWp4gjL1pdamTJmSFixYUMnpAANu/fr1WdWZRK3h7FTpWqPOgN7XWckLNTOrkfS3kv69uvbY+pWZrU4pveEdU19frw984AM9ZjU1NeF4V1xxhZutXbvWzaIn7sOHDw/HvPTSS93sySefdLOJEyeG5z1y5EhJc7r22mvdbOPGjeGY69evd7MlS5a4WbQgXbFiRThmY2Ojm7W2trrZ3r17w/POmTPHzaZO9a/5p556ys2uu+66cMzo9n388ccbwoPL1NdaW7BggdatW1fJKQEDzsyyqjOJWsPZqZK1Rp0BXXpbZ+W89fFySdtSSttTSm2SHpF0QxnnA9Azag2oPOoMqDzqDOiDchZqsyXt7PbvxsLnfoOZ3WFm68xsXfRWOACuorXWvc727ds3oJMDzhJ9fkyj1oA+o86APihnodbTL1O95xePUkr3p5RWpJRWRL9bBMBVtNa611n09k8Arj4/plFrQJ9RZ0AflLNQa5Q0t9u/50jaVd50APSAWgMqjzoDKo86A/qgnIXarySda2bvM7M6STdJWt0/0wLQDbUGVB51BlQedQb0QcldH1NKHWb2RUlPqqvF6sqUUth6sLa21u2IePjw4XC8F1980c2il8Wj9vLF2vpv3brVzaLW/vPmzQvPG3U1PHnypJs999xzbrZz5043k6T3v//9bhZ9L9HbVV955ZVwzKj7ZdQOf8KECeF5X331VTeLWvB7HUcl6de//nU45pYtW8K8kkqpNQB9Q50BlUedAX1T1j5qKaWfSvppP80FgINaAyqPOgMqjzoDeq+ctz4CAAAAACqAhRoAAAAAZIaFGgAAAABkhoUaAAAAAGSGhRoAAAAAZIaFGgAAAABkpqz2/H3V0dGhQ4cO9Zg98cQT4bEf+tCH3GzSpElu1tra6mbF9uuK9lkbM2aMm0X7r0nSnDlz3Ky5udnNli1b5mbF9m6L9kPbs2ePm0XfZ7TnmySNGDHCzXbt2uVm48ePD8977bXXutkbb7zhZp2dnW4W7cUnSQsXLnSz+++/Pzz2bBHdflGdbd++PTxvtK9gVCvnnHNOeN7a2gG9ewMAAOhXvKIGAAAAAJlhoQYAAAAAmWGhBgAAAACZYaEGAAAAAJlhoQYAAAAAmWGhBgAAAACZGdD+1bW1tZo4cWKP2Yc//OHw2KuvvtrN3n33XTcbN26cm9XU1IRj7t69283q6+vdrFjb/507d7rZhRde6GYHDx50sxkzZoRjdnR0uNncuXPd7MSJE242duzYcMy3337bzWbNmlVSJkkjR450s4svvtjN9u/f72bDhsX/ZzFz5swwHwq8rTUk6S//8i/drKGhITxv9PN+7rnn3GzNmjXheadMmRLmAAavlNKAj2lmAz4mgKGNV9QAAAAAIDMs1AAAAAAgMyzUAAAAACAzLNQAAAAAIDMs1AAAAAAgMyzUAAAAACAzA9qev6OjQ4cPH+4xGz9+fHjsk08+6WZRe/mo9XdTU1M4ZnTeqVOnutmBAwfC81522WVu5t0+kjRnzhw3O3XqVDjm7Nmz3ezIkSNu1tra6majRo0Kx5w+fXpJx0a3rSS1t7e7WUtLi5vV1vqX+zvvvBOO+eyzz4b52SLaxiHaVuJb3/qWm61evTocM7rmoy0VirXKjtp302YbqL5yWux3dna6WTXqm/sUAJXAK2oAAAAAkBkWagAAAACQGRZqAAAAAJAZFmoAAAAAkBkWagAAAACQGRZqAAAAAJCZstrzm9kOSUcknZLUkVJaEX39yJEjtWjRoh6z119/PRxr/vz5btbW1uZmI0aMcLMZM2aEY27YsMHNhg3z17g1NTXheUtthz9lyhQ327ZtWzjm5MmT3ezo0aNuNnHixPC8kai1+hVXXOFm27dvD89bX1/vZgcPHnSzaKuGYttDFNv+oNL6Wmuliq7r6Pr76le/6mZPP/10OOaYMWPc7Atf+IKbFfuZ0S4bfTVQdYbiymndH20zErX1r6urK3lO3N/03mCvs3KuTQwt/XG/0B/7qH04peQ/IwfQX6g1oPKoM6DyqDOgF3jrIwAAAABkptyFWpL0lJmtN7M7+mNCAHpErQGVR50BlUedAb1U7lsfr0wp7TKzaZKeNrPNKaUXun9BoQjvkKSxY8eWORwwZIW11r3O5s2bV605AoNdnx7TqDWgJNQZ0EtlvaKWUtpV+HuvpB9JuryHr7k/pbQipbQiagIBwFes1rrX2dSpU6sxRWDQ6+tjGrUG9B11BvReyQs1MxttZmNPfyzpOkl+m0QAJaHWgMqjzoDKo86AvinnrY/TJf2o0HqyVtLfp5Se6JdZAeiOWgMqjzoDKo86A/qg5IVaSmm7pIv7ckx7e7uampp6zEaOHBkeG/1+W7T3U3Nzs5uNGzcuHPOqq65ys71797rZoUOHwvNG77eeO3eum0X7nUV7w0jS1Vdf7WZr1qxxs/b2djerrY0vn0svvdTNor3Siu1vF+2VFu3JFe2pV+xtuUuWLHGzt956Kzy2XKXUWqmiWpo9e7ab/dEf/ZGbFduDLqrRqJZaWlrC80b3KdHPm/2QhqaBrDMUF+2PKkkbN250s3379rnZzJkz3azYYw9vwSvfYKmzcvZKY5+1+HH0bLp9BuL5Au35AQAAACAzLNQAAAAAIDMs1AAAAAAgMyzUAAAAACAzLNQAAAAAIDMs1AAAAAAgM+Xso9ZnnZ2dbrv3Yq3yx48f72Zvv/22m1122WVutmnTpnDM8847z80mTJjgZh0dHeF5d+3a5WaLFy92s1GjRrlZsXan69atc7OojfnJkyfdrK6uLhxz9OjRbha1Od6wId77MmqHf84557jZsWPH3KzY9hDRNTZURG1oo60aim3jMGnSJDfbsmWLm61cuTI875EjR9zs5ptvdrMPfvCD4XkB9I9o6461a9eGxz7++ONuFm3dccEFF7jZihUrwjGjx4noOQoGn3Lay1eqNX10bDRmOS3kc2ulH80n2sJKkvbs2eNmjz32mJt95CMfcbOLL678ThO8ogYAAAAAmWGhBgAAAACZYaEGAAAAAJlhoQYAAAAAmWGhBgAAAACZYaEGAAAAAJkZ0Pb8Zua2CR0xYkR4bNRWc8qUKW62bds2N4va3UvSvn37wtwza9asMI/alUet/aMsakUuSXPmzHGzqKVpTU2NmzU3N4djRu35y2mVH23l0Nra6mbR7dfY2BiOGf1Mi20ngFh0LfzkJz8p6Tgpvo6i66Szs9PNhg3j/7YweEWtrUtt/V3s2MiJEyfc7Mc//nF47KOPPupmUZ1GWbRViCRdfvnlYQ5IlWtpX06bfU85c42214jqLHpeLkkvvPCCmzU0NLjZ7t27w/O++eabbrZ582Y3O/fcc92M9vwAAAAAMASxUAMAAACAzLBQAwAAAIDMsFADAAAAgMywUAMAAACAzLBQAwAAAIDMsFADAAAAgMwM6D5qKSV3L6toLzRJamtrc7Noj7WJEye62fDhw8Mxo73d3n33XTe76KKLwvPW1dW5WbRPRrQP2Pve975wzOh7nT9/vps1NTW5WbTHmhTvaRHtv7F169bwvDNnznSz9vZ2N1uwYIGbRdeXVPz6ROnGjBnjZrfffrubrVq1Kjzvrl273GzhwoVuFu0rGO3hB1RbsT2RovvdcvZoKnUvpuh+94033giP3blzp5tFj/vRPqYHDx4Mx2QfxaGjUnuhlSOqlyiLHtN+9rOfhWNGe5p99KMfdbOTJ0+62fe+971wzOj5dbQ/avR9SvHPdNmyZW52ySWXhOetNO51AAAAACAzLNQAAAAAIDMs1AAAAAAgMyzUAAAAACAzLNQAAAAAIDMs1AAAAAAgM0Xb85vZSkkfl7Q3pXRh4XOTJP1A0gJJOyTdmFI6VOxcdXV1bov0JUuWhMe+9NJLbjZnzhw3i9rtTps2LRyzoaHBza688ko3O3LkSHjeY8eOuVnU/jtqrRy1HC6mtbXVzWbMmOFmb731VnjeUtsgX3DBBeF5I9Gx//AP/+Bm48ePL3nM/tKftTaYRNs8zJs3z83uvvvu8Lz33nuvmz3yyCNutmPHDjf70pe+FI4ZbQFRTvtz9J+hWmdS3F6+nHbkpV7b+/fvd7OWlpbw2Ojxe8KECW524MABNxs7dmw4Zo4t23M1lOssEm0r8fOf/zw89sSJE24Wta3ft2+fm0VbWUjx1k8rV650s6jFfvScU4rrd+nSpW4WtdiXpOXLl7tZdH8SbQk1EHrzitoqSdef8bl7JD2TUjpX0jOFfwMozypRa0ClrRJ1BlTaKlFnQNmKLtRSSi9IOvPljxskPVT4+CFJn+jneQFDDrUGVB51BlQedQb0j1J/R216SqlJkgp/u68ZmtkdZrbOzNZFL9sC6FGvaq17nUVvcwDQo5Ie06g1oE+oM6CPKt5MJKV0f0ppRUppRX19faWHA4ak7nU2derUak8HOGtRa0DlUWdAl1IXanvMbKYkFf7e239TAtANtQZUHnUGVB51BvRRqQu11ZJuK3x8m6R/7p/pADgDtQZUHnUGVB51BvRRb9rzf1/S1ZKmmFmjpD+XdK+kR83s9yS9I+nTvRmss7NTR48e7TEr1up9/fr1bnb99Wc2Fvo3HR0dbha1u5ek6dOnu9nu3bvdbNGiReF5o5bE0dtDo9bKxdqdfupTn3KzZ555xs02bdrkZueee2445ogRI9zs7bffdjPvGjlt717/P+Eee+wxN4u2C4jmKklbt24N8/7Qn7V2toh+ZidPngyPjbb0+P3f/303i7blKGcbDORhKNdZpdrLR+eN2uw/+OCDblbscTR6rNy1a5ebbd682c2Ktedni43eG6g6K+ea3r59u5vNnj3bzerq6sLztrW1udmPf/xjN7vvvvvC837uc59zs6i9fHt7u5tFz/+keDuBqLbnz5/vZtGWWpL02c9+1s0uv/xyN4ueIw9mRZ91pJRudqJr+nkuwJBGrQGVR50BlUedAf3j7Fx+AgAAAMAgxkINAAAAADLDQg0AAAAAMsNCDQAAAAAyw0INAAAAADIzoL2mzcxtb11O+/N33nnHzVasWOFmxdp7jxs3zs0mTZrkZkeOHAnPO3z4cDeLWvd/5CMfcbMtW7aEY0atyqP5nnfeeW4W3e5S3C42at0atdiXpMWLF7tZ1Np/2bJlblasRe2oUaPCHAMvqkFJ+vKXv+xmjY2Nbha17i/WCjqqpej+BBgI0fVbTmvr6PH77rvvdrN/+Zd/cbOrrroqHDPakqazs9PNohqdOnVqOCYGn+haiH7ex48fd7Po8UOSHnjgATdbuXKlmxXbZil6fvP666+7WbTlTPR8VJIuvvhiN/v0p/3dFaLjij2fqkab/Zy33uAVNQAAAADIDAs1AAAAAMgMCzUAAAAAyAwLNQAAAADIDAs1AAAAAMgMCzUAAAAAyAwLNQAAAADIzIDuo1ZTU6OJEyf2mB06dCg8duHChW4W7as1duxYNxszZkw4ZnTsvn373Gz06NHheb295KR4r6X169e7WV1dXThmdPtG+0e0t7e7WXNzczjmjBkz3Cza82PevHnheaPbfvLkyW62adOmko6TpAMHDoQ5Bl6xfU+uueYaNzt16pSbrVq1ys2iPXmkeG/G3/3d33WzYvdFQH+oqalxs2iPtba2tvC83/72t90s2hcz2sdq6dKl4ZjRfmjHjh1zs+i+/JZbbgnHjO43qrH3E4qLruunnnrKzdauXetmTzzxRDhmtK9tVEvFHtMefvhhN5s9e7ab3XDDDW4W7YUmxc/jij3vLFX0M8t5v7NK4Z4FAAAAADLDQg0AAAAAMsNCDQAAAAAyw0INAAAAADLDQg0AAAAAMsNCDQAAAAAyM6Dt+VNKbvvqYu1/vbb+Utymd9KkSW5WX18fjhm1c49aB7e0tITnPXr0qJtFWwJErb/3798fjhl9r4cPH3azOXPmuFnUQlWSXn75ZTeLbr+o/b4UbzXw5ptvutn8+fPd7Pjx4+GYo0aNCnPkp9Q2vtH2GsXup0aOHOlm0bUZbT8yYcKEcEzgtGLXfLH7bE/0GCFJa9ascbO5c+e62T333ONmn/nMZ8Ixo61j9uzZ42bPPvtseN5ItL0BBp/o/jraDmnBggXheaPnNw0NDW4WbVchSdddd52b3XTTTW4WPQ8uR3R/U+p9TbHzlmOwtvbnFTUAAAAAyAwLNQAAAADIDAs1AAAAAMgMCzUAAAAAyAwLNQAAAADIDAs1AAAAAMhM0fb8ZrZS0scl7U0pXVj43F9Iul3S6T7qX04p/bSciTz33HNhfvPNN7tZ1NJ0+PDhbnbixIlwzEWLFrlZ1CK+WKv86LxRa//o+4za6EtSY2Ojm9XW+pdB1I48ao8sSfPmzXOzaOuDYq3yo1azF154oZsdPHjQzXbu3BmOuXz58jDvDwNVa4iv+RtvvNHNtm3bFp73O9/5jptt3rzZzc4//3w3+53f+Z1wzMHacrhaqLOeRe20n3/++fDY6LHgj//4j93s9ttvd7Nhw0r/f+Tx48e72YgRI9yMrTD6Ty511tra6mZjxoxxswceeMDNou2iJOnUqVNu1tzcXPJ5o+uzGo8D5bTgr4Zovjk/jvbmnnCVpOt7+Pw3UkrLCn+G1AMaUCGrRK0BlbZK1BlQaatEnQFlK7pQSym9IMl/KQJAv6DWgMqjzoDKo86A/lHO76h90cxeM7OVZha/XgugHNQaUHnUGVB51BnQB6Uu1O6TtFDSMklNkr7mfaGZ3WFm68xsXbHfCQPwHr2qte51tm/fvp6+BICvpMc0ag3oE+oM6KOSFmoppT0ppVMppU5JD0i6PPja+1NKK1JKK+rr60udJzAk9bbWutdZ1HQGwHuV+phGrQG9R50BfVfSQs3MZnb7529L2tA/0wHQHbUGVB51BlQedQb0XW/a839f0tWSpphZo6Q/l3S1mS2TlCTtkPSF3gx26tQpt/181FZdilu97927182mTJniZkePHg3HjF5uj9r/Fmvxe+DAgZLOe+zYMTd7/fXXwzGnT5/uZkeOHHGzadOmudnJkyfDMaM2+9H/kK1fvz487+HDh93slltucbP77rvPzaLrS4pb6vaX/qy1ShisrW37qqamxs2KbUkRXSczZsxws7lz57pZZ2dnOGY0X7xX7nVWjmLtskut4V/84hfheaN25KNGjSppPsVEx0bbaDzxxBNuduedd4Zjnk33c5WWS51F948XXXSRm0XPxYpdB1E+efLkks9bar2Uc91WogV/teYzWOu36EItpdTTBmYPVmAuwJBGrQGVR50BlUedAf2jnK6PAAAAAIAKYKEGAAAAAJlhoQYAAAAAmWGhBgAAAACZYaEGAAAAAJlhoQYAAAAAmSnanr8/dXZ2qrW1tcds2bJl4bHRPmE33XSTm23atMnNhg2L16ljxoxxs7q6OjeL9g+T4n1lDh065GbXX3+9mzU0NIRjFtvzzPPLX/7SzUaOHBkeG90OY8eOdbMFCxaE521qanKzF198saTjop+JJK1evdrNvvWtb4XHni0G6x4k/Wnp0qVh/ld/9VdutmvXLjfr6Ohws1dffTUcM9qTMNqfDegueuwptpff8OHD3WzNmjVuFu0teOWVV4ZjRs8Jovvkyy67zM2KPScYKntJnk1Kva4nTpzoZuX8rKt17GAaM5LbfAYCr6gBAAAAQGZYqAEAAABAZlioAQAAAEBmWKgBAAAAQGZYqAEAAABAZlioAQAAAEBmBrQ9/+TJk3Xrrbf2mP3whz8Mj21vb3ezp59+2s2iVu8tLS3hmFHL9qjd/fTp08Pz1tfXu9nLL7/sZlGbWW/bg9Pa2trcLLqNom0ImpubwzGnTJniZtH3sm/fvvC80c/ttddec7Ply5e7WbHtCz75yU+GOYaGYq2BJ0+e7GbRdhXf/OY33WzhwoXhmG+++aabffWrX3WzqKU6hp7ovn7x4sXhsdG1HdXMtm3b3KzYffIrr7ziZlHNXHLJJW5WrD3/UGwNPthFW0AAgwGvqAEAAABAZlioAQAAAEBmWKgBAAAAQGZYqAEAAABAZlioAQAAAEBmWKgBAAAAQGYGtD3/nj179PWvf73HbNq0aeGxS5cudbN3333XzZqamtxs7Nix4ZhR++qodfDGjRvD886fP9/NLrzwQjeL2hEfOHAgHHPWrFlutmHDBjfbtWuXm0XbF0hSY2Ojm0VbFES3jxS3I49a8UYtpIttCbBnz54wB4qprfXvbqN6KLaNyFVXXVXSmBh6ovbyo0ePdrPPf/7z4XlvvPFGNxsxYoSbRY+xxVrlR48TKSU3W7ZsmZvV1NSEY+LswnYLGAx4RQ0AAAAAMsNCDQAAAAAyw0INAAAAADLDQg0AAAAAMsNCDQAAAAAyw0INAAAAADJTdKFmZnPN7Dkz22RmG83sDwqfn2RmT5vZ1sLfEys/XeDsRJ0BA4NaAyqPOgP6R2822emQdFdK6WUzGytpvZk9Lelzkp5JKd1rZvdIukfSn0YnGjZsmLtXyznnnBNOYufOnW62fPlyN9u+fbubHT58OBwz2nss2n9j4sT4fifa4+XIkSNuFu31dcUVV4RjRrdDtJfN4sWL3Wzz5s3hmA0NDW42adIkN2tvbw/PO2/ePDcbN26cm0U/761bt4ZjdnR0hHk/6Lc6Q55mzpzpZnfddZebFbv2oj3+fvGLX7jZ3Llzw/POmTMnzAcxaq2Piu0vFj3mdXZ2ulmxvdIi0eN+pUSP3ezJ9R5Z1Bk/Fwx2Re8lU0pNKaWXCx8fkbRJ0mxJN0h6qPBlD0n6RKUmCZztqDNgYFBrQOVRZ0D/6NN/Z5nZAkmXSForaXpKqUnqKkhJ0/p7csBQRJ0BA4NaAyqPOgNK1+uFmpmNkfSPkv4wpdTSh+PuMLN1Zraura2tlDkCQ0Z/1Fn0FlkAXag1oPKoM6A8vVqomdlwdRXa36WU/qnw6T1mNrOQz5S0t6djU0r3p5RWpJRW1NXV9cecgbNSf9XZ1KlTB2bCwCBFrQGVR50B5etN10eT9KCkTSmlr3eLVku6rfDxbZL+uf+nBwwN1BkwMKg1oPKoM6B/9Kbr45WSbpX0upm9UvjclyXdK+lRM/s9Se9I+nRlpggMCdQZMDCoNaDyqDOgHxRdqKWU1kjy+pte05fBUkrJaJwlAAAH8klEQVTyfk9t9+7d4bH19fVu9utf/9rNxo4d62ZjxowJx9y2bZubRS1f58+fH553//79bjZ9+nQ3mzJlipu1tMRv/Y7e4x21w4/aEUet8CXp/PPPLylrbGwMzztjxgw3W7dunZtFbf2LtUCv9Fsv+rPOMPiMHz/ezYr9fsZDDz3kZtG2HU899VR43j/7sz9zs3Laqlfb2VxrxVqRl9pevlh7/ui80bWSW+v06PuQ8ptvznKpM7ZUwGA3eB9tAQAAAOAsxUINAAAAADLDQg0AAAAAMsNCDQAAAAAyw0INAAAAADLDQg0AAAAAMtObfdT6TWdnp06cOFHSsVH79Ki9/PDhw91s8+bN4ZhRG/ioXfGGDRvC806bNs3NXnrpJTe76KKLSh7z8OHDbrZgwQI3i7ZNiLZMkKSDBw+62erVq91s0aJF4Xmj7zXacuHRRx91s2i7AEmaO3dumAOVUqwVfl1dnZvt2LHDzaLW/ZLU0NDgZrNmzXKzESNGhOfF2adYW3tP1B69Gq3Tadd+9uFnisGOV9QAAAAAIDMs1AAAAAAgMyzUAAAAACAzLNQAAAAAIDMs1AAAAAAgMyzUAAAAACAzA9qev7a2VpMmTeoxi9qqS9LatWvdrK2tzc0OHTrkZqdOnQrHjFqyv/POO27W3Nwcnjdqt33s2DE3a2xsdLNt27aFY0ZzmjNnjpu99dZbblasPX9kyZIlbvbiiy+Gx3rXkBS3Kr/mmmvcbMuWLeGY0c8bqKTJkyeH+Ve+8hU3e+yxx9ysWJ09+OCDbvahD33Ize688043o1X22YmfK6qFaw9nO15RAwAAAIDMsFADAAAAgMywUAMAAACAzLBQAwAAAIDMsFADAAAAgMywUAMAAACAzLBQAwAAAIDMDOg+asOGDdPIkSN7zJqamsJjjx8/XtKYw4cPd7Ni+2/s2LHDzU6ePOlm0V5oxUR7mr366qtuNmrUqPC8s2bNcrPodnj++efdbMWKFeGY3s9aiveSK6a1tdXNfvCDH7jZ0qVL3aylpSUcM9pTD6imlJKbPfzww2526623hueN9g6M6qWzs9PNampqwjExOJVzfw4A8HHvCgAAAACZYaEGAAAAAJlhoQYAAAAAmWGhBgAAAACZYaEGAAAAAJlhoQYAAAAAmbGotXO/D2a2T1JDt09NkbR/wCZQHPMpLrc55TCf+SmlqVWew7+izkqS25xym49U/TllVWfSe2qt2rdPT3KbE/MpLoc5ZVVrPKb1WW7zkfKbUw7z6VWdDehC7T2Dm61LKcWbcQ0g5lNcbnPKbT45yu02ym0+Un5zym0+Up5zykmOt09uc2I+xeU4p9zkdhsxn+Jym1Nu84nw1kcAAAAAyAwLNQAAAADITLUXavdXefwzMZ/icptTbvPJUW63UW7zkfKbU27zkfKcU05yvH1ymxPzKS7HOeUmt9uI+RSX25xym4+rqr+jBgAAAAB4r2q/ogYAAAAAOENVFmpmdr2ZvWlm28zsnmrM4UxmtsPMXjezV8xsXRXGX2lme81sQ7fPTTKzp81sa+HviRnM6S/M7N3C7fSKmX1sAOcz18yeM7NNZrbRzP6g8Pmq3k65os7cOWRVa9TZ4JdbrVFnvZ4PdTaI5FZnUvVrLbc6C+ZErZVowBdqZlYj6W8lfVTSEkk3m9mSgZ6H48MppWVVatm5StL1Z3zuHknPpJTOlfRM4d/VnpMkfaNwOy1LKf10AOfTIemulNJiSf9O0p2Fa6fat1N2qLPQKuVVaz3NR6LOBoWMa406Kz4fiTobFDKuM4nnjr2Zk0StlaQar6hdLmlbSml7SqlN0iOSbqjCPLKSUnpB0sEzPn2DpIcKHz8k6RMZzKlqUkpNKaWXCx8fkbRJ0mxV+XbKFHXmyK3WqLNBj1rrAXUWo876jDrrQW51FsypagZ7rVVjoTZb0s5u/24sfK7akqSnzGy9md1R7ckUTE8pNUldF5qkaVWez2lfNLPXCi9vV+WlYjNbIOkSSWuV7+1UTdRZ3+R4DVFng0OOtUad9R51NjjkWGdSnrWW6zVErZWgGgs16+FzObSevDKldKm6Xla/08x+q9oTytR9khZKWiapSdLXBnoCZjZG0j9K+sOUUstAjz9IUGeDG3U2eORYa9RZ71Bng0eOdSZRa71FrZWoGgu1Rklzu/17jqRdVZjHb0gp7Sr8vVfSj9T1Mnu17TGzmZJU+HtvleejlNKelNKplFKnpAc0wLeTmQ1XV6H9XUrpnwqfzu52ygB11jdZXUPU2aCSXa1RZ71DnQ0q2dWZlG2tZXcNUWulq8ZC7VeSzjWz95lZnaSbJK2uwjz+lZmNNrOxpz+WdJ2kDfFRA2K1pNsKH98m6Z+rOBdJ/3oxn/bbGsDbycxM0oOSNqWUvt4tyu52ygB11jdZXUPU2aCSVa1RZ71HnQ0qWdWZlHWtZXcNUWtlSCkN+B9JH5O0RdJbkr5SjTmcMZ9zJL1a+LOxGnOS9H11vRzcrq7/Ofo9SZPV1Ylma+HvSRnM6buSXpf0mrou8pkDOJ+r1PVWh9ckvVL487Fq3065/qHO3HlkVWvU2eD/k1OtUWd9mg91Noj+5FRnhflUvdZyq7NgTtRaiX+s8E0AAAAAADJRlQ2vAQAAAAA+FmoAAAAAkBkWagAAAACQGRZqAAAAAJAZFmoAAAAAkBkWagAAAACQGRZqAAAAAJAZFmoAAAAAkJn/DwN6W6kuJS7cAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1080x1440 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from random import randint\n",
    "# look at importances on a log scale\n",
    "plt.figure(figsize=(15,20))\n",
    "plt.subplot(1,4,1)\n",
    "plt.imshow(np.log(imp_as_image), cmap=plt.cm.gray)\n",
    "plt.title('Importances')\n",
    "\n",
    "for i in range(2,5):\n",
    "    plt.subplot(1,4,i)\n",
    "    plt.imshow(np.reshape(X[randint(0,len(X))],(25,25)),cmap=plt.cm.gray)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Other Methods for Making the Tree Generalize\n",
    "Everything that sciki-learn implements is used for pre-pruning the tree: that is, we stop making nodes during the decision tree induction process. However, there are other methods that post-prune the tree where:\n",
    "- We build a tree completely\n",
    "- Then select a group a leaf nodes that belong to a split \n",
    "- And, finally decide if we should prune the split (making the parent node a leaf node)\n",
    "- Then repeat until some criteria is met\n",
    "\n",
    "This is called post pruning and can be a good method for creating a decision tree that is not \"overlearned\". Let's look at these in lecture now."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}