01-intro-regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning to Rank with Linear Regression\n",
    "\n",
    "In this notebook we load in the data set and train a linear regression model on the data. We then use this to rank data sets. \n",
    "\n",
    "In later notebooks we implement more sophisticated algorithms for Learning to Rank on the same data set. \n",
    "\n",
    "The data set we use here is the MSLR-WEB10K data set, which is open sourced by Mircosoft and can be downloaded from [here](https://www.microsoft.com/en-us/research/project/mslr/). We will only work with Fold1 of the data set. This notebook assumes that the contents of Fold1 are stored in a subdirectory named 'data'. If your data is stored elsewhere you will need to ammend the paths appropriately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = pd.read_csv(\"data/train.txt\", header=None, sep=\" \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect a sample of 5 rows from the data: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "np.random.seed(0xc00f33e)\n",
    "df_samp = train.sample(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>129</th>\n",
       "      <th>130</th>\n",
       "      <th>131</th>\n",
       "      <th>132</th>\n",
       "      <th>133</th>\n",
       "      <th>134</th>\n",
       "      <th>135</th>\n",
       "      <th>136</th>\n",
       "      <th>137</th>\n",
       "      <th>138</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>84134</th>\n",
       "      <td>0</td>\n",
       "      <td>qid:10591</td>\n",
       "      <td>1:2</td>\n",
       "      <td>2:0</td>\n",
       "      <td>3:2</td>\n",
       "      <td>4:2</td>\n",
       "      <td>5:2</td>\n",
       "      <td>6:1</td>\n",
       "      <td>7:0</td>\n",
       "      <td>8:1</td>\n",
       "      <td>...</td>\n",
       "      <td>128:0</td>\n",
       "      <td>129:3</td>\n",
       "      <td>130:65142</td>\n",
       "      <td>131:55648</td>\n",
       "      <td>132:1</td>\n",
       "      <td>133:1</td>\n",
       "      <td>134:0</td>\n",
       "      <td>135:0</td>\n",
       "      <td>136:0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>625314</th>\n",
       "      <td>2</td>\n",
       "      <td>qid:17722</td>\n",
       "      <td>1:0</td>\n",
       "      <td>2:2</td>\n",
       "      <td>3:0</td>\n",
       "      <td>4:0</td>\n",
       "      <td>5:2</td>\n",
       "      <td>6:0</td>\n",
       "      <td>7:0.50000</td>\n",
       "      <td>8:0</td>\n",
       "      <td>...</td>\n",
       "      <td>128:32</td>\n",
       "      <td>129:0</td>\n",
       "      <td>130:36833</td>\n",
       "      <td>131:50372</td>\n",
       "      <td>132:23</td>\n",
       "      <td>133:4</td>\n",
       "      <td>134:0</td>\n",
       "      <td>135:0</td>\n",
       "      <td>136:0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>707552</th>\n",
       "      <td>1</td>\n",
       "      <td>qid:27727</td>\n",
       "      <td>1:1</td>\n",
       "      <td>2:0</td>\n",
       "      <td>3:1</td>\n",
       "      <td>4:0</td>\n",
       "      <td>5:1</td>\n",
       "      <td>6:1</td>\n",
       "      <td>7:0</td>\n",
       "      <td>8:1</td>\n",
       "      <td>...</td>\n",
       "      <td>128:3</td>\n",
       "      <td>129:2</td>\n",
       "      <td>130:620</td>\n",
       "      <td>131:16</td>\n",
       "      <td>132:25</td>\n",
       "      <td>133:126</td>\n",
       "      <td>134:0</td>\n",
       "      <td>135:0</td>\n",
       "      <td>136:0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>458013</th>\n",
       "      <td>2</td>\n",
       "      <td>qid:26539</td>\n",
       "      <td>1:2</td>\n",
       "      <td>2:2</td>\n",
       "      <td>3:2</td>\n",
       "      <td>4:0</td>\n",
       "      <td>5:2</td>\n",
       "      <td>6:1</td>\n",
       "      <td>7:1</td>\n",
       "      <td>8:1</td>\n",
       "      <td>...</td>\n",
       "      <td>128:12</td>\n",
       "      <td>129:8</td>\n",
       "      <td>130:4682</td>\n",
       "      <td>131:51303</td>\n",
       "      <td>132:4</td>\n",
       "      <td>133:7</td>\n",
       "      <td>134:0</td>\n",
       "      <td>135:0</td>\n",
       "      <td>136:0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4 rows × 139 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        0          1    2    3    4    5    6    7          8    9    ...  \\\n",
       "84134     0  qid:10591  1:2  2:0  3:2  4:2  5:2  6:1        7:0  8:1  ...   \n",
       "625314    2  qid:17722  1:0  2:2  3:0  4:0  5:2  6:0  7:0.50000  8:0  ...   \n",
       "707552    1  qid:27727  1:1  2:0  3:1  4:0  5:1  6:1        7:0  8:1  ...   \n",
       "458013    2  qid:26539  1:2  2:2  3:2  4:0  5:2  6:1        7:1  8:1  ...   \n",
       "\n",
       "           129    130        131        132     133      134    135    136  \\\n",
       "84134    128:0  129:3  130:65142  131:55648   132:1    133:1  134:0  135:0   \n",
       "625314  128:32  129:0  130:36833  131:50372  132:23    133:4  134:0  135:0   \n",
       "707552   128:3  129:2    130:620     131:16  132:25  133:126  134:0  135:0   \n",
       "458013  128:12  129:8   130:4682  131:51303   132:4    133:7  134:0  135:0   \n",
       "\n",
       "          137 138  \n",
       "84134   136:0 NaN  \n",
       "625314  136:0 NaN  \n",
       "707552  136:0 NaN  \n",
       "458013  136:0 NaN  \n",
       "\n",
       "[4 rows x 139 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_samp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each row represents a query - document pair. \n",
    "\n",
    "Column '0' is an integer taking values from 0 to 4, where 0 denotes \"this document is irrelevant for this query\", and 4 denotes \"this document is perfectly relevant for this query\".\n",
    "\n",
    "Column '1' holds an integer identifying the query and is of the form 'qid:int'.\n",
    "\n",
    "Columns '2' through to '137' hold the features for the query - document pair. The data in these columns are of the form 'feature_id:feature_value'. Full details about each of the features can be found here. \n",
    "\n",
    "Finally, column '138' is NaN for every row. (It is an artifact of the way we split the data set into columns.)\n",
    "\n",
    "#### Processing the data\n",
    "\n",
    "We want to do three things to the data frame in order to get it into an easier form to work with: \n",
    "1. replace 'qid:int' with 'int' in column '1'\n",
    "2. replace 'feature_id:feature_value' with 'feature_value' for all entries in columns '2' to '137'\n",
    "3. delete column '138'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_qid(qid_str):\n",
    "    return qid_str[4:]\n",
    "\n",
    "def extract_val(feat):\n",
    "    return feat.split(':')[1]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>128</th>\n",
       "      <th>129</th>\n",
       "      <th>130</th>\n",
       "      <th>131</th>\n",
       "      <th>132</th>\n",
       "      <th>133</th>\n",
       "      <th>134</th>\n",
       "      <th>135</th>\n",
       "      <th>136</th>\n",
       "      <th>137</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>84134</th>\n",
       "      <td>0</td>\n",
       "      <td>10591</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>65142</td>\n",
       "      <td>55648</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>625314</th>\n",
       "      <td>2</td>\n",
       "      <td>17722</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0.50000</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>44</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "      <td>36833</td>\n",
       "      <td>50372</td>\n",
       "      <td>23</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>707552</th>\n",
       "      <td>1</td>\n",
       "      <td>27727</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>29</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>620</td>\n",
       "      <td>16</td>\n",
       "      <td>25</td>\n",
       "      <td>126</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>458013</th>\n",
       "      <td>2</td>\n",
       "      <td>26539</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>40</td>\n",
       "      <td>12</td>\n",
       "      <td>8</td>\n",
       "      <td>4682</td>\n",
       "      <td>51303</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4 rows × 138 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        0      1   2   3   4   5   6   7        8   9    ... 128 129 130  \\\n",
       "84134     0  10591   2   0   2   2   2   1        0   1  ...  33   0   3   \n",
       "625314    2  17722   0   2   0   0   2   0  0.50000   0  ...  44  32   0   \n",
       "707552    1  27727   1   0   1   0   1   1        0   1  ...  29   3   2   \n",
       "458013    2  26539   2   2   2   0   2   1        1   1  ...  40  12   8   \n",
       "\n",
       "          131    132 133  134 135 136 137  \n",
       "84134   65142  55648   1    1   0   0   0  \n",
       "625314  36833  50372  23    4   0   0   0  \n",
       "707552    620     16  25  126   0   0   0  \n",
       "458013   4682  51303   4    7   0   0   0  \n",
       "\n",
       "[4 rows x 138 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_samp[1] = df_samp[1].apply(extract_qid)\n",
    "\n",
    "df_samp[df_samp.columns[2:-1]] = df_samp[df_samp.columns[2:-1]].applymap(extract_val)\n",
    "#applymap applies the function to every element of the data frame. \n",
    "\n",
    "#remove the last column. \n",
    "df_samp.drop( 138, axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We wrap these transformations up in one function which we can then apply to the full training set. We also import and transform the testing set, which we will use to evaluate our model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def df_transform(df):\n",
    "    df[1] = df[1].apply(extract_qid)\n",
    "    df[df.columns[2:-1]] = df[df.columns[2:-1]].applymap(extract_val)\n",
    "    df = df.drop(138, axis=1)\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = df_transform(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>128</th>\n",
       "      <th>129</th>\n",
       "      <th>130</th>\n",
       "      <th>131</th>\n",
       "      <th>132</th>\n",
       "      <th>133</th>\n",
       "      <th>134</th>\n",
       "      <th>135</th>\n",
       "      <th>136</th>\n",
       "      <th>137</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>542060</th>\n",
       "      <td>2</td>\n",
       "      <td>8167</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>43</td>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>9559</td>\n",
       "      <td>39270</td>\n",
       "      <td>14</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120977</th>\n",
       "      <td>2</td>\n",
       "      <td>15121</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.50000</td>\n",
       "      <td>0.50000</td>\n",
       "      <td>...</td>\n",
       "      <td>57</td>\n",
       "      <td>9945</td>\n",
       "      <td>17</td>\n",
       "      <td>58604</td>\n",
       "      <td>60996</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>2405</td>\n",
       "      <td>99.6056888844168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>396343</th>\n",
       "      <td>0</td>\n",
       "      <td>18904</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>154</td>\n",
       "      <td>799</td>\n",
       "      <td>134</td>\n",
       "      <td>72</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 138 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        0      1   2   3   4   5   6   7        8        9    ... 128   129  \\\n",
       "542060    2   8167   2   0   2   1   2   1        0        1  ...  43     3   \n",
       "120977    2  15121   2   1   1   1   2   1  0.50000  0.50000  ...  57  9945   \n",
       "396343    0  18904   0   0   0   0   0   0        0        0  ...  21     0   \n",
       "\n",
       "       130    131    132  133 134 135   136               137  \n",
       "542060   7   9559  39270   14   5   0     0                 0  \n",
       "120977  17  58604  60996    8   4   0  2405  99.6056888844168  \n",
       "396343   0    154    799  134  72   0     0                 0  \n",
       "\n",
       "[3 rows x 138 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.sample(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = pd.read_csv(\"data/test.txt\", header=None, sep=\" \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df = df_transform(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = test_df[test_df.columns[2:]]\n",
    "y_test = test_df[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now go ahead and implement linear regression. We will use the features in columns '2' to '137' as the 'explanatory variables' to predict the relevence value, which is held in column '0'. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = train_df[train_df.columns[2:]]\n",
    "y = train_df[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "reg = LinearRegression().fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.13349313112592232"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reg.score(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can make predictions for the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = reg.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.76923034, 0.29349399, 0.3523804 , ..., 0.80657282, 0.43872805,\n",
       "       0.19482099])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For now, we consider just the mean squared error between the predictions and the results. We use the metrics already in scikit learn to compute these. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6055521003389398"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_squared_error(preds, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We plot the predicted score vs true score for the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_data = pd.concat([pd.DataFrame(y_test), pd.DataFrame(preds)], axis=1)\n",
    "plot_data.columns = plot_data.columns.astype(str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_data = pd.DataFrame(data= {\"true score\": y_test, \"predicted score\": preds})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>true score</th>\n",
       "      <th>predicted score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>217031</th>\n",
       "      <td>0</td>\n",
       "      <td>0.212263</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>191702</th>\n",
       "      <td>1</td>\n",
       "      <td>0.681021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110571</th>\n",
       "      <td>1</td>\n",
       "      <td>0.504319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1248</th>\n",
       "      <td>0</td>\n",
       "      <td>0.539843</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>227961</th>\n",
       "      <td>0</td>\n",
       "      <td>0.281012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42937</th>\n",
       "      <td>1</td>\n",
       "      <td>0.579672</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>194958</th>\n",
       "      <td>1</td>\n",
       "      <td>0.781614</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>115409</th>\n",
       "      <td>0</td>\n",
       "      <td>0.595637</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129065</th>\n",
       "      <td>0</td>\n",
       "      <td>0.830366</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198109</th>\n",
       "      <td>0</td>\n",
       "      <td>0.922282</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        true score  predicted score\n",
       "217031           0         0.212263\n",
       "191702           1         0.681021\n",
       "110571           1         0.504319\n",
       "1248             0         0.539843\n",
       "227961           0         0.281012\n",
       "42937            1         0.579672\n",
       "194958           1         0.781614\n",
       "115409           0         0.595637\n",
       "129065           0         0.830366\n",
       "198109           0         0.922282"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plot_data.sample(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda2/envs/ltr/lib/python3.7/site-packages/altair/utils/core.py:294: FutureWarning: A future version of pandas will default to `skipna=True`. To silence this warning, pass `skipna=True|False` explicitly.\n",
      "  attrs['type'] = infer_vegalite_type(data[attrs['field']])\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "var spec = {\"config\": {\"view\": {\"width\": 400, \"height\": 300}}, \"data\": {\"name\": \"data-7173ed480c1e87d32a039902b58f3ab6\"}, \"mark\": \"point\", \"encoding\": {\"x\": {\"type\": \"quantitative\", \"field\": \"true score\"}, \"y\": {\"type\": \"quantitative\", \"field\": \"predicted score\"}}, \"selection\": {\"selector001\": {\"type\": \"interval\", \"bind\": \"scales\", \"encodings\": [\"x\", \"y\"]}}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v2.6.0.json\", \"datasets\": {\"data-7173ed480c1e87d32a039902b58f3ab6\": [{\"true score\": 2, \"predicted score\": 0.7598852596445251}, {\"true score\": 1, \"predicted score\": 0.6772550372755961}, {\"true score\": 0, \"predicted score\": 0.6587997194163839}, {\"true score\": 0, \"predicted score\": 0.7126778906765421}, {\"true score\": 0, \"predicted score\": 0.24531145775302815}, {\"true score\": 1, \"predicted score\": 0.6925955504057342}, {\"true score\": 0, \"predicted score\": 0.40726091925717334}, {\"true score\": 1, \"predicted score\": 1.0521937415449438}, {\"true score\": 1, \"predicted score\": 0.766492847985498}, {\"true score\": 0, \"predicted score\": 0.33789952647197347}, {\"true score\": 1, \"predicted score\": 1.0702712859926375}, {\"true score\": 2, \"predicted score\": 0.5914883966134558}, {\"true score\": 1, \"predicted score\": 0.8146137142656665}, {\"true score\": 0, \"predicted score\": 0.5701355053636388}, {\"true score\": 0, \"predicted score\": 0.788979125186159}, {\"true score\": 2, \"predicted score\": 1.1453934742623415}, {\"true score\": 1, \"predicted score\": 0.6638576658681826}, {\"true score\": 0, \"predicted score\": -0.04727917999993103}, {\"true score\": 0, \"predicted score\": 0.9629991579307791}, {\"true score\": 0, \"predicted score\": 0.29194646120354695}, {\"true score\": 1, \"predicted score\": 0.9411929181005743}, {\"true score\": 0, \"predicted score\": 0.7242855310413361}, {\"true score\": 0, \"predicted score\": 0.9863286321451701}, {\"true score\": 1, \"predicted score\": 0.8411443254986622}, {\"true score\": 0, \"predicted score\": 1.6254231182472978}, {\"true score\": 0, \"predicted score\": 1.1916241806181675}, {\"true score\": 0, \"predicted score\": 1.0745764202192802}, {\"true score\": 0, \"predicted score\": 0.4817266799625288}, {\"true score\": 1, \"predicted score\": 0.9182245887737266}, {\"true score\": 0, \"predicted score\": 1.032395610370716}, {\"true score\": 1, \"predicted score\": 0.7899826557551879}, {\"true score\": 0, \"predicted score\": 0.4427328331805501}, {\"true score\": 1, \"predicted score\": 0.650105694219542}, {\"true score\": 1, \"predicted score\": 0.7672070091432986}, {\"true score\": 0, \"predicted score\": 0.5480581977397716}, {\"true score\": 0, \"predicted score\": 0.9167040806042892}, {\"true score\": 0, \"predicted score\": 1.2207157952244252}, {\"true score\": 1, \"predicted score\": 0.4482032128537181}, {\"true score\": 0, \"predicted score\": 0.4117838279216939}, {\"true score\": 1, \"predicted score\": 0.2630624575067449}, {\"true score\": 2, \"predicted score\": 0.8617524177577125}, {\"true score\": 2, \"predicted score\": 0.9969552112752089}, {\"true score\": 1, \"predicted score\": 0.583832650377389}, {\"true score\": 0, \"predicted score\": 0.8199504107291464}, {\"true score\": 0, \"predicted score\": 0.4780546601924214}, {\"true score\": 0, \"predicted score\": 0.010105376597438709}, {\"true score\": 1, \"predicted score\": 0.762186871436846}, {\"true score\": 0, \"predicted score\": 0.32700168681693026}, {\"true score\": 1, \"predicted score\": 0.6847288511475789}, {\"true score\": 0, \"predicted score\": 0.6219035067796452}, {\"true score\": 0, \"predicted score\": 0.6979268162233272}, {\"true score\": 0, \"predicted score\": 0.40035952615827336}, {\"true score\": 2, \"predicted score\": 0.5800584810432904}, {\"true score\": 0, \"predicted score\": 0.22783124110686137}, {\"true score\": 0, \"predicted score\": 0.8132814472153018}, {\"true score\": 0, \"predicted score\": 0.6909415892690935}, {\"true score\": 1, \"predicted score\": 0.4116450347471855}, {\"true score\": 0, \"predicted score\": 0.656057135146041}, {\"true score\": 0, \"predicted score\": 0.12406839789614676}, {\"true score\": 0, \"predicted score\": 0.3806527004434147}, {\"true score\": 1, \"predicted score\": 0.7922688789374199}, {\"true score\": 0, \"predicted score\": 0.38960138510944253}, {\"true score\": 1, \"predicted score\": 1.5824221736586783}, {\"true score\": 3, \"predicted score\": 0.7650810192232349}, {\"true score\": 1, \"predicted score\": 0.4500034901950495}, {\"true score\": 1, \"predicted score\": 0.07748243234846874}, {\"true score\": 0, \"predicted score\": 0.5142816368481058}, {\"true score\": 0, \"predicted score\": -0.011033737892128692}, {\"true score\": 1, \"predicted score\": 0.9346261529652753}, {\"true score\": 1, \"predicted score\": 0.640811698373573}, {\"true score\": 2, \"predicted score\": 0.9304038288689562}, {\"true score\": 1, \"predicted score\": 0.9546489981126478}, {\"true score\": 2, \"predicted score\": 0.5366959972965911}, {\"true score\": 0, \"predicted score\": 0.7996086931445172}, {\"true score\": 0, \"predicted score\": 0.46732461151664095}, {\"true score\": 2, \"predicted score\": 0.8973493328701607}, {\"true score\": 0, \"predicted score\": 0.3428104306754062}, {\"true score\": 1, \"predicted score\": 0.776542699236776}, {\"true score\": 0, \"predicted score\": 0.6037229843025947}, {\"true score\": 1, \"predicted score\": 0.8199199650328852}, {\"true score\": 1, \"predicted score\": 1.0578117674980385}, {\"true score\": 0, \"predicted score\": 0.614468995543466}, {\"true score\": 0, \"predicted score\": 0.46955460655219206}, {\"true score\": 1, \"predicted score\": 0.8659813037940234}, {\"true score\": 0, \"predicted score\": 0.5540448262765078}, {\"true score\": 2, \"predicted score\": 1.3444275838364974}, {\"true score\": 2, \"predicted score\": 0.775881188369192}, {\"true score\": 0, \"predicted score\": 1.097248392555303}, {\"true score\": 1, \"predicted score\": 1.1032106150163872}, {\"true score\": 0, \"predicted score\": 0.6021307780416133}, {\"true score\": 1, \"predicted score\": 1.0120698643317665}, {\"true score\": 0, \"predicted score\": 0.797111405143341}, {\"true score\": 0, \"predicted score\": 0.4283568973661477}, {\"true score\": 1, \"predicted score\": 0.671377522377835}, {\"true score\": 0, \"predicted score\": 0.44691389735361964}, {\"true score\": 0, \"predicted score\": 1.1317295132057286}, {\"true score\": 1, \"predicted score\": 0.9769642995960808}, {\"true score\": 0, \"predicted score\": 0.5333190908210558}, {\"true score\": 1, \"predicted score\": 0.9138215461202854}, {\"true score\": 0, \"predicted score\": 0.866502244487134}, {\"true score\": 2, \"predicted score\": 0.8684959013407526}, {\"true score\": 0, \"predicted score\": 0.6202349765994463}, {\"true score\": 0, \"predicted score\": 0.6076020538626752}, {\"true score\": 0, \"predicted score\": 0.8894914165339256}, {\"true score\": 1, \"predicted score\": 0.9290362772389074}, {\"true score\": 0, \"predicted score\": 1.3067565866762807}, {\"true score\": 0, \"predicted score\": 0.7451706804558663}, {\"true score\": 1, \"predicted score\": 0.8779142270135527}, {\"true score\": 2, \"predicted score\": 1.7274467274293515}, {\"true score\": 0, \"predicted score\": 0.6129529793593852}, {\"true score\": 0, \"predicted score\": 1.0526506072678274}, {\"true score\": 0, \"predicted score\": 0.8353957953308355}, {\"true score\": 0, \"predicted score\": 0.7637317199544396}, {\"true score\": 1, \"predicted score\": 0.8517766724230711}, {\"true score\": 1, \"predicted score\": 0.5488043521690368}, {\"true score\": 0, \"predicted score\": 0.7131163256898385}, {\"true score\": 1, \"predicted score\": 0.4275397939160779}, {\"true score\": 0, \"predicted score\": 0.7243475114936231}, {\"true score\": 1, \"predicted score\": 1.1065787324853567}, {\"true score\": 1, \"predicted score\": 0.5655479667047469}, {\"true score\": 0, \"predicted score\": 0.5806986443658524}, {\"true score\": 1, \"predicted score\": 0.8649387732461002}, {\"true score\": 0, \"predicted score\": 0.29209035115680776}, {\"true score\": 0, \"predicted score\": 0.3306632381857305}, {\"true score\": 1, \"predicted score\": 0.8318900378836998}, {\"true score\": 0, \"predicted score\": 0.5615616024202916}, {\"true score\": 0, \"predicted score\": 0.9563296303813962}, {\"true score\": 0, \"predicted score\": 0.9728521433920885}, {\"true score\": 2, \"predicted score\": 0.7911303382100157}, {\"true score\": 0, \"predicted score\": 0.32692089463897644}, {\"true score\": 1, \"predicted score\": 1.060437156732473}, {\"true score\": 1, \"predicted score\": 0.9689050209589326}, {\"true score\": 2, \"predicted score\": 0.9172198703276571}, {\"true score\": 1, \"predicted score\": 0.5901080301091406}, {\"true score\": 0, \"predicted score\": 0.8848270998009318}, {\"true score\": 2, \"predicted score\": 1.0714714407101362}, {\"true score\": 0, \"predicted score\": 0.24310906100030777}, {\"true score\": 0, \"predicted score\": 0.45886687621667366}, {\"true score\": 0, \"predicted score\": 0.2511098903784657}, {\"true score\": 0, \"predicted score\": 0.5102843339462334}, {\"true score\": 1, \"predicted score\": 1.0762548562144914}, {\"true score\": 0, \"predicted score\": 1.0788911993705013}, {\"true score\": 2, \"predicted score\": 1.0448263213414868}, {\"true score\": 0, \"predicted score\": 0.9898746032330534}, {\"true score\": 2, \"predicted score\": 1.3006753079910767}, {\"true score\": 0, \"predicted score\": 0.6191717413790682}, {\"true score\": 1, \"predicted score\": 1.0719799333435875}, {\"true score\": 0, \"predicted score\": 0.4491014124681245}, {\"true score\": 1, \"predicted score\": 0.8826830893641731}, {\"true score\": 0, \"predicted score\": 0.7500385682904231}, {\"true score\": 0, \"predicted score\": 0.5594449669111882}, {\"true score\": 1, \"predicted score\": 1.050819018971661}, {\"true score\": 1, \"predicted score\": 1.0926533579173192}, {\"true score\": 2, \"predicted score\": 0.7803735645087272}, {\"true score\": 1, \"predicted score\": 1.0706855615296231}, {\"true score\": 0, \"predicted score\": 0.958157772574811}, {\"true score\": 1, \"predicted score\": 0.9568263880356629}, {\"true score\": 0, \"predicted score\": 2.607990896619768}, {\"true score\": 0, \"predicted score\": 0.7593593778373859}, {\"true score\": 2, \"predicted score\": 0.4503857851265325}, {\"true score\": 2, \"predicted score\": 0.7687277847063947}, {\"true score\": 0, \"predicted score\": 0.4706885394893812}, {\"true score\": 0, \"predicted score\": 0.8927571746109625}, {\"true score\": 0, \"predicted score\": 0.9271274305445248}, {\"true score\": 0, \"predicted score\": 0.5947994347204547}, {\"true score\": 0, \"predicted score\": 0.3461683022967914}, {\"true score\": 0, \"predicted score\": 0.5615162892780792}, {\"true score\": 2, \"predicted score\": 0.5306283590098132}, {\"true score\": 1, \"predicted score\": 0.4849333292579967}, {\"true score\": 2, \"predicted score\": 0.4708283694575045}, {\"true score\": 0, \"predicted score\": 0.32598674278399137}, {\"true score\": 0, \"predicted score\": 0.20071960178658915}, {\"true score\": 3, \"predicted score\": 0.5019052239012572}, {\"true score\": 0, \"predicted score\": 0.9534103205808048}, {\"true score\": 1, \"predicted score\": 0.7542975514167954}, {\"true score\": 1, \"predicted score\": 0.7804451716204641}, {\"true score\": 0, \"predicted score\": 0.23815897513339845}, {\"true score\": 0, \"predicted score\": 0.10317080654467181}, {\"true score\": 0, \"predicted score\": 0.6921894606338197}, {\"true score\": 0, \"predicted score\": 0.9423262960192402}, {\"true score\": 0, \"predicted score\": 1.191785060500953}, {\"true score\": 2, \"predicted score\": 0.5871941821710782}, {\"true score\": 0, \"predicted score\": 0.8034217330999069}, {\"true score\": 2, \"predicted score\": 0.8178403148563533}, {\"true score\": 0, \"predicted score\": 1.0606160951876302}, {\"true score\": 0, \"predicted score\": 0.20136496508890467}, {\"true score\": 2, \"predicted score\": 0.8044738101215925}, {\"true score\": 1, \"predicted score\": 0.6286482759449729}, {\"true score\": 0, \"predicted score\": 0.4229224723940936}, {\"true score\": 0, \"predicted score\": 0.43197579069513375}, {\"true score\": 0, \"predicted score\": 0.5150975777963546}, {\"true score\": 0, \"predicted score\": 0.8665358284220119}, {\"true score\": 0, \"predicted score\": 0.37205084279813383}, {\"true score\": 2, \"predicted score\": 0.8687957725547224}, {\"true score\": 0, \"predicted score\": 0.2021496813644968}, {\"true score\": 0, \"predicted score\": 0.26316261659410717}, {\"true score\": 1, \"predicted score\": 0.7660011059173354}, {\"true score\": 0, \"predicted score\": 0.286581076158684}, {\"true score\": 1, \"predicted score\": 0.493622953017557}, {\"true score\": 0, \"predicted score\": 0.7983185904649166}, {\"true score\": 1, \"predicted score\": 0.9792138784735362}, {\"true score\": 1, \"predicted score\": 0.90579621454581}, {\"true score\": 2, \"predicted score\": 0.4981042066750967}, {\"true score\": 0, \"predicted score\": 0.5176602552641358}, {\"true score\": 0, \"predicted score\": 0.7545498201722995}, {\"true score\": 0, \"predicted score\": 0.48929585968958733}, {\"true score\": 2, \"predicted score\": 0.6958859666436964}, {\"true score\": 0, \"predicted score\": 0.015616364752371159}, {\"true score\": 1, \"predicted score\": 0.5768682294348015}, {\"true score\": 0, \"predicted score\": 0.5597345270032852}, {\"true score\": 1, \"predicted score\": 0.404786302092051}, {\"true score\": 0, \"predicted score\": 0.8213858803946509}, {\"true score\": 2, \"predicted score\": 0.7969198644127214}, {\"true score\": 0, \"predicted score\": 0.42035301011888093}, {\"true score\": 0, \"predicted score\": 0.5984016882508616}, {\"true score\": 0, \"predicted score\": -0.4615189588950809}, {\"true score\": 0, \"predicted score\": 0.8927586958083951}, {\"true score\": 1, \"predicted score\": 0.6001711191469539}, {\"true score\": 1, \"predicted score\": 0.48385864116344124}, {\"true score\": 0, \"predicted score\": 0.16437048290898976}, {\"true score\": 0, \"predicted score\": 0.6623265822095393}, {\"true score\": 1, \"predicted score\": 1.3620642074535578}, {\"true score\": 1, \"predicted score\": 0.8030997001429274}, {\"true score\": 0, \"predicted score\": 0.700255994072595}, {\"true score\": 0, \"predicted score\": 0.5980329316838798}, {\"true score\": 0, \"predicted score\": 0.3317750226907307}, {\"true score\": 2, \"predicted score\": 0.7527930376822454}, {\"true score\": 1, \"predicted score\": 0.6168937159743664}, {\"true score\": 1, \"predicted score\": 0.44980763932453915}, {\"true score\": 1, \"predicted score\": 0.8065485441038637}, {\"true score\": 1, \"predicted score\": 0.5263995941571266}, {\"true score\": 0, \"predicted score\": 0.5518825312487112}, {\"true score\": 0, \"predicted score\": 0.16392196202616138}, {\"true score\": 0, \"predicted score\": 0.651369428552457}, {\"true score\": 1, \"predicted score\": 1.0913484048178705}, {\"true score\": 0, \"predicted score\": 0.4278372538251485}, {\"true score\": 1, \"predicted score\": 0.20841070172936377}, {\"true score\": 1, \"predicted score\": 0.9806053783165958}, {\"true score\": 0, \"predicted score\": 0.3626474349481905}, {\"true score\": 0, \"predicted score\": 0.8956119266935085}, {\"true score\": 1, \"predicted score\": 0.941664739691748}, {\"true score\": 0, \"predicted score\": 1.0772071973853494}, {\"true score\": 1, \"predicted score\": 0.5717505454373986}, {\"true score\": 0, \"predicted score\": 0.44360761457894293}, {\"true score\": 0, \"predicted score\": 0.5368053304546371}, {\"true score\": 1, \"predicted score\": 0.8180939988066197}, {\"true score\": 2, \"predicted score\": 1.0721310130179285}, {\"true score\": 1, \"predicted score\": 0.3934819743561957}, {\"true score\": 4, \"predicted score\": 0.78243689084624}, {\"true score\": 1, \"predicted score\": 0.4506922886410767}, {\"true score\": 0, \"predicted score\": 0.6800519080370563}, {\"true score\": 2, \"predicted score\": 0.8350969434960409}, {\"true score\": 0, \"predicted score\": 1.0381471650773129}, {\"true score\": 1, \"predicted score\": 0.2640982482334616}, {\"true score\": 1, \"predicted score\": 0.5825402570600655}, {\"true score\": 2, \"predicted score\": 0.8530600113360229}, {\"true score\": 0, \"predicted score\": 0.872625313015478}, {\"true score\": 0, \"predicted score\": 0.4151615017167681}, {\"true score\": 0, \"predicted score\": 1.5585386319806562}, {\"true score\": 0, \"predicted score\": 0.728652135662055}, {\"true score\": 0, \"predicted score\": 0.40577694830616934}, {\"true score\": 0, \"predicted score\": 0.36784147503313647}, {\"true score\": 1, \"predicted score\": 0.7837992779378641}, {\"true score\": 2, \"predicted score\": 1.0543135007447555}, {\"true score\": 1, \"predicted score\": 0.8082958158367282}, {\"true score\": 1, \"predicted score\": 1.029027863119767}, {\"true score\": 2, \"predicted score\": 1.1719551724220376}, {\"true score\": 1, \"predicted score\": 0.5915410251707415}, {\"true score\": 0, \"predicted score\": 0.3911994900368807}, {\"true score\": 0, \"predicted score\": 0.10855625626032528}, {\"true score\": 1, \"predicted score\": 0.4560421378634879}, {\"true score\": 0, \"predicted score\": 1.0064842351482093}, {\"true score\": 1, \"predicted score\": 0.5642746507969327}, {\"true score\": 0, \"predicted score\": 0.4716677493410465}, {\"true score\": 0, \"predicted score\": 1.1834637719953753}, {\"true score\": 1, \"predicted score\": 0.7440270890956369}, {\"true score\": 0, \"predicted score\": 0.6397928178285629}, {\"true score\": 0, \"predicted score\": 0.5144160033626235}, {\"true score\": 2, \"predicted score\": 0.9484058726393503}, {\"true score\": 0, \"predicted score\": 0.6450563466421735}, {\"true score\": 1, \"predicted score\": 0.7888707252162124}, {\"true score\": 1, \"predicted score\": 0.5124759572692514}, {\"true score\": 1, \"predicted score\": 0.9498105816047715}, {\"true score\": 0, \"predicted score\": 0.7488598789253035}, {\"true score\": 0, \"predicted score\": 0.441376581777826}, {\"true score\": 0, \"predicted score\": 0.7006685227136835}, {\"true score\": 1, \"predicted score\": 0.5286231927573987}, {\"true score\": 0, \"predicted score\": 0.5418605902760516}, {\"true score\": 0, \"predicted score\": 0.8911733561568806}, {\"true score\": 2, \"predicted score\": 0.4316801040170608}, {\"true score\": 2, \"predicted score\": 0.6776979014136933}, {\"true score\": 2, \"predicted score\": 1.0145803720518969}, {\"true score\": 2, \"predicted score\": 0.7463597039906756}, {\"true score\": 0, \"predicted score\": 0.9401418326742028}, {\"true score\": 4, \"predicted score\": 0.7142385683933012}, {\"true score\": 0, \"predicted score\": 0.7930835422268395}, {\"true score\": 1, \"predicted score\": 0.8671984085961146}, {\"true score\": 2, \"predicted score\": 0.7114159828649289}, {\"true score\": 1, \"predicted score\": 1.1753308430690173}, {\"true score\": 2, \"predicted score\": 0.9233751542985892}, {\"true score\": 0, \"predicted score\": 0.6951992460987308}, {\"true score\": 0, \"predicted score\": 1.135258303174267}, {\"true score\": 2, \"predicted score\": 1.336045417059736}, {\"true score\": 0, \"predicted score\": 0.24056096779459246}, {\"true score\": 0, \"predicted score\": 0.12686833252188817}, {\"true score\": 0, \"predicted score\": 0.5812605606001624}, {\"true score\": 1, \"predicted score\": 1.4211591066275122}, {\"true score\": 1, \"predicted score\": 0.7299662252383797}, {\"true score\": 4, \"predicted score\": 0.34974687316083863}, {\"true score\": 0, \"predicted score\": 0.3403732506765375}, {\"true score\": 0, \"predicted score\": 0.2763317205037728}, {\"true score\": 1, \"predicted score\": 0.9338393889253767}, {\"true score\": 1, \"predicted score\": 0.6436775418572304}, {\"true score\": 0, \"predicted score\": 0.5664832633215652}, {\"true score\": 1, \"predicted score\": 0.5461122286151174}, {\"true score\": 2, \"predicted score\": 1.0114077011644287}, {\"true score\": 0, \"predicted score\": 0.5219318025021079}, {\"true score\": 0, \"predicted score\": 0.8677678046754489}, {\"true score\": 1, \"predicted score\": 0.9651667935770593}, {\"true score\": 1, \"predicted score\": 0.7317022085242864}, {\"true score\": 1, \"predicted score\": 0.7936565803003854}, {\"true score\": 0, \"predicted score\": 0.48739049205343365}, {\"true score\": 2, \"predicted score\": 1.0909287256067066}, {\"true score\": 0, \"predicted score\": 0.5567729959150979}, {\"true score\": 2, \"predicted score\": 0.7794965079221825}, {\"true score\": 0, \"predicted score\": 0.6629376527010442}, {\"true score\": 0, \"predicted score\": 0.5058965670870902}, {\"true score\": 2, \"predicted score\": 0.8726809522884005}, {\"true score\": 0, \"predicted score\": 0.6092542216385154}, {\"true score\": 0, \"predicted score\": 0.850118062048411}, {\"true score\": 0, \"predicted score\": 0.4196046532609869}, {\"true score\": 1, \"predicted score\": 0.7821915695750968}, {\"true score\": 0, \"predicted score\": 0.6510992792268644}, {\"true score\": 0, \"predicted score\": 0.9845321745936808}, {\"true score\": 1, \"predicted score\": 0.7531602360937614}, {\"true score\": 0, \"predicted score\": 0.24569602946256786}, {\"true score\": 0, \"predicted score\": 0.35114854411505847}, {\"true score\": 1, \"predicted score\": 1.0344474012515423}, {\"true score\": 1, \"predicted score\": 0.7753042468861575}, {\"true score\": 1, \"predicted score\": 1.0509792360399155}, {\"true score\": 3, \"predicted score\": 0.10492045029988661}, {\"true score\": 1, \"predicted score\": 0.6374412867242271}, {\"true score\": 0, \"predicted score\": 0.9219042186944965}, {\"true score\": 1, \"predicted score\": 0.8108301718967234}, {\"true score\": 1, \"predicted score\": 0.9362746920112395}, {\"true score\": 1, \"predicted score\": 0.7178358089999236}, {\"true score\": 2, \"predicted score\": 1.1354486051815988}, {\"true score\": 0, \"predicted score\": 0.23434312987633654}, {\"true score\": 0, \"predicted score\": 0.49030513393918584}, {\"true score\": 0, \"predicted score\": 0.5286812727604735}, {\"true score\": 0, \"predicted score\": 0.5292182199972293}, {\"true score\": 0, \"predicted score\": 1.0514093443886745}, {\"true score\": 0, \"predicted score\": 1.0439539188823583}, {\"true score\": 0, \"predicted score\": 0.25596935674744725}, {\"true score\": 0, \"predicted score\": 0.11484387021723691}, {\"true score\": 0, \"predicted score\": 0.792541672505884}, {\"true score\": 1, \"predicted score\": 0.6842976678247287}, {\"true score\": 2, \"predicted score\": 0.9361664871947453}, {\"true score\": 1, \"predicted score\": 0.9385985529470942}, {\"true score\": 0, \"predicted score\": 0.41558837027470474}, {\"true score\": 0, \"predicted score\": 0.5925932668274233}, {\"true score\": 1, \"predicted score\": 0.7590132496817665}, {\"true score\": 2, \"predicted score\": 1.086517811535411}, {\"true score\": 0, \"predicted score\": 1.1000815616817672}, {\"true score\": 1, \"predicted score\": 0.573403606985617}, {\"true score\": 1, \"predicted score\": 0.8514727601093532}, {\"true score\": 1, \"predicted score\": 0.49799340638550954}, {\"true score\": 0, \"predicted score\": 1.138537953828651}, {\"true score\": 1, \"predicted score\": 0.9507228718376666}, {\"true score\": 1, \"predicted score\": 0.7802219734929372}, {\"true score\": 0, \"predicted score\": 0.4889781061869037}, {\"true score\": 1, \"predicted score\": 1.0612583936931435}, {\"true score\": 1, \"predicted score\": 0.19372785633568657}, {\"true score\": 1, \"predicted score\": 0.6025551739682238}, {\"true score\": 0, \"predicted score\": 0.5964880117584647}, {\"true score\": 0, \"predicted score\": 0.7796306668312072}, {\"true score\": 0, \"predicted score\": 0.47566945298177493}, {\"true score\": 0, \"predicted score\": 0.6956373651348883}, {\"true score\": 1, \"predicted score\": 0.9747716407606415}, {\"true score\": 0, \"predicted score\": 0.5128630662200283}, {\"true score\": 2, \"predicted score\": 0.9917512695316041}, {\"true score\": 0, \"predicted score\": 0.4435722618228157}, {\"true score\": 1, \"predicted score\": 0.3851536004926204}, {\"true score\": 1, \"predicted score\": 0.8901771619317109}, {\"true score\": 0, \"predicted score\": 0.23436710584359044}, {\"true score\": 1, \"predicted score\": 0.39543632391856176}, {\"true score\": 0, \"predicted score\": 0.7152963150803608}, {\"true score\": 1, \"predicted score\": 0.6639151329204324}, {\"true score\": 0, \"predicted score\": 0.6145474354974652}, {\"true score\": 1, \"predicted score\": 0.34672206627692287}, {\"true score\": 1, \"predicted score\": 0.875546803099498}, {\"true score\": 2, \"predicted score\": 1.3655261991003056}, {\"true score\": 1, \"predicted score\": 0.7225908013033193}, {\"true score\": 2, \"predicted score\": 0.9422539685022306}, {\"true score\": 0, \"predicted score\": 0.5908850055023369}, {\"true score\": 1, \"predicted score\": 0.8987697754999844}, {\"true score\": 0, \"predicted score\": 1.0510075732851405}, {\"true score\": 2, \"predicted score\": 1.1603291136048153}, {\"true score\": 2, \"predicted score\": 0.9750388334583964}, {\"true score\": 0, \"predicted score\": 0.4505949910441382}, {\"true score\": 1, \"predicted score\": 0.8398300430719325}, {\"true score\": 1, \"predicted score\": 0.6705965551672243}, {\"true score\": 0, \"predicted score\": -0.1608062302421892}, {\"true score\": 2, \"predicted score\": 0.9235705956778408}, {\"true score\": 0, \"predicted score\": 0.40728341784981836}, {\"true score\": 1, \"predicted score\": 0.5231052464460993}, {\"true score\": 3, \"predicted score\": 1.0100024867296657}, {\"true score\": 0, \"predicted score\": 0.7297025447123173}, {\"true score\": 0, \"predicted score\": -0.1216160692587811}, {\"true score\": 2, \"predicted score\": 0.5471719160783047}, {\"true score\": 0, \"predicted score\": 0.16724257575338664}, {\"true score\": 1, \"predicted score\": 0.9938333457426887}, {\"true score\": 0, \"predicted score\": -0.1591975587381529}, {\"true score\": 0, \"predicted score\": 0.5856782560645971}, {\"true score\": 0, \"predicted score\": -0.254469332920392}, {\"true score\": 2, \"predicted score\": 0.558249835304326}, {\"true score\": 2, \"predicted score\": 0.478281480133225}, {\"true score\": 1, \"predicted score\": 0.507242772974212}, {\"true score\": 1, \"predicted score\": 0.1714312570597113}, {\"true score\": 0, \"predicted score\": 0.42735389702198345}, {\"true score\": 0, \"predicted score\": 0.27931598279201253}, {\"true score\": 0, \"predicted score\": 0.622167125443062}, {\"true score\": 0, \"predicted score\": 1.2807017847666073}, {\"true score\": 1, \"predicted score\": 0.5657112005937159}, {\"true score\": 2, \"predicted score\": 0.903258793765972}, {\"true score\": 0, \"predicted score\": 0.8079099895312716}, {\"true score\": 1, \"predicted score\": 1.0466567897614747}, {\"true score\": 0, \"predicted score\": 0.6155253983300675}, {\"true score\": 4, \"predicted score\": 1.0036075330912044}, {\"true score\": 1, \"predicted score\": 0.7119935715304431}, {\"true score\": 0, \"predicted score\": 0.3975181450175138}, {\"true score\": 3, \"predicted score\": 0.8560580160075284}, {\"true score\": 0, \"predicted score\": 0.17909704161136414}, {\"true score\": 1, \"predicted score\": 1.1038662283709924}, {\"true score\": 0, \"predicted score\": 0.3867549484605202}, {\"true score\": 1, \"predicted score\": 0.9331905238035068}, {\"true score\": 0, \"predicted score\": 0.725048626930312}, {\"true score\": 0, \"predicted score\": 0.5342263714432924}, {\"true score\": 0, \"predicted score\": 0.3362931076894064}, {\"true score\": 1, \"predicted score\": 0.7336669993197429}, {\"true score\": 2, \"predicted score\": 0.37366318597884973}, {\"true score\": 0, \"predicted score\": 0.007323205780644271}, {\"true score\": 0, \"predicted score\": 0.886617468516194}, {\"true score\": 1, \"predicted score\": 0.6747526841715248}, {\"true score\": 1, \"predicted score\": 0.629797101727323}, {\"true score\": 0, \"predicted score\": 0.6126078969979699}, {\"true score\": 0, \"predicted score\": 0.2616326207536278}, {\"true score\": 2, \"predicted score\": 0.7191219267622585}, {\"true score\": 0, \"predicted score\": 0.7865722771801806}, {\"true score\": 0, \"predicted score\": 0.03818926038341597}, {\"true score\": 1, \"predicted score\": 0.34294946373874907}, {\"true score\": 0, \"predicted score\": 0.39683967604820336}, {\"true score\": 0, \"predicted score\": 0.22298909298440545}, {\"true score\": 0, \"predicted score\": 0.6958734487908039}, {\"true score\": 1, \"predicted score\": 0.6260147932487344}, {\"true score\": 0, \"predicted score\": 1.0210825369740943}, {\"true score\": 0, \"predicted score\": 0.4221950120312968}, {\"true score\": 1, \"predicted score\": 0.5550984196526663}, {\"true score\": 2, \"predicted score\": 0.3794014110642894}, {\"true score\": 1, \"predicted score\": 0.45247260210687285}, {\"true score\": 0, \"predicted score\": 0.327892423384261}, {\"true score\": 1, \"predicted score\": 0.2565114383856801}, {\"true score\": 0, \"predicted score\": 0.3735859516078366}, {\"true score\": 1, \"predicted score\": 0.44669463091354134}, {\"true score\": 0, \"predicted score\": 0.6609936588000518}, {\"true score\": 0, \"predicted score\": 0.510319696996041}, {\"true score\": 2, \"predicted score\": 0.7277188567786401}, {\"true score\": 0, \"predicted score\": 0.43830188558096317}, {\"true score\": 1, \"predicted score\": 1.059394893929658}, {\"true score\": 1, \"predicted score\": 0.9549307990774853}, {\"true score\": 1, \"predicted score\": 0.4705959321298342}, {\"true score\": 0, \"predicted score\": 0.5189890980737141}, {\"true score\": 0, \"predicted score\": 0.2510798096051408}, {\"true score\": 0, \"predicted score\": 0.7337750990353644}, {\"true score\": 2, \"predicted score\": 1.374752920382802}, {\"true score\": 0, \"predicted score\": 0.6114386274858222}, {\"true score\": 1, \"predicted score\": 0.7308579865192982}, {\"true score\": 0, \"predicted score\": 0.44836493328411187}, {\"true score\": 0, \"predicted score\": 0.5510526778828622}, {\"true score\": 2, \"predicted score\": 1.0604366544533979}, {\"true score\": 0, \"predicted score\": -0.2586564316378539}, {\"true score\": 0, \"predicted score\": 0.5410360997513777}, {\"true score\": 1, \"predicted score\": 0.7540815544583586}, {\"true score\": 1, \"predicted score\": 0.5534208413151313}, {\"true score\": 2, \"predicted score\": 1.2777763470175887}, {\"true score\": 2, \"predicted score\": 0.8879072411385815}, {\"true score\": 1, \"predicted score\": 0.7176756378858099}, {\"true score\": 1, \"predicted score\": 0.8532143917684558}, {\"true score\": 0, \"predicted score\": 1.2166090882353076}, {\"true score\": 1, \"predicted score\": 0.9378099979620014}, {\"true score\": 0, \"predicted score\": 0.45387776225028365}, {\"true score\": 1, \"predicted score\": 0.7929759507890086}, {\"true score\": 2, \"predicted score\": 1.3070495110305218}, {\"true score\": 0, \"predicted score\": 0.483600945213492}, {\"true score\": 0, \"predicted score\": 0.68465333822811}, {\"true score\": 0, \"predicted score\": 1.1100644805703526}, {\"true score\": 0, \"predicted score\": 0.6218433409004425}, {\"true score\": 1, \"predicted score\": 0.5232001607381734}, {\"true score\": 0, \"predicted score\": 0.31944745417828774}, {\"true score\": 0, \"predicted score\": 0.646058861593448}, {\"true score\": 1, \"predicted score\": 0.6026119830915414}, {\"true score\": 0, \"predicted score\": 0.763855211167402}, {\"true score\": 0, \"predicted score\": 0.48147244476027173}, {\"true score\": 2, \"predicted score\": 0.8908541886389287}, {\"true score\": 3, \"predicted score\": 1.0074869962403776}, {\"true score\": 0, \"predicted score\": 0.9168199055861594}, {\"true score\": 0, \"predicted score\": 0.2541553583477977}, {\"true score\": 3, \"predicted score\": 1.2033763830114164}, {\"true score\": 2, \"predicted score\": 0.7241359592032952}, {\"true score\": 0, \"predicted score\": 0.5145305191587426}, {\"true score\": 0, \"predicted score\": 0.7143213223391558}, {\"true score\": 1, \"predicted score\": 1.063937309018147}, {\"true score\": 0, \"predicted score\": 0.5110736501293903}, {\"true score\": 0, \"predicted score\": 0.6832414924995028}, {\"true score\": 0, \"predicted score\": 0.028963665709010433}, {\"true score\": 0, \"predicted score\": 0.790766354046285}, {\"true score\": 2, \"predicted score\": 0.6263931989870768}, {\"true score\": 2, \"predicted score\": 0.45957658070496454}, {\"true score\": 0, \"predicted score\": 1.0300226478342434}, {\"true score\": 1, \"predicted score\": 0.8601239273501756}, {\"true score\": 0, \"predicted score\": 1.2187416370301751}, {\"true score\": 1, \"predicted score\": 0.919381015905852}, {\"true score\": 0, \"predicted score\": 0.019637023620761473}, {\"true score\": 1, \"predicted score\": 0.5862139413421232}, {\"true score\": 1, \"predicted score\": 0.9967196258578517}, {\"true score\": 0, \"predicted score\": 0.04153446889073931}, {\"true score\": 0, \"predicted score\": 1.3153126472817522}, {\"true score\": 0, \"predicted score\": 0.6976980426573496}, {\"true score\": 1, \"predicted score\": 0.5327698095463682}, {\"true score\": 1, \"predicted score\": 0.9968775005626929}, {\"true score\": 1, \"predicted score\": 1.1182191769954621}, {\"true score\": 0, \"predicted score\": 1.0573447604153845}, {\"true score\": 1, \"predicted score\": 0.46136552046886903}, {\"true score\": 1, \"predicted score\": 0.32187089986703027}, {\"true score\": 1, \"predicted score\": 0.9201124117003535}, {\"true score\": 1, \"predicted score\": 1.0124635329172575}, {\"true score\": 0, \"predicted score\": 0.4586178450570514}, {\"true score\": 1, \"predicted score\": 0.49144438812155955}, {\"true score\": 2, \"predicted score\": 0.7387068114279808}, {\"true score\": 1, \"predicted score\": 0.30125056438959585}, {\"true score\": 1, \"predicted score\": 1.059439572928931}, {\"true score\": 0, \"predicted score\": 0.7965953992507476}, {\"true score\": 0, \"predicted score\": 0.7195501892888044}, {\"true score\": 0, \"predicted score\": 0.7714363348799258}, {\"true score\": 2, \"predicted score\": 1.1223929126644405}, {\"true score\": 2, \"predicted score\": 0.4978422877662035}, {\"true score\": 0, \"predicted score\": 0.6291761651064998}, {\"true score\": 0, \"predicted score\": 0.6048572626249578}, {\"true score\": 0, \"predicted score\": 0.26485061694659084}, {\"true score\": 0, \"predicted score\": 0.38777564671390224}, {\"true score\": 0, \"predicted score\": 0.7890720249027856}, {\"true score\": 2, \"predicted score\": 0.33053582769267065}, {\"true score\": 1, \"predicted score\": 0.8573522699033769}, {\"true score\": 4, \"predicted score\": 1.2165093376116651}, {\"true score\": 0, \"predicted score\": 0.6265487987400822}, {\"true score\": 0, \"predicted score\": -0.15326810428699944}, {\"true score\": 0, \"predicted score\": 0.33155108993427707}, {\"true score\": 0, \"predicted score\": 0.6666646187696161}, {\"true score\": 1, \"predicted score\": 0.38554615133819786}, {\"true score\": 2, \"predicted score\": 1.4969684076162295}, {\"true score\": 3, \"predicted score\": 1.1982643782202025}, {\"true score\": 1, \"predicted score\": 0.9902144620254549}, {\"true score\": 2, \"predicted score\": 0.5756176010712148}, {\"true score\": 1, \"predicted score\": 0.939835498154673}, {\"true score\": 4, \"predicted score\": 0.9778767168567291}, {\"true score\": 2, \"predicted score\": 1.0181110649911032}, {\"true score\": 1, \"predicted score\": 0.9579984560625447}, {\"true score\": 0, \"predicted score\": 0.901492572809609}, {\"true score\": 1, \"predicted score\": 1.1389719016177284}, {\"true score\": 1, \"predicted score\": 0.9958709899179997}, {\"true score\": 1, \"predicted score\": 0.40132251241162037}, {\"true score\": 0, \"predicted score\": 0.15684016315608834}, {\"true score\": 1, \"predicted score\": 1.0471220707769104}, {\"true score\": 1, \"predicted score\": 0.598563387201181}, {\"true score\": 1, \"predicted score\": 0.7706652831816584}, {\"true score\": 1, \"predicted score\": 0.9160225403909422}, {\"true score\": 0, \"predicted score\": 1.0557317817834697}, {\"true score\": 0, \"predicted score\": 0.5560279607538873}, {\"true score\": 0, \"predicted score\": 0.6924364493771742}, {\"true score\": 0, \"predicted score\": 0.4663793614218579}, {\"true score\": 1, \"predicted score\": 0.8401613125607277}, {\"true score\": 1, \"predicted score\": 1.1169615051431916}, {\"true score\": 0, \"predicted score\": 0.6585678044034438}, {\"true score\": 1, \"predicted score\": 1.0447959157464999}, {\"true score\": 0, \"predicted score\": 1.213325095024576}, {\"true score\": 0, \"predicted score\": 0.2930243646432056}, {\"true score\": 1, \"predicted score\": 0.9952443783111742}, {\"true score\": 1, \"predicted score\": 0.8180772018552888}, {\"true score\": 0, \"predicted score\": 0.7126886101468775}, {\"true score\": 0, \"predicted score\": 0.1660389606745476}, {\"true score\": 1, \"predicted score\": 0.8696046834308366}, {\"true score\": 0, \"predicted score\": 0.7949968947680368}, {\"true score\": 0, \"predicted score\": 0.47815612135131697}, {\"true score\": 0, \"predicted score\": 0.6384130042304685}, {\"true score\": 0, \"predicted score\": 0.580666456751047}, {\"true score\": 0, \"predicted score\": -0.40745061477335043}, {\"true score\": 0, \"predicted score\": 0.2526034921504776}, {\"true score\": 1, \"predicted score\": 0.7521370586457475}, {\"true score\": 0, \"predicted score\": 0.5355941942646262}, {\"true score\": 0, \"predicted score\": 0.9783630688727025}, {\"true score\": 0, \"predicted score\": 0.5944750170112153}, {\"true score\": 2, \"predicted score\": 1.1605434085889998}, {\"true score\": 0, \"predicted score\": 0.9606746850030892}, {\"true score\": 0, \"predicted score\": 0.3827906470969935}, {\"true score\": 0, \"predicted score\": 1.1399095819521272}, {\"true score\": 0, \"predicted score\": 0.638525634944795}, {\"true score\": 1, \"predicted score\": 0.6676642190217507}, {\"true score\": 0, \"predicted score\": 0.4834004493334909}, {\"true score\": 0, \"predicted score\": 0.2041403691273762}, {\"true score\": 1, \"predicted score\": 0.3987993694818577}, {\"true score\": 1, \"predicted score\": 0.9860150774253016}, {\"true score\": 3, \"predicted score\": 0.9866734214694172}, {\"true score\": 2, \"predicted score\": 0.5951415686301054}, {\"true score\": 0, \"predicted score\": 0.5295883560763207}, {\"true score\": 0, \"predicted score\": 0.5161655549431001}, {\"true score\": 0, \"predicted score\": 0.4675348130696849}, {\"true score\": 0, \"predicted score\": 1.2074287177936351}, {\"true score\": 1, \"predicted score\": 1.1002352476186208}, {\"true score\": 0, \"predicted score\": 0.35966164357121727}, {\"true score\": 0, \"predicted score\": 0.9152538396663756}, {\"true score\": 1, \"predicted score\": 0.9459249295553134}, {\"true score\": 0, \"predicted score\": 0.32707176551383293}, {\"true score\": 1, \"predicted score\": 0.9501364099300988}, {\"true score\": 2, \"predicted score\": 0.7742523595915936}, {\"true score\": 2, \"predicted score\": 0.6117511593396628}, {\"true score\": 2, \"predicted score\": 1.4792576741069992}, {\"true score\": 0, \"predicted score\": 0.49218850912367235}, {\"true score\": 1, \"predicted score\": 0.9093622201245317}, {\"true score\": 0, \"predicted score\": 0.21046052132483645}, {\"true score\": 1, \"predicted score\": 1.02237783162319}, {\"true score\": 2, \"predicted score\": 0.5570043287562688}, {\"true score\": 1, \"predicted score\": 0.8663775647049644}, {\"true score\": 1, \"predicted score\": 0.8609733884883479}, {\"true score\": 2, \"predicted score\": 1.3709231441763892}, {\"true score\": 1, \"predicted score\": 0.8448266587957247}, {\"true score\": 2, \"predicted score\": 0.7399374880777327}, {\"true score\": 1, \"predicted score\": 0.595619197223143}, {\"true score\": 1, \"predicted score\": 0.8490963247379604}, {\"true score\": 0, \"predicted score\": 0.6516279894616496}, {\"true score\": 1, \"predicted score\": 0.9112714124135111}, {\"true score\": 1, \"predicted score\": 0.702316098802178}, {\"true score\": 0, \"predicted score\": 0.5281251406251373}, {\"true score\": 1, \"predicted score\": 0.5728415924360716}, {\"true score\": 1, \"predicted score\": 1.0139490446732249}, {\"true score\": 0, \"predicted score\": 0.4355813856225967}, {\"true score\": 1, \"predicted score\": 0.8600219334430849}, {\"true score\": 0, \"predicted score\": 0.4945793103273816}, {\"true score\": 0, \"predicted score\": 0.4393638077901314}, {\"true score\": 0, \"predicted score\": 0.19953021336838206}, {\"true score\": 2, \"predicted score\": 0.9773220712618507}, {\"true score\": 0, \"predicted score\": 0.8453964946731655}, {\"true score\": 0, \"predicted score\": 0.5464842855039942}, {\"true score\": 0, \"predicted score\": 0.5396445837389868}, {\"true score\": 0, \"predicted score\": 0.7215829170168301}, {\"true score\": 0, \"predicted score\": 0.22896643050465998}, {\"true score\": 0, \"predicted score\": 0.4791980851090662}, {\"true score\": 2, \"predicted score\": 0.48255281997455757}, {\"true score\": 1, \"predicted score\": 0.8593765232404725}, {\"true score\": 0, \"predicted score\": 0.5391078444123635}, {\"true score\": 0, \"predicted score\": 0.2701831764280342}, {\"true score\": 1, \"predicted score\": 0.9407534675828312}, {\"true score\": 1, \"predicted score\": 0.5584926673039418}, {\"true score\": 0, \"predicted score\": 0.045595955969784274}, {\"true score\": 1, \"predicted score\": 0.418379801712513}, {\"true score\": 1, \"predicted score\": 1.108805574872055}, {\"true score\": 1, \"predicted score\": 0.5688509918023358}, {\"true score\": 0, \"predicted score\": 0.701921611416823}, {\"true score\": 0, \"predicted score\": 0.4690783799559369}, {\"true score\": 0, \"predicted score\": 0.442387352027781}, {\"true score\": 0, \"predicted score\": 0.6107251733937575}, {\"true score\": 1, \"predicted score\": 1.3210309604097845}, {\"true score\": 0, \"predicted score\": 0.6132213143791599}, {\"true score\": 2, \"predicted score\": 0.9868488874986869}, {\"true score\": 0, \"predicted score\": 0.5247551613158409}, {\"true score\": 2, \"predicted score\": 0.7430656096898388}, {\"true score\": 2, \"predicted score\": 0.8053467882225733}, {\"true score\": 0, \"predicted score\": 0.4506307953498523}, {\"true score\": 1, \"predicted score\": 0.6518141682146458}, {\"true score\": 0, \"predicted score\": 0.9140091874714855}, {\"true score\": 1, \"predicted score\": 0.5246815047116037}, {\"true score\": 0, \"predicted score\": 0.24691003778802745}, {\"true score\": 2, \"predicted score\": 0.8284973862500127}, {\"true score\": 0, \"predicted score\": 0.34893945941171794}, {\"true score\": 1, \"predicted score\": 1.0405336593889594}, {\"true score\": 1, \"predicted score\": 0.5709673492606244}, {\"true score\": 0, \"predicted score\": 0.9617159484634662}, {\"true score\": 2, \"predicted score\": 1.0631255217848494}, {\"true score\": 0, \"predicted score\": 0.78354097415355}, {\"true score\": 0, \"predicted score\": 0.8163658255582874}, {\"true score\": 0, \"predicted score\": 0.2513355512092883}, {\"true score\": 1, \"predicted score\": 0.4406940700071421}, {\"true score\": 1, \"predicted score\": 0.9527386582041155}, {\"true score\": 1, \"predicted score\": 1.2769598786292802}, {\"true score\": 0, \"predicted score\": 0.029258879849664243}, {\"true score\": 0, \"predicted score\": 0.6712637384697594}, {\"true score\": 0, \"predicted score\": 0.46499146670389646}, {\"true score\": 0, \"predicted score\": 0.2309526832299712}, {\"true score\": 2, \"predicted score\": 1.3332618187745038}, {\"true score\": 0, \"predicted score\": 0.7151346530066354}, {\"true score\": 1, \"predicted score\": 0.6557318727243224}, {\"true score\": 0, \"predicted score\": 0.5750273917137699}, {\"true score\": 1, \"predicted score\": 0.4464173683106265}, {\"true score\": 2, \"predicted score\": 1.0801066417392016}, {\"true score\": 0, \"predicted score\": 0.4206462398957726}, {\"true score\": 0, \"predicted score\": 0.5852496208900385}, {\"true score\": 0, \"predicted score\": 0.42248523280684835}, {\"true score\": 1, \"predicted score\": 0.42790821346527674}, {\"true score\": 1, \"predicted score\": 0.9962542597467299}, {\"true score\": 0, \"predicted score\": 0.40875878010645683}, {\"true score\": 1, \"predicted score\": 1.3133761010291112}, {\"true score\": 1, \"predicted score\": 0.740997572888871}, {\"true score\": 1, \"predicted score\": 0.6953330799876027}, {\"true score\": 1, \"predicted score\": 0.7918777261217261}, {\"true score\": 1, \"predicted score\": 0.6768558768759313}, {\"true score\": 0, \"predicted score\": 0.6679718389174945}, {\"true score\": 1, \"predicted score\": 0.30514175408012495}, {\"true score\": 1, \"predicted score\": 0.6155572288867895}, {\"true score\": 2, \"predicted score\": 0.8850855960853861}, {\"true score\": 0, \"predicted score\": 0.36133183962096593}, {\"true score\": 0, \"predicted score\": 0.7147068965326584}, {\"true score\": 0, \"predicted score\": 0.8752205505113524}, {\"true score\": 2, \"predicted score\": 0.9772980172917799}, {\"true score\": 0, \"predicted score\": 0.6866699874408969}, {\"true score\": 0, \"predicted score\": 0.2670543542080526}, {\"true score\": 0, \"predicted score\": 0.42103864968449045}, {\"true score\": 2, \"predicted score\": 0.6695258288637338}, {\"true score\": 0, \"predicted score\": -0.17412152681178028}, {\"true score\": 0, \"predicted score\": 1.2254486914191256}, {\"true score\": 1, \"predicted score\": 0.8988332909933356}, {\"true score\": 1, \"predicted score\": 0.38811689214653555}, {\"true score\": 1, \"predicted score\": 0.6746407640498969}, {\"true score\": 0, \"predicted score\": 0.9415555742084935}, {\"true score\": 0, \"predicted score\": 0.39347918039881546}, {\"true score\": 2, \"predicted score\": 0.8163339457102818}, {\"true score\": 2, \"predicted score\": 0.9701434554019377}, {\"true score\": 1, \"predicted score\": 0.3625222566619731}, {\"true score\": 0, \"predicted score\": 0.5425336122159138}, {\"true score\": 1, \"predicted score\": 0.7413301106795285}, {\"true score\": 0, \"predicted score\": 0.00306214591367282}, {\"true score\": 2, \"predicted score\": 1.0841354908000997}, {\"true score\": 0, \"predicted score\": 0.25618079655947407}, {\"true score\": 1, \"predicted score\": 1.0081548362971282}, {\"true score\": 2, \"predicted score\": 1.0544540367146165}, {\"true score\": 1, \"predicted score\": 0.4911794973160358}, {\"true score\": 0, \"predicted score\": 0.8276569369753841}, {\"true score\": 0, \"predicted score\": 0.5044848394128655}, {\"true score\": 1, \"predicted score\": 0.8911305206235586}, {\"true score\": 1, \"predicted score\": 0.6622313217913064}, {\"true score\": 2, \"predicted score\": 0.9664876707408332}, {\"true score\": 2, \"predicted score\": 0.623636912038797}, {\"true score\": 1, \"predicted score\": 0.9874956854255155}, {\"true score\": 0, \"predicted score\": 0.41388983355475784}, {\"true score\": 1, \"predicted score\": 0.7251415509570314}, {\"true score\": 1, \"predicted score\": 0.8760567161870698}, {\"true score\": 0, \"predicted score\": 0.6491432234688049}, {\"true score\": 3, \"predicted score\": 1.1044113157702844}, {\"true score\": 0, \"predicted score\": 0.278838361621642}, {\"true score\": 2, \"predicted score\": 0.6419939867296299}, {\"true score\": 0, \"predicted score\": 0.3099143097383287}, {\"true score\": 0, \"predicted score\": 0.9550918089601926}, {\"true score\": 0, \"predicted score\": 0.44712046551233375}, {\"true score\": 0, \"predicted score\": -0.3881095821448668}, {\"true score\": 0, \"predicted score\": 0.572536565666143}, {\"true score\": 0, \"predicted score\": 0.48695846965980705}, {\"true score\": 0, \"predicted score\": -0.13674946908468033}, {\"true score\": 0, \"predicted score\": 0.8093417026974665}, {\"true score\": 1, \"predicted score\": 0.8422901314076339}, {\"true score\": 1, \"predicted score\": 1.024933900546993}, {\"true score\": 0, \"predicted score\": 0.5806392154616153}, {\"true score\": 0, \"predicted score\": 0.7638891180999883}, {\"true score\": 0, \"predicted score\": 0.6557254988753827}, {\"true score\": 1, \"predicted score\": 0.7962860332458367}, {\"true score\": 2, \"predicted score\": 0.36733810872772105}, {\"true score\": 0, \"predicted score\": 0.06128524037683347}, {\"true score\": 1, \"predicted score\": 1.1843603054695404}, {\"true score\": 0, \"predicted score\": 0.8056385143033858}, {\"true score\": 2, \"predicted score\": 0.9750137873097666}, {\"true score\": 1, \"predicted score\": 0.5586602708374594}, {\"true score\": 4, \"predicted score\": 1.0932780465008742}, {\"true score\": 0, \"predicted score\": 0.20767586635255952}, {\"true score\": 0, \"predicted score\": 0.19688835307216196}, {\"true score\": 1, \"predicted score\": 0.6072634454309229}, {\"true score\": 1, \"predicted score\": 0.09317339954547532}, {\"true score\": 2, \"predicted score\": 0.5918647936214934}, {\"true score\": 0, \"predicted score\": 0.3874161501538984}, {\"true score\": 0, \"predicted score\": 0.09509186095350153}, {\"true score\": 0, \"predicted score\": 0.49926896458224745}, {\"true score\": 0, \"predicted score\": 0.8417570218812053}, {\"true score\": 3, \"predicted score\": 0.8428365048290105}, {\"true score\": 1, \"predicted score\": 0.7428461931169418}, {\"true score\": 0, \"predicted score\": 1.125780818156807}, {\"true score\": 0, \"predicted score\": 0.3621601471988468}, {\"true score\": 0, \"predicted score\": 0.33948756547657244}, {\"true score\": 0, \"predicted score\": 0.7348082021904964}, {\"true score\": 0, \"predicted score\": 0.51910878695469}, {\"true score\": 0, \"predicted score\": 0.2801426645001052}, {\"true score\": 2, \"predicted score\": 0.7957815170263871}, {\"true score\": 0, \"predicted score\": 0.7101803522572037}, {\"true score\": 1, \"predicted score\": 0.8867311506075232}, {\"true score\": 0, \"predicted score\": 0.28457894632142405}, {\"true score\": 1, \"predicted score\": 0.9924081678820366}, {\"true score\": 3, \"predicted score\": 0.41436517649344445}, {\"true score\": 0, \"predicted score\": 0.6203807363351657}, {\"true score\": 0, \"predicted score\": 0.841783791633622}, {\"true score\": 1, \"predicted score\": 0.30470056500137166}, {\"true score\": 0, \"predicted score\": 0.8406151630171342}, {\"true score\": 0, \"predicted score\": 0.4938151018700741}, {\"true score\": 2, \"predicted score\": 0.19461839284367666}, {\"true score\": 1, \"predicted score\": 0.695481301810904}, {\"true score\": 2, \"predicted score\": 0.8729550888755162}, {\"true score\": 0, \"predicted score\": 0.8167097202899227}, {\"true score\": 0, \"predicted score\": 0.5505363868346281}, {\"true score\": 0, \"predicted score\": 0.45718443205868753}, {\"true score\": 0, \"predicted score\": 0.7550670451705134}, {\"true score\": 1, \"predicted score\": 0.8467404030194714}, {\"true score\": 0, \"predicted score\": 0.8700776877384475}, {\"true score\": 0, \"predicted score\": 0.05411335740430556}, {\"true score\": 1, \"predicted score\": 1.1077961709495205}, {\"true score\": 0, \"predicted score\": 0.5129393417396534}, {\"true score\": 2, \"predicted score\": 0.6207855916651239}, {\"true score\": 0, \"predicted score\": 0.20181459130563295}, {\"true score\": 0, \"predicted score\": 0.5252843486513812}, {\"true score\": 0, \"predicted score\": 0.3853388020096091}, {\"true score\": 0, \"predicted score\": 0.5891335852032202}, {\"true score\": 2, \"predicted score\": 0.649170948859066}, {\"true score\": 1, \"predicted score\": 0.8402348635432486}, {\"true score\": 2, \"predicted score\": 0.7093393702019832}, {\"true score\": 1, \"predicted score\": 0.774888812436088}, {\"true score\": 2, \"predicted score\": 0.987437518978932}, {\"true score\": 0, \"predicted score\": 0.4605469704871898}, {\"true score\": 2, \"predicted score\": 0.6671478993855726}, {\"true score\": 0, \"predicted score\": 0.999080924863081}, {\"true score\": 2, \"predicted score\": 0.932826009505396}, {\"true score\": 0, \"predicted score\": 0.1815362078033259}, {\"true score\": 0, \"predicted score\": 0.6921317040523192}, {\"true score\": 2, \"predicted score\": 0.9505804790079323}, {\"true score\": 0, \"predicted score\": 0.31509393412610515}, {\"true score\": 0, \"predicted score\": 0.6038855817711479}, {\"true score\": 0, \"predicted score\": 0.5545282138653901}, {\"true score\": 0, \"predicted score\": 0.4089949349218725}, {\"true score\": 1, \"predicted score\": 0.8852685347218654}, {\"true score\": 2, \"predicted score\": 0.9635969067858362}, {\"true score\": 0, \"predicted score\": 1.0346360436358948}, {\"true score\": 0, \"predicted score\": 0.6062353891206927}, {\"true score\": 0, \"predicted score\": 0.22053520517974823}, {\"true score\": 0, \"predicted score\": 0.809537456378016}, {\"true score\": 0, \"predicted score\": 0.42873976072254477}, {\"true score\": 0, \"predicted score\": -0.032047009866547516}, {\"true score\": 2, \"predicted score\": 1.1461137877156542}, {\"true score\": 0, \"predicted score\": 1.078114544611925}, {\"true score\": 0, \"predicted score\": 0.49403892829617235}, {\"true score\": 0, \"predicted score\": 0.9603267467538943}, {\"true score\": 0, \"predicted score\": 0.5669596008879237}, {\"true score\": 4, \"predicted score\": 0.6378435851343239}, {\"true score\": 2, \"predicted score\": 0.8079092524705594}, {\"true score\": 0, \"predicted score\": 0.6683957061173861}, {\"true score\": 1, \"predicted score\": 0.32316510687694705}, {\"true score\": 0, \"predicted score\": 0.40903275980309733}, {\"true score\": 1, \"predicted score\": 0.5409677639357661}, {\"true score\": 1, \"predicted score\": 0.7324339375734567}, {\"true score\": 2, \"predicted score\": 0.8258521751543302}, {\"true score\": 2, \"predicted score\": 1.0415036968955942}, {\"true score\": 0, \"predicted score\": 0.7526551408470581}, {\"true score\": 0, \"predicted score\": 0.43175719552794534}, {\"true score\": 1, \"predicted score\": 0.9749948682808677}, {\"true score\": 0, \"predicted score\": 0.9398535139596532}, {\"true score\": 0, \"predicted score\": 0.19599366681472818}, {\"true score\": 1, \"predicted score\": 0.4860798928296308}, {\"true score\": 1, \"predicted score\": 0.6504046327294106}, {\"true score\": 0, \"predicted score\": 0.5017844995554589}, {\"true score\": 1, \"predicted score\": 0.9748806958131607}, {\"true score\": 0, \"predicted score\": 0.5330440764927351}, {\"true score\": 0, \"predicted score\": 1.2627350710334824}, {\"true score\": 0, \"predicted score\": 0.37054454263324704}, {\"true score\": 0, \"predicted score\": 0.2821260020646043}, {\"true score\": 0, \"predicted score\": 0.36053969246769646}, {\"true score\": 0, \"predicted score\": 0.45142139425151395}, {\"true score\": 0, \"predicted score\": 0.3071088475161638}, {\"true score\": 0, \"predicted score\": 1.2969075951360391}, {\"true score\": 0, \"predicted score\": 0.511672048946337}, {\"true score\": 0, \"predicted score\": 0.6759852105474139}, {\"true score\": 0, \"predicted score\": 1.0628886858922864}, {\"true score\": 1, \"predicted score\": 0.8413122741783465}, {\"true score\": 0, \"predicted score\": 0.7205117758412836}, {\"true score\": 3, \"predicted score\": 1.2071847897005856}, {\"true score\": 0, \"predicted score\": 0.4853225608946612}, {\"true score\": 2, \"predicted score\": 0.41119057673317994}, {\"true score\": 0, \"predicted score\": 0.49907824720203675}, {\"true score\": 1, \"predicted score\": 0.7489853567951487}, {\"true score\": 1, \"predicted score\": 0.5959092451127931}, {\"true score\": 0, \"predicted score\": 0.4198167896744911}, {\"true score\": 0, \"predicted score\": 0.8870809385148655}, {\"true score\": 1, \"predicted score\": 0.39960418311698515}, {\"true score\": 0, \"predicted score\": 0.3521504760968014}, {\"true score\": 1, \"predicted score\": 1.1546514060158874}, {\"true score\": 1, \"predicted score\": 0.9476781279540262}, {\"true score\": 1, \"predicted score\": 0.8345644522024448}, {\"true score\": 1, \"predicted score\": 1.2897190145941404}, {\"true score\": 0, \"predicted score\": 0.4927309975172566}, {\"true score\": 0, \"predicted score\": 0.4692785951762401}, {\"true score\": 1, \"predicted score\": 0.6936329409248199}, {\"true score\": 1, \"predicted score\": 0.8812477016533081}, {\"true score\": 0, \"predicted score\": 0.4774561888051785}, {\"true score\": 3, \"predicted score\": 1.0886386104036125}, {\"true score\": 0, \"predicted score\": 0.8317342836315516}, {\"true score\": 0, \"predicted score\": 0.517526419537418}, {\"true score\": 0, \"predicted score\": 0.6152308806064251}, {\"true score\": 0, \"predicted score\": 0.916579917125202}, {\"true score\": 1, \"predicted score\": 0.8223960969601236}, {\"true score\": 2, \"predicted score\": 1.0779290958047252}, {\"true score\": 1, \"predicted score\": 0.8812157495230292}, {\"true score\": 0, \"predicted score\": 0.5166996769712289}, {\"true score\": 0, \"predicted score\": 0.3590936723215982}, {\"true score\": 1, \"predicted score\": 0.6878916340366379}, {\"true score\": 0, \"predicted score\": 0.43802077557457386}, {\"true score\": 2, \"predicted score\": 0.7168873161757028}, {\"true score\": 1, \"predicted score\": 0.667810350256033}, {\"true score\": 1, \"predicted score\": 0.35929617113318657}, {\"true score\": 0, \"predicted score\": 0.346981569758091}, {\"true score\": 0, \"predicted score\": -0.22298051254226958}, {\"true score\": 2, \"predicted score\": 0.24260241120121234}, {\"true score\": 1, \"predicted score\": 0.9620401260422249}, {\"true score\": 1, \"predicted score\": 0.7326030952751921}, {\"true score\": 0, \"predicted score\": 0.6595179223542982}, {\"true score\": 0, \"predicted score\": 0.2672301304878227}, {\"true score\": 0, \"predicted score\": 0.6247794106430902}, {\"true score\": 0, \"predicted score\": 0.48192815813974066}, {\"true score\": 1, \"predicted score\": 0.8311175877298823}, {\"true score\": 0, \"predicted score\": 0.1003120550838445}, {\"true score\": 1, \"predicted score\": 0.6791579433509097}, {\"true score\": 1, \"predicted score\": 0.42879158418680785}, {\"true score\": 0, \"predicted score\": 0.474441692117718}, {\"true score\": 2, \"predicted score\": 1.0333848435602808}, {\"true score\": 0, \"predicted score\": 0.3520213876688544}, {\"true score\": 1, \"predicted score\": 1.0244847631836262}, {\"true score\": 0, \"predicted score\": 0.5493175943140206}, {\"true score\": 0, \"predicted score\": 0.8188725380216334}, {\"true score\": 0, \"predicted score\": 0.4018793353529808}, {\"true score\": 2, \"predicted score\": 1.1130345136424742}, {\"true score\": 0, \"predicted score\": 1.3220375615199043}, {\"true score\": 1, \"predicted score\": 0.26409420067757855}, {\"true score\": 0, \"predicted score\": 0.37477281654245587}, {\"true score\": 1, \"predicted score\": 0.80127837019967}, {\"true score\": 2, \"predicted score\": 0.7739095540952061}, {\"true score\": 0, \"predicted score\": 0.5008511646798586}, {\"true score\": 1, \"predicted score\": 0.8695919595680446}, {\"true score\": 0, \"predicted score\": 0.6319124445536057}, {\"true score\": 0, \"predicted score\": 0.562328572860801}, {\"true score\": 0, \"predicted score\": 1.100238657391875}, {\"true score\": 1, \"predicted score\": 0.7875057594157466}, {\"true score\": 0, \"predicted score\": 0.10781348722812056}, {\"true score\": 0, \"predicted score\": 0.8208175250325485}, {\"true score\": 1, \"predicted score\": 0.5135352769975997}, {\"true score\": 0, \"predicted score\": 0.4468200887972369}, {\"true score\": 0, \"predicted score\": 0.6713326748731687}, {\"true score\": 0, \"predicted score\": 0.11175444655720956}, {\"true score\": 0, \"predicted score\": 0.9110817637466134}, {\"true score\": 0, \"predicted score\": 0.42754558779975615}, {\"true score\": 0, \"predicted score\": 0.30999939936285187}, {\"true score\": 0, \"predicted score\": -0.0057884618866141935}, {\"true score\": 0, \"predicted score\": 0.6626390654695526}, {\"true score\": 0, \"predicted score\": 0.41993390220884375}, {\"true score\": 0, \"predicted score\": 0.10991132846544993}, {\"true score\": 0, \"predicted score\": 0.7884369520750225}, {\"true score\": 0, \"predicted score\": 0.5886917534262053}, {\"true score\": 1, \"predicted score\": 0.9374671833363115}, {\"true score\": 0, \"predicted score\": 0.3275815239743179}, {\"true score\": 1, \"predicted score\": 0.9420541860632201}, {\"true score\": 0, \"predicted score\": 0.6477032076578455}, {\"true score\": 0, \"predicted score\": 0.5513956305335722}, {\"true score\": 0, \"predicted score\": 0.8508942473135439}, {\"true score\": 1, \"predicted score\": 0.44601250994868363}, {\"true score\": 1, \"predicted score\": 1.0168902897293743}, {\"true score\": 0, \"predicted score\": 0.6166478587837719}, {\"true score\": 1, \"predicted score\": 0.8348667125794497}, {\"true score\": 0, \"predicted score\": 0.6181830710997384}, {\"true score\": 0, \"predicted score\": 0.5484107948301804}, {\"true score\": 0, \"predicted score\": 0.4878471997356369}, {\"true score\": 1, \"predicted score\": 0.9522807959918418}, {\"true score\": 0, \"predicted score\": 0.6636472151599588}, {\"true score\": 1, \"predicted score\": 0.8344221236570342}, {\"true score\": 1, \"predicted score\": 0.7667289754702357}, {\"true score\": 0, \"predicted score\": 0.45589010986039064}, {\"true score\": 0, \"predicted score\": 0.24116231646134956}, {\"true score\": 1, \"predicted score\": 0.48344765683839974}, {\"true score\": 3, \"predicted score\": 1.0394773654329332}, {\"true score\": 0, \"predicted score\": 0.8432771866667607}, {\"true score\": 0, \"predicted score\": 0.5672665905730464}, {\"true score\": 1, \"predicted score\": 0.6272506846448431}, {\"true score\": 0, \"predicted score\": 0.31374351061780087}, {\"true score\": 0, \"predicted score\": 0.34057454944744503}, {\"true score\": 0, \"predicted score\": 0.2923945100818917}, {\"true score\": 1, \"predicted score\": 0.6093118084150897}, {\"true score\": 1, \"predicted score\": 0.6075079612994865}, {\"true score\": 0, \"predicted score\": 0.5980216408121446}, {\"true score\": 1, \"predicted score\": 0.7532913756940544}, {\"true score\": 1, \"predicted score\": 1.0209783079830639}, {\"true score\": 1, \"predicted score\": 0.6533972857683179}, {\"true score\": 0, \"predicted score\": 0.6248419833200447}, {\"true score\": 0, \"predicted score\": 0.3793692708202284}]}};\n",
       "var opt = {};\n",
       "var type = \"vega-lite\";\n",
       "var id = \"1d18947e-20bb-4f91-8e3d-6d0284027ee1\";\n",
       "\n",
       "var output_area = this;\n",
       "\n",
       "require([\"nbextensions/jupyter-vega/index\"], function(vega) {\n",
       "  var target = document.createElement(\"div\");\n",
       "  target.id = id;\n",
       "  target.className = \"vega-embed\";\n",
       "\n",
       "  var style = document.createElement(\"style\");\n",
       "  style.textContent = [\n",
       "    \".vega-embed .error p {\",\n",
       "    \"  color: firebrick;\",\n",
       "    \"  font-size: 14px;\",\n",
       "    \"}\",\n",
       "  ].join(\"\\\\n\");\n",
       "\n",
       "  // element is a jQuery wrapped DOM element inside the output area\n",
       "  // see http://ipython.readthedocs.io/en/stable/api/generated/\\\n",
       "  // IPython.display.html#IPython.display.Javascript.__init__\n",
       "  element[0].appendChild(target);\n",
       "  element[0].appendChild(style);\n",
       "\n",
       "  vega.render(\"#\" + id, spec, type, opt, output_area);\n",
       "}, function (err) {\n",
       "  if (err.requireType !== \"scripterror\") {\n",
       "    throw(err);\n",
       "  }\n",
       "});\n"
      ],
      "text/plain": [
       "<vega.vegalite.VegaLite at 0x14ad72978>"
      ]
     },
     "metadata": {
      "jupyter-vega": "#1d18947e-20bb-4f91-8e3d-6d0284027ee1"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcoAAAFfCAYAAADON4wsAAAAAXNSR0IArs4c6QAAQABJREFUeAHtnQmcHEXd/rt7Nsnu5k4IZwgJSwiX8HIql29ArlwISELAA/FARVHxDySBVxgU2AXh5VREFOTeBFQgCQjycimCEG4JcmwSIByBhJBrj+xu9/95enqW2c307MzOTE9Pz/PL50lXV3VXV31rZ35d1dU1hiETAREQAREQAREQAREQAREQAREQAREQAREQAREQAREQAREQAREQAREQAREQAREQAREQAREQAREQAREQAREQAREQAREQAREQAREQgbwJDEAOA3vJZWgv6UoWAREQAREQgUgSuAq1eh6aCz0ADYZSbX/svAo9CC2GDoBkIiACIiACIlARBOgUX0ip6bMIz0jZZ/Bx6Ggv7qvYPuGFtREBERABERCBiiGwJ2r6G+hDaLMetX4P++O8uM9j+36PdO2KgAiIgAiIQNEIVBUt59wytnB4LdQP2hfiEGzShiDQ7u2sxtZOJtTX18dN0zw/uc9tbW2tMWXKlNQohUVABERABEQgKwJ1dXVmVgcGeNB2uNYJKde7HOE/puwz+BTE55Q0Pp98yA35/NfQ0OD4JJVVdFNTk+oRohZTe4SoMVAUtYfaoxgE/P6u2JMrpW3Axa+FxkAsC4dY+cxyEHQgROP+JCgGnQy9AslEQAREQAREIBACpR56XYla/gHirNc26DXoNmgCNB8aAXFWLCf0fBeiY032LhGUiYAIiIAIiEBxCZTaUbJ2s6FfQHxGScdJWwXRSdJeh0ZD7HUuhSIxJIl6yERABERABMqAQBgcJTE1e/JD1oGEJX6JihcBERABERCBYhEo9TPKYtVL+YqACIiACIhAQQjIURYEozIRAREQARGIKgE5yqi2rOolAiIgAiJQEAJylAXBqExEQAREQASiSkCOMqotq3qJgAiIgAgUhIAcZUEwKhMREAEREIGoEpCjjGrLql4iIAIiIAIFISBHWRCMykQEREAERCCqBOQoo9qyqpcIiIAIiEBBCMhRFgSjMhEBERABEYgqATnKqLas6iUCIiACIlAQAnKUBcGoTERABERABKJKQI4yqi2reomACIiACBSEgBxlQTAqExEQAREQgagSkKOMasuqXiIgAiIgAgUhIEdZEIzKRAREQAREIKoE5Cij2rKqlwiIgAiIQEEIyFEWBKMyEQEREAERiCoBOcqotqzqJQIiIAIiUBACkXSUE+OPVhWEjjIRAREQARGoeAKRdJSDWlfcOO3sxi9UfOsKgAiIgAiIQN4EIukoQWW4bRmzj5l959i8CSkDERABERCBiiYQSUfpOMb9pmHEOgxzakW3riovAiIgAiKQN4FIOkrTsZ4jGdMwN8+bkDIQAREQARGoaAKRdJSG5Uxiqzqms7SiW1eVFwEREAERyJtANB2l4ezjGM669lbr3rwJKQMREAEREIGKJhDN1ygc56+xmHP3witmfFLRravKi4AIiIAI5E0gLI6S5RgCFcSxLbjkxF/nTUYZiIAIiIAIiAAIhGHo9VyUYzF0PfQAtBuUaodj503oPk+npCYqLAIiIAIiIALFJFDqHuUAVO4CaDi0DroIOgP6NpS0nRC4DroGak9GaisCIiACIiACQRAodY+yDZXcAqKTHAgdCf0dSrUJ2GGvsxmaD42HZCIgAiIgAiIQCIFSO0pWchXE5eaehd6B5kKptgw7P4JGQiuh2ZBMBERABERABAIhgAVsSm6TUYKboLOhm9OUphZx7E3SDoDugMZCRn19fdw0zfMZTrXp06en7iosAiIgAiIgAlkRqKurC4Nf7FZWFog9yoO6xRrGIOwf6MWxp3myF6ZTpFP1tYaGBsc3sYwSmpqaVI8QtZfaI0SNgaKoPdQexSDg93dV6qFXPn8cAT0B2Z449Mp4Po+kxaFfQpz5ejB0JSQTAREQAREQgUAIlHrW639QS79uLh0obSE0DhoFfQjJREAEREAERCAwAqXuUWZb0U4cKCeZLS0dJwIiIAIiUDAC5eIoC1ZhZSQCIiACIiACuRCQo8yFlo4VAREQARGoOAJylBXX5KqwCIiACIhALgTkKHOhpWNFQAREQAQqjoAcZcU1uSosAiIgAiKQCwE5ylxo6VgREAEREIGKIyBHWXFNrgqLgAiIgAjkQkCOMhdaOlYEREAERKDiCMhRVlyTq8IiIAIiIAK5EJCjzIWWjhUBERABEag4AnKUFdfkqrAIiIAIiEAuBOQoc6GlY0VABERABCqOgBxlxTW5KiwCIiACIpALATnKXGjpWBEQAREQgYojIEdZcU2uCouACIiACORCQI4yF1o6VgREQAREoOIIyFFWXJOrwiIgAiIgArkQkKPMhZaOFQEREAERqDgCcpQV1+SqsAiIgAiIQC4E5ChzoaVjRUAEREAEKo6AHGXFNbkqLAIiIAIikAsBOcpcaOlYERABERCBiiMgR1lxTa4Ki4AIiIAI5EJAjjIXWjpWBERABESg4gjIUVZck6vCIiACIiACuRCQo8yFlo4VAREQARGoOAJylBXX5KqwCIiACIhALgTkKHOhpWNFQAREQAQqjkBYHGUVyI/ohf7QXtKVLAIiIAIiIAIFJxAGR3kuarUYuh56ANoNSrX9sfMqNA/icQdAMhEQAREQAREIhECpHeUA1PICaG9oOvQ8dAaUag3YmQMdCV0EcV8mAiIgAiIgAoEQKLWjbEMtt4DWQQMhOsO/Q6m2A3Ze8SLewpb7MhEQAREQAREIhIAZyFV6v8gXcMiN0H+gr0ItUNLoRHeGlkM7Qo9AoyGjvr4+bprm+Qyn2vTp7JzKREAEREAERCA3AnV1dWHxi90KPhl7K6CTu8V+tvMUgnxOSePzyYfckM9/DQ0Njk9SWUU3NTWpHiFqMbVHiBoDRVF7qD2KQcDv76rUQ6/03LdCX4FuTqn4IIQP9PZfwHYSFIPoTJPDsAjKREAEREAERKC4BErtKCegenwt5AnI9jQXW8bPh2hXQadCHHr9EqTJPIAgEwEREAERCIZAVTCX8b0Kn0n6jQcn36t8HcfwmeQYaCkUiSFJ1EMmAiIgAiJQBgRK7SizRdSBA5dke7COEwEREAEREIFCESj10Guh6qF8REAEREAERKAoBOQoi4JVmYqACIiACESFgBxlVFpS9RABERABESgKATnKomBVpiIgAiIgAlEhIEcZlZZUPURABERABIpCQI6yKFiVqQiIgAiIQFQIyFFGpSVVDxEQAREQgaIQkKMsClZlKgIiIAIiEBUCcpRRaUnVQwREQAREoCgE5CiLglWZioAIiIAIRIWAHGVUWlL1EIEMBOLxuDXtf+aNW7WuzWA4w6FKEgER6EFAH5geQLQrAlEjcPTsO/dc1DLht06HffWf/vmOwTDjolZP1UcEikVAjrJYZJWvCISAwJfPvXNb27B+bpjmVvjhneXDBvbH7/WYWzGOaSEoooogAqEnIEcZ+iZSAUWg7wTsTnMaHGQ/xzEeWtBw4g9mHLSdwTDjEml9z1tnikClEJCjrJSWVj0rkoBjOuhJGoZlGk8nASTDybRkvLYiIALpCchRpueiWBGIBAH8KvoyryJHTIw/WmXb7u+eH8G4lDTvEG1EQATSEZCjTEdFcSIQEQIdRuw+VGUD3OMXBrWtuPXWx5YYDDPOS4tITVUNESgeATnK4rFVziJQcgIP1M/4OBZzzjId40l4yIEb222DYcYxreQFVAFEoAwIVJVBGXMu4rQ5d+y094A33sD7YnbOJ+sEEYgYgXsvOvFdVKlhUvy2IV8/ZPs1u+68Y0PEqqjqiEBRCUSyR+k41q8Wte48CzP78BhGJgIiQAIPxL+2trp/TDBEQARyJBBJRwnv2AInecDRc+YdkCMPHS4CIiACIiAC3QhE0lHaptHIWnY69kHdaqudwAgcceYtA6fNbtz3nY83GAwHdmFdSAREQAQKTCCSjhIjrms9TliGRBY0gWmzGif1rxpwK2ZXnvfX5983GGZc0OXQ9URABESgEAQi6ShNw/m+4+D5pGM8XwhIyiN7ApPnNO4I8j/A0HcVXtT791YjanCyU8U4pmWfk44UAREQgXAQiKSjBNqDDNMa9uHqtViqSxYkAcs22HPEGwjmPQvqZ86Ztu9o3K+Y9zDOSwuyOLqWCIiACORNIKqO8jXTcdZuOXzIvnkTUgY5ETBNcyRPsGxncfLEZDiZlozXVgREQATKgUAkHaVpmNdi2K8TOrgcGiFKZcQKaW+49bHMY44985bN17W0c6HRYxjXlRalCqsuIiACkScQJkc5BLQL8t6jY9mfei2nyTwB/wnbNe1cMu0Tx3B2ba/q//vGJ5Zh6NXZlXFeWsAl0uVEQAREID8CfVmZZ3tccgzEiTIboE4oH+OvG+wN3QztBn0Apdrh2PkN9JoX+Rdsb/LC6Te2OZMJpmO9mP4AxRaLAF9qPyb+l590tLYeibueSbXV/BNzbquqrn5wQXxmcjZysS6vfNMQOGzWvKE1lnMIZiEfdveTbxvT5sw9psU2H334khlr0hyuKBEQgR4EcnGU7H3eAn3Vy4PO6HzoEGiFF9eXzaE4iUOk7FGms50QeR10DYRxvKxsSxy17IPVn/41q6N1UEEJ3BM/lj36uXCQ865oWnLyed86DGFZKQhMj88b1NxqX4lJ4OPQHsOWrFjP36P86QDT/jLSTr8rPmN9Kcqla4pAORGg88vWTsCBdJJ/805Yju1Y6Bxvv6+b23Hi96E2nwwmIP5cqBmaD42HMpppWpeZ1QNnPfe772XrWDPmp8S+EjC1hmBf0RXovNaWzqORFVaocraFVg0c4PbwETYO8NIKdCVlIwLRJZCLo9wdGDh0dpKH49/YPgbt4O0Xa7MMGf8I4mzKldBsKKPNr5/x+Pz4NDpWmQhUNAHbso7AEPhQPCheioU4bt5vR3yMEGYc0yoajiovAlkSwOcla/sejvwtdAP0XegK6FSIjutaKF/jEBB7iz2fUdYiLun0cGds3AGNhYz6+vo4Xjng8G83mz59erd97YhApRL43YNvGlxGcK/tRxjHHcCpBYbx53++Yzy/5BNjzKiBxqlH9jpAU6noVO8KJVBXV5eLX9yE0gDEcEYjhzMxL8DVUmzZ0yuE0VFyYg9tEHSgGzKMZ7E92QvTKd7khdNuGhoanLQJZRbZ1NRU1vWYGH+0iivxPPX8qw7DZYZ/k+KWa3tMnT33qqmzGz+cOqvxqSlzGi+4dt4TDsNuHNI2qWiZRJRre/TEq3r0JFLafb/2yGXodRyqwEk120Ecfp0CfQ5aBRXKks6BzyX5PJIWh34JvQlx0s+VkCzEBKaePW+/wa0f/t5yjMvvefpdg2HGhbjIkS2abXQ+gMotNyxjG9M2j3rtXUx0RZhxXlpk666KiUChCOTiKCfiovdDe0F3emH2Agtl7EV+6GX2HLYjvPBCbOmk6SQPg16CZCElMPnc27czTHsOlq3jSMOyEYP7Y+gBYcS5aSEtd1SL1Vy91cNogMdRv8V4X6rDstxRpcWMc9OiWnHVSwQKSCAXR8khUDrGy6FzoB97+iK2xbZOXCDpRIt9LeWfBwGrw5qKZSOq8HX8wIKGmacffwD8JsKMc9PyyFun5k7gsfghHetrWs5FN/ICtMGf9hg3HJlYFzCOabnnqDNEoPII5OIovwA87PXtCF0E8fkGdRwkEwGXAF7N2YIBx7YWJZEkw8m0ZLy2wRAYbGzGzzmHW/d+d6U7L24bLy6YAugqIlDmBHKZZHE36vpymvq+lyZOURVKwMZwK3qQ+5iWc9qUcxqPXrhoOX42xDkND59jTKtQLCWr9t6nXt/Pad3wN6xThfkEZv+Xln7CaXg3Iu4VpE3Uu8YlaxpduIwI5NKj5Oo7b0PsWZ4P8R2s1dASSCYCLoEBVhUnj4zH+q5HYRX0bz39+kp8LyOMOC9NpAIksPXIYWfhcnuhDfrBQ66u7V/F9kDY2MtLC7A0upQIlCeBXBzlEFSRk2wuhQ6B/gdiD5MTbGQi4BJodzo+75jmUvQgmyzDfHPUkGp8MRtNjGOaMAVLAD9gfiJ6klVYI8nG0nVV/I9hxiXSgi2PriYC5UggF0f5Q1RwM4jPJGugfSE+8DgFCpVNnXP75CPOvGVgqApVKYWxnd3xq83t8I6NWL/un+O2wGNthN04pFUKhtDU0+HscacK3nEAFucY0N6J2xaE3Tg3LTQlVUFEILQEcnGUnKTBWa9/hTZCnKzBYdetoXCZHbu5f1X/xkmn348vBFmgBEynDTN5djdNrM/rGKf8642VRiJs744vaL/1fAMtYkVdzDQ5zMqZx/CTzspB+DUXhl0GXlpF8VBlRaAPBHJxlHSM6B4YD0LnQX+GMEHAaITCZab7SySHVg1c++1wFSz6pbHRa8H38Cg8BxuK2rZV4b29RNgclUiLPoNw1dB2b07Qj7Qw6jp21Trcx3DJAdcSaeEqr0ojAuEjkMus19tRfD5j+jrEl//5AbwbuhEKl5lmB3o11bZhfgsF+024Chft0uD5197oOfIZGHstg9s6EEz0YWw3LdrVD13tTMPs4FNJGGYfm50WhmE7bSyPjuZBA+k9ytC1mAoURgK59ChxI2qcDtVB/w1xyJWrj7dDoTL0YdwvalRudKgKVgGFcQybQ/T98CXcCbX0i7lfyFwwop+XVgEUwlNFfBDW4Xkk/SI2TidX5nHDiEukhaesKokIhJVALo6yGpW4EXoIegK6HuLwa+gmzeDmmZONMH+Eo0yyIAngG5l/U67QDs+M2Zyj9Yl9Ly3I4lT8teAX2+Aj4RNdcRycZjOOae6e/hMBEchIIBdHORE5nQK94OX4IrZHQz/39sOzcdwvZn4lrApPoSqjJOjLt6Cm+GJ2bMd09nkXP/HEMOO8tMoAEZJaOo65AYOu7ehHrsKNyuN1Ww3GkKu5inFMC0kxVQwRCDWBXBzlAagJn2n8P69GF2HL9yj38fZDs8FrCRzq412zHGXArYJhvY/hGNegQ78C71GuHdAvhhKYWKzCWZNIC7hAFX459OqfBYKNXGQAzyl3+XB1KydXcSbsRi+twgmp+iLQO4FcJvO8jex4/G3Qn6DdoD2hq6FQGRxkDOOuHXCYy0JVsAoojGlb9zqWvZ/7MoLjfFo7AI6ST8gco5lpFYAgVFU0TfufGPneC71IzkSu7l+F2xfDXI7hlo8TaaEqbgUVxjGnzbpr29XrN6LO+KZynxRVUPVDVNVJ8duGVLVVfRFDYIf6FSsXR3krMpkGTYGmehk2YVvvhcO0wQRLTB7hi++yYAnU1vzJaN1wmGPbu8BBbr2mGWsP2DaH/RYbTJMFSqBzw9D7YwPXHIGh1mY4yKrNhgwY//HaVtz0OsuZFmhhdDGXwNQ58/Y3nLkX48Zll8vvedWYOvulRwxz3jkL6mc8JUTBEpgWn19rt2z4X/iLLZIP8NOVIJehV976HAONgjjb9XPQDlBof/4K/Zgvo3yyAAnMj09r7qzuON2wrGvRq39xm5G1vF/+NeOYFmBRdCkQeOCayW011bGzLNu+GvNcF40ZNdBgmHFME6RgCUw96/bxhtO5ED3I/aHBre18fI8w4ty0YIujq7U0T4Wf4Ez9JsyhONsPSC6OknmcAnECDz9gd0K/g7isXagMPcnkbNdhoSpYhRTGaqv6Ft7c+z7G+PBMrAUzq8zvMa5Cqh+6au5qLG7uiFlvm461eIvhNQbDjAtdQSuhQLHY5RgKx92jvQ49/MsO3X1LdO5tvMKDODetEiCEp464edyFpbFM866Fl858za9kuTjKLyGTG6Gx0LXQOOgb0IlQqIyPxBLWFQhV+aJcmCmz7jwATvKnqONY3LDE3PmuCDOOaVGuezjr5piLWif83HLMeSjf5fc8/Q6W6DHnMS7xbCycpY5sqRx3bgc6kcbd8xtOOG/y3viZUITd+ibTIlv5EFbMNNeyVHjHe/tMpcvFUR6MjJZCj0BjoAuhOyA60JAZBvtg+E9DSwG3jGVZp2KsdbDH/+NBNd5jcMS5aQGXp9Ivd+ycu76I55Nn8HkYXtcZuaEVi1YxjDimVTqfwOtvOsme/M7xeJzLCtJ2dv//LM3d1X/FJ2CZ9sO8iuNYx0+b1Xie3xVzcZTvIxO+yP8TLzO+T8nfpnzX2w/jJsxlCyOvvMvkOPaOuENxVxnAdvN1Ld58KsS5aXlfQRnkQmCjYc/BDeMQPC/mz2olbiARZhzTcslLx+ZPAI4RnQu6R3O/59p2euPCeXzDztyPcYm0/K+hHLIncN/FJ7EBsF65g4EW9xex0p6ci6O8Bzmwe3A8hCnnxgSId0ILoFAa3nBn71cWJAGHi6Lj78QxhnZdNhFGnJvWFa1A8QlgyHsfXAWPw/jlbLVW4fWQRBhzkhNpxS+ErtBFoLYmdhm+lPluayfuW0Z/uoFzJPnet/NsIq3rUAUCIrCgYebtnab1LTyTuNLvkrk4yo+QCYdcj4IOh/4F8T3KB6FQmmVaB4WyYBEulGNhZR6M6+EfRh/Mcfy1CjeMODctwnUPZ9XM2kS5Ei+zom1gyWf3ybRwljyKpborPmOjWT3oKNOInYH6PT0WSzwyzDimRbHO5VCnB+pnfLzg4hP/z6+s3gMkv+RN4rk8WdIx0lGG1fh1gHcpnbqwFjCq5cIQn7tGWqJ+iaE+NwzX6aZFteIhrRc+CFyhisbNgA78dIgXps90dxghC46A07L+FPyUy2Q8K3b7+ZhIcqzRsp4jMb8OrhS6UpLA3qde32/LEcN3x+xjDIGnt1x6lOlzCGds4rsBz2LCWbxIl2p8htplSstwmpLyIJCcPMIs+nfabpcymV1qWjJO2yISmHz2nRgKN7+GodY9cJnNNrRxVVCGza8l0op4cWW9CQFOqNpq5NBfmIYdxzjL5E0O8CKi6igT1fvsfUq/+iu+wATwJIy/MpPWMqWlPUGR+RNwDHf6e9qMMqWlPUGR+RKIWdbx6NuPQ3d+BTqU9++yLR7lM4w4Ny3fC+j8nAgs2jjhEPDfDSetwS3kLX4nZ9Pj4nO+TK+API305HCs33VKEo87BG/UqSSXr8iLJp9+pat8prR0xysufwJ4Mrk5v4nTWSItXYriikUA9+67M2/8iPYjC+pPOOvqpqYzH31lxSOY8vr1ZFqxrq180xCwLQy3YgKFaTQuqJ+54OrvNqU5KDGLNW1CSiQdZTxlv2fwKkSE0lHijy+bG4Ge9dF+HgQ4t9LPITJNFjQBeyBaxOeiTJMFSQAtsZ4fAzjFPafMbrxw4aL33DDLwLQgy6JroR0cG/eLfp+PzwhlM/T6BxzO8fQZENv499Ah0Pch2j8Sm/D9794nhK9YkS5Rpj+5TGmRhlLCyuEVhBTsmE6VYt3TUhIULBoB23YeRINU45uU76Cf+s/X8DIBwoxjWtEurIzTEzCtZ5iAm/iZ02bP/Ur6g/BilV9CSvzHCPOlTL4zyQ/a2dBj0PUQ18Y7EQql4Ssi5UsilEWMXqEyMc+UFj0SIaxRyizkEJauIopkxj7El+gGDLu0oSfTz51chXAiLvZhRTAIUSX3rX7tEXwtvYoiDUUv/5t+RcvGUSbPXYEAj/8zdCZ0M0TnuQwqhA1BJpkcG55652a4S+h2B53b2TpaBMqfAD5Qvq+AZEor/5qHswaWYfNXl9ajC9MPqm7vxFcUw4jz0sJZ8IiWCrNe7XXtnZfhhxsew82Lu/Rmuqrm4ij/iAzmQnj4afwK+jrEYdcLoXxsK5w8FVoKYSn9TWx/xNDjz4MWQ1kvrJ066ITzZAEQyHRzkiktgKJV5iVMrvriY5nSfE5RdH4EcNcyATls5+ZiGiuGJNdCRpyXlt8FdHZOBI4485aBg6qsS23DmYgfcPB9RpyLo2xDCWZCo6GvQWOhg6HVUD52KE6mo2SPMp01IHIOdCR0EcT9bE0/3JwtqQIdl2m4O1NagS6vbDYlkOkzkClt05wUUwACzo7IhJMMYxjv+mhQDTuTCLtxbhr3ZQERGFA14Ghcamu0RQxL1HT4XTYXR8k82Hv8ALoNYk/vQci3u4q0bOx2HMSJQXTE6WwHRL7iJbyFLfezMgxq+FY8qwx0kAiUOQHM6uvvV4VMaX7nKD4/ArhZTHQIHIPL1SV+PSQR5mxxv85CfhfV2b4E0IvfA+B3xdP7jOyrfHPYNGEaos6FOPy5C7QKYm+Qvb1zoGIZK5C882XvteuZS319fRwPxM/3u3CVZdY2NTXBX5anlWPZf3LDsxlhl2OdkhUqx7L/9PeL8Ags/UfAsmL9yrFO5dwe/3vvYuO9Vc3GkJp+w/baYeTerMuG1vZha/ErO9uMHLib2iPZusFsb3l0ifHGe2uNMaMGjv7axHG+F83FUXI6M1f5oHPk7Cx+I/4f9F9QMe3fyHxbaDm0GURH7dqcOXPiCFBd1tDQ0PWt0IH1+evq6nLtNXflVcoAPzAoO+ZblJdNmdWIF5PST8riM0q1R7Dtmak98DqC2iPY5jCmzmq8BJ+OH32ybuPGh1/8oHlobf+t12zY+Cni+r+9YsO1+HzMCrhIBblcuX5fTZ1953WYQ3rsq+98+sbsW1949Orv7HteOiC5OBG88ON2T0/1MjoF232hJ7z9Qm4GIbMDvQxfwHYSxHH8k6HkMCyCsrARyOTZM6WFrR5RKY/fTQvrlyktKvUPWz3wWuvfUKZn4Bg/wTaGexXO9Wf4GS8tbEWOdHnwGeArjuyE8Tdb9/KrbC6O8o/IhE7qF15mV2DL82/z9guxSfYGOTNsvpchV/6hc2ZluJRe1pN59EUAWkGbT2/SLUamtKDLWSHXyzTTOFNaheAJvJp8bw+O8WlceDE+Di+M3qyWZVjMODct8BJV9gWdDvsB3Km8h7b4AG3guy5yLo6yBUg5y5VDrXwm+UOIr4rkO+sVWbjGXiSHdGnPQSPckGG8ju1oiD3M8dDHkEwERCALArhZTN58bnJ0prRNDlZEQQjwvb0PVq05zzGsOHqT9+02ZhgayIozjmkFuYgyyZrAgl999U18QK7BU6ENaAjfCT1VWeY4BscdB/0YmghxpipH0v4AcRbs16FiGmevLsn1ArpjzpVY/seTuV9PXu2RP1/lUP4Envvd9zg5kZ2B564+tek3J0/bn2FZiQgsbDjh4elnzHuytcbmAjp8VXETy9ZRvp1yZmqY0RekpIUq6PeFHapCRqwwmZhnSosYhvBUxzTxLrWTfuSIaTIREAHjritmcMT0+Su/3ZSWRraO8kKcvSfEVXF+m5ITVzJgr1ImAiIQQgIm3g3xG3tlWgiLHPkiHX32vYMds+UgwJ943zPLjWmzGieZTs0/7rv0y+siX/kyrWC2jvLnqN+20B4QnxH+C+KEGy4A0AnJREAEQkgAX8acLZ7WMqWlPUGReRPAEF9Nq9l8NZ5L7oKnFCNfWvoJHo2ZccNsXoy007yeTd7XUQaFJZB+SCb9Ndh7vAhK9ig5+5TPDfn8UiYCIhBOApxL4Jr3/Di5y21XWmqkwsUj0FrTeSwe4h/qmM7mlmm8h/co8VDf2ZxxTCvelZVzPgRycZRH4EK7Q3d4F7wTW85MPdfbD91Gk0dC1yQqUMAE0GvsGvHhM+LUwdbUtICLVbmXc6wvo/IDcIfynGnZpx9/4BjerXAyzwAjkVa5bEJc81wc5edQD84+bfTq82dsl0M7evvh25j6/b3wNYpKFCwBx28NZRQjU1qwpaycq9nDWVe8tbP4votPennrEbVuOFH/RFrlsCifmmb7jJI1WgTx+IegR6GdPP0R21Ca3m8PZbOoUAESwGfA9zOeKS3AIlbWpRyDS3Lu7jjmFEziacYydpiUbE5xB8ETaZXFIyS1nR6fN6il2V3DPG2JfD9EaY6+B3EcZv0q9ANoDcTh12sgmQiIQDgJ9MtQrExpGU5TUl8JdFqxWyy780uGaW2NHv03//XmSnQvTYTtFUzra746r+8EOOu4pdX5Ln7LxffzkMvQK0tyMbQrtB00CjoJaoZkIpAggKl8vigypfmepIQ8CWSasJMpLc/L6vR0BO6/eMaLeCL0ezjJ1XgwNGp9C59mOasZx7R05yiueASOPnveBLQDOn5OFXr17O2ntWwcJZeqewaa4W0Zvht60tv/GbYyEUgQyDTenSlN/ESgAghMmX37cCz/8AUMty7HItyvbTmsmkOvyxnHtApAEKoq2qZ9FAqEpjDvWVA/kz8ZmdaycZQDceZWUI23ZThVvuvj4bgSm3/npsQF0+VFIBACqbNce14wU1rPY7VfGAKWgVmvWFMUr4Y83dax8cgfTJ5gMOzGMU0WKAH8nvFIXtCyna6fb0xXgGwc5aU4kYsN3OxtGU5VHPshNY0shbRhVCwRqFAC1vasuGObCx667Bsb+ldZbjgBI5FWoWBKUm3bMd7ghfEe67HTZjfyZyPTWjaO8ts485UMmpU2Z0WKgAiEgECmUZVMaSEoegSLgMf0H7FaWGZ3z2T1kuFkWjJe2+ITsGva73NnHJvGTHwa5vtdMZtZr1wCa4CXwXhvi6laBsfTmdbmxYVuowUHQtckKlDABDC05HvFTGm+JykhLwJmlbXQ7rAPw5czXw/Z6S9PvYPejDEFX9KdFtLyylwn50ygf3tsqI3uJJY9Xok2qUUGW6TLJJse5e9wIhcVOAPiLegkaHMIU5rdX+ZOOlHshsv0axXhag+VJngC3Z5DosuS6je7pQVftIq84vwLZyy1HOsyNMV6OMgdPl7bhi9VZz3jmFaRUEpYabvTnAb+mMtj3LSg4YRd/IqSjaNMnvt5L7AcWzrMNVArNBGSiYBLIFMvPlOa8BWHABzjZz+lhVnHqc6xW1pxLq9c0xCYf8mMf9RWx75pOvbso/cbbTDMuDSHKqrIBLDOLiemcvT1X+7kV5/r5eIo2ZAcx3kWehp6H2Kvci4kEwERCCcB3sz6WaY0v3MUXwACd8VnbJx/yUmvbjm8xmC4AFkqiz4QgENbxtPwis6Re596fUEWHHgI+U2H+MCTQ7H/gX4K3QzJRMAlkGm4O1Oa8BWHAHqQ6/xyzpTmd47iRSBSBDY696I+G+AqJ241YtgjfnXLpUfJPO6HuNgAV+jhUnbsTXIYViYCIhBCApiw4/sZz5QWwqqoSCJQcALzLz9pJZ7c/xtjpRgddZKTVTe5ju+HaJMjDWMw4jj8Suf4K4jPLJsg/vSWTAREIJQE7NQfbu5xU9stLZSlV6FEoJgE3HcnTQe+zPwPRryu9LtWLo7yJGSyJ3SZl9mr2KLL6s6G9aK0qXQCmEHmiyBTmu9JSsiPAGYrpGbASQZd1iOtK14BEagQAvhwTExU1Z47v35mg1+1c3GU45DJWoi9Sdq70PPQUO7IRMAlkOnLN1Oa8BWFAO6Sq1My7uY1e6SlHKagCFQGAXwgalhT04x9mKnGuTjKRciI67ry4SeNP7t1BHQXd2QiIALhI4BefOrQa7cCZkrrdqB2RCCiBHDvzlXnMOu1c+bkOXcmX4HcpLa5OEpO4jkfGunlMhHbByE6TJkIuAQyzWzNlCZ8RSKAW+XPcu7WoXRvoz9LU0gEKo9A54ahmKDqYG0Ac7TlmP/jRyAXR3koMumE9oX4esgW0CSoBZKJgAiEkYBjpHzGsRZMqnVLS01QWAQqg8AD10xuq6mOnWXZ9tVYj+MFv1pX+SWkieeydRdCH0A3pklXlAhwMSguk9b9C9njwjQhEgEREIEwEcCCD+tRnr9RV3636efpypaLo3wPGTBDznqd7IWxMbgQwR0MyETAz0mSTKY0kSsSAdNox+1J+s8502QiIAK9Ekj/AUp/2ucQPchL+krKIZ8inK+j5PDQQMh3FZGU6ykYagJ8DtZjiK+rvD2ekXXFK1A0Ao77uCR99pnS0p+h2AIRmBi/qXpQS82E91Y1Gww/Fj9FywkWiG0xskl5ftFr9tfjCDrKnjq71zMzH3AakjnziLNpH4e2hFLtcOy8Cd3n6ZTURIXDRsDPSbKcmdLCVo/IlOezXiOHxbtX67O07vHaKyKBKefMPWRgW/XNeEBx4cJF7xkMM66Il1TWeRLIxVFyIs9UiL1HvnOyAPoWxPi+GhehvQI6CDoUegOi40y1nbBzHcRe7NHQTZAstAQy9RozpYW2QmVdMHhGPjJJGH89JBnGtltaSryCxSMw6azGOtM2fmo6Zg1uWl7ffFg1GgVhxDGteFdWzvkQyMVR7owL3QkdCXFB9L2hq6GfQX21sTiRTne1l8Fb2O7ghZObCQicCzVD86HxkCy0BHp2WlILmikt9TiFC0XAtB1+btJaprS0JygybwJVVc4U3KJYeGA/f37DzDOP+fy2fHiP7zXHSqTlfQllUAQCuTjK43F9jtwcAPEVEf6O1wpvH5s+GRcwSB3+ocP87PfzElkuw+ZH0EhoJTQbkoWUQKaZrZnSQlqd8i+WaaY8yujWoeQXdEpa+Ve1LGrgWJuxnLZhv5wsb1fYS0vGaxseArlM5nnUKzaHB7h03SioDfo71Fd7DSfyw0oHzFEh/hEthlLtN9hJ3hXfgHDXxKH6+vo4fgHh/NSDe4abmppSR5t6Jod6vxzL/pMbnvVlylmv5VinZIXKseyz/vi80dreCZ9oGlUx/IePWkcnHlbirqW6X2xMOdapnNvj2TdXGi8sWW1sM7J26qWvvWG0bOw0jvqvbZ7mpJ49tx8+1e/1hGSdw7wt57+l3rjSQWVrM3DgrRB7oe9CYyGe3wSxF8i4L0G5Gh0je4x0vg9DF0B0yntAT0L85r0WuhmiUxwLnQKltYaGBucfn47tSlvQMNPs2imjAP/o6urqyq7sU2Y12n6vgbBHufCSmbmMYoSmxcq1PabOnrsW96CDcRvq4NPa2q/KqmlvtxnG35a5bkHDCRzVKTsr1/Y4Jv6XYZ2trdegAYahRTp2Hzf8mJeXrV5gGs6nserq0++JH8u3CMrOyrU9eoL2q0cuX1odyPQl6AWIQ6CLIDqxTyA2Lj6QfbI4zuIyeEsgOtsFEJ9L8nkkLQ79EnoTOhi6EpKFlAB7Ln6WKc3vHMXnScCxOerDW1o2zAD2Jr0w/KeX5h6g/4IgQEfYiZVg0AovGZY5dtW6Ng6lvcS4cnWSQXAr9TVyGXr9MwpLFdrmIcP7oWHQci/z57Ad4YUXYjsO4lBvxhXeveO1KSmBTBN2MqWVtNDRvbhprkLl+Hzfu4PB13LC0Klx05L72gZAYGL80SqrdQXnWdThifF7wwb2/y8Mu+5httqzkXbmY/FD2CGRhYxALj3KYhZ9PTJPOsl01+ErKHKS6ciELC7ThJ1MaSGrRmSKA++4EtxpGG81Oq1Ej99mHNMiU9Eyqcigto+OQFHr0I38GF36q7+46+acnYGwUeellUlNKquYYXGUlUVdtRWBgAjAH66Gb2yHU+yELM9BdjKOaQEVQ5dJErCNvRi0HPtOvB7y0E6jh7phN9lLSx6qbXgIyFGGpy1UEhEoPAHH+QCZxig4SYtzepL7mPrKNFmABBzTdl+Hsy1zUPKyyXAyLRmvbXgI5PKMMjylVklEQASyI2Ca4+ggTdPBA2JzDV4JGda6sQM+0kScOS67THRUwQiYxj9xr3IQGmPmlNlzhz/9+kqukDTTfcOVabLACUyK3zbEaokdjM/HoX4XV4/Sj4zi+0SAX8h+J2ZK8ztH8XkScIztMMxq24a5Ctt3hw8agHe53LCNL+zt8sxdp+dIYGH9zH/AM/4Ly9bV4pWQY/FqCJewq2Wcm5Zjfjo8PwLT4vNrY61Vl+Om8ft4aL+jX26RdpTeMJNf3RVfFALWBv9sM6X5n6WUvhNAb4U3LvyNUAy9OtXtnXCTCDPOS+t75jqzDwRMp2aJWR8zrVm4cZy35/YjDIYZh+lVvjeZfbiQTsmGQEvzVByGRW/MJaZjczZyWov00Cu60mkrrciiEtiYIfdMaRlOU1JfCeAT8ArO3QHucgjeB1nNFXkYRpzlpfU1a53XRwJ33TWDs/i50MriK7/TdMvMo/ZjWFYCAnguvAvnf6NXf9f8S0569cpTuX7OphbpHuWm1VVM0Ql0uzvpcYfcLa3oJdEFXAImHWUrxBWTtlmzITGXJBHnprlH6T8RqEQCeHy/zq231ZnxeX2ke5SV2PClrjNG+jCzMmkY3ePNmhfBtGSKtkERoH+0nsaLk58isPOIwf13/nB16wLM5BnmwHcGVQpdRwTCSCBmdf7Ntq2JjmMdP2X2nWP9yqgvLj8yiu8bAcdI3nxx/V88BnO/ixnmkF8yzd3VfwEQMI1/41l9M5zk+7GY850TvziWzyjfZxya5t8BlECXEIHQErjv4pNexu38XHw54VGEuZ9fQSPtKNGT+axz40dA8QUlgJ4Lh/lo/NuKuUvAJMJYGqYrzT1A/xWfQE1rbKH3vuTYzk5zzt9edBe4Gss4N634RdAVRCDUBBZecsJt/To2fhvP8K/yK2ikHSXunDW05NfyRYrHkB7XFnWNNypogy5LTeuKVKCoBO66YkZL/1j/8/n6AZpi1IZWPKNEmHFMK+rFlbkIlAmBv1z2jY8WNpzAX69Ka5F2lGlrrMiiEsDLB5xRSaOTxJCr6yndnn1KWuII/V90AhPjN1W32+1xPCn+PBrh44HV/dAkzucZx7SiF0AXEIEIEJCjjEAjhqkKmLlTg/LwmSSXR1tb3c/9E2PY9tIQlAVFYHBr7RQ4yK3xYPJty3DOm7zPNnCU5tuMY1pQ5dB1RKCcCchRlnPrhbDsnCiCJ8PsRvJnEVYNrkEPhmH+qBPTZIESgEPczb2gY99t2LHm/lX4yDMM60oLtES6mAiUHwHNQiy/Ngt3ie3OBsOK3YxC4m/LqvtojTu3pwquE28jIE0WKAHTtDfYjjkcr4j80rbsT+95mr+Nbv2Sj48tpAVaGF1MBMqUgHqUZdpwYS12p1m1HcrG3xftgG/E6yGu8cdo13tpiRj9HwgBvCn5Gua01eFiO0Ajmtvc3wXegXFMC6QQuogIlDmBiDvKHivDlHljlUPxLdPZ35vA8x5mvT6z3eburwm9x7hEWjnUIjplNDuNsRhk/Qg3Lfxx4E9qB7ide/5o8EeJtOjUVTURgWIRiLijTHZoioVP+W5CwDQGI64/epIc2/vTHuOG84ElHocZ/RFgmixAAqZpbYHLvWM71hw8Jb7uoF02x/usCCPOSwuwNLqUCJQngYg7SvUoA/+ztI1P4RfRmTT7ObZz4NIV6zmPBzN6EOWmBV6iir6gYzpLCQC/VLHP+gHNj4zferAbZlwyjWGZCIiAP4GIO0onsXSaf/2VUngCeO5lYl1RrMJjGXu+u3IDv5gxo8eEA9UzscLjzpxje6t1LybuYOFnZ59BLbW33vboEpzg7MM4pmU+W6kiIAIkEGlHifG+NjVzsAQwzIpfabcX4xZlDbqS7w0d2B9vIyCMuERasOWp9Ks9eMWMT5zO2M/A4UE4yCqbg+AIM45p7p7+C5iAY046q7Fu6pw7j39x6WqDYfbvAy6ELpcDgYi/HuLoiyCHP4ZCHFpdYz3Q0tp5FJ5H7ooh15GtG/HTe6bB1xDeYFohrqE8ciNw/69mcIHXa6efMe8P3zhk++adJoy/NrccdHQhCUyZPe9n8IoT6RufeWOlEYsZVyLusYUNxuWFvI7yKhyBSPcoLcfkRAZZgAR2NRZ34Cdr+GByIGZaDm2ho0SYcUwLsCi6VA8CXNu1HxcckJWMwLSzG7+Az8VEFKANj+3/uvO2Q1mWNsYxjTuy8BGI6Kcm8WwSkxXcZWHChz26JVq0ccIheA3kSPQit4CDXOGuzIMw45gW3ZqrZiLQOwEnZn6RR2EEfO6CS0789cGYhcywG+el9Z6LjgiaQCQdpftzwSCJbeKJTNBUK/l6jnkMoPPT/y5enJy71/bD+a3wbiLOPKaS0ajuImDaTmIhestc2UXDC3eldSUoEBYCkXSUWFPUfTCO9/j0jDLgvzQsfM4VYPhc8u8L6k+49Et7bOWG3ahkmnuA/hOByiNgW86LrLXZaZ84bda8g5Z8uN4NMy6ZxrAsXAQi6SiTiHGH9nIyrG0wBDCBhyvA0PaYMqfxuJeWrXbD/C8ljbsyEag4ArUDYn/F+1JvQ1s5pj3r4ZfwwzoIM85Nqzgi5VHhSDtKDPd9rjyaITqlxB/UfCwtsB7sd8TA908ff2UFn8HsyDimRaemqokI5E7grviMjeaA2rPxU61X4PPx7JhRAzH4gjDimJZ7jjojCAJheT2EDnsghBejfY3Tw/A+Xg5mGsNyOFqHFoDA2uqW+fidwyPwQvsEZDeci3BjHPxd/MbW62urm+UoC8BYWZQ3gfnxac2owSPUlU1NF/xw+sEMy0JMIAw9ytPA5xWIq4Q8Dm0Jpdr+2HkVmgcthg6AsjO8k5DdgTqqUAQei5/SWl1tnoGJVBdhsHX+btsN46SqixjHtEJdR/mIgAiIQFAESt2j5OsbV0B0jnyYdQNEx3kelDT+huEc6D7oqxD33SnW2GY0PBNzJ/VkPEiJBSeAIST+zBZWgjEevKqpac5PZv43wzIREAERKEsCpe5xjQU1rhrizvjA9i0oMWsSAc+4zx4nLV16IiXN/1hjFI/HZCIgAiIgAiLQdwKl7lEOQdHbU4pPh2mn7DOYeky39Pr6+jiefZ3f4/iu3X6xmNXU1FS2zrKcy97VCAioHqk0Sh9We5S+DVJLoPZIpaFwOgK1iOQwXXKI9ByEqVR7Cjt8Tknj88mH3JDPfw0NDc7U2Y2upsxuXOpzWOij9eEJVxOpPdQexSCgv6tiUO17nn7tUeqhV87+egc6BOIM1eMgDrMOgg6EaC9Ak6AYdDKUHIZF0Mcw5dL9ZzgrfI5QtAiIgAiIgAhkRaDUQ68sZBy6B+KC2Zz1ugDaC5oPjYCughj/XYi/QpHsXSKY3jCFZy0W5o+ZjvlR+iMUKwIiIAIiIALZEQiDo+RrH/dD7FEu94r9HLZ0krTXodHQGIhDqb0+c8QBeMvdxC9W2E/geJkIiIAIiIAI9JlAGBwlC8/nlJSfsbe5xC+xZ7zDVdFN42Xb6WzsmaZ9ERABERABEciFQKmfUeZS1qyPtUyzGtODXnvg0q8le6hZn6sDRUAEREAERCCVQCQdpW2YVXjJZOLUOfN6fZ6ZCkNhERABERABEehJIJKOEo8x+drJWAzBTu9ZYe2LgAiIgAiIQC4EwvKMMpcy93osfo5ykIEpr6bh7NnrwTpABERABERABDIQiGqPkgsYoG7ejwhnAKAkERABERABEchEIKKO0lsGz0HPUiYCIiACIiACeRCIqqPciFXxuCJ66jqyeWDSqSIgAiIgApVKIKqOcgCGXfFjlOa/K7VhVW8REAEREIHCEIimo3T4fNLssE3nusJgUi4iIAIiIAKVSiCSs17RmKuxKvpaLGOHnqVMBERABERABPpOIJo9SrwfgmmvlmE54/uORmeKgAiIgAiIgPsKRfQwwEvCT5pbWI65XfRqpxqJgAiIgAgESSCaPUrH+JQQbcfhL47IREAEREAERKDPBKLpKA1jHfqULYmeZZ/Z6EQREAEREAER4OzQSBrm8jhtpmO8GMnaqVIiIAIiIAKBEYjkrFcsNLAFnlF+2tbZrz4wkrqQCIiACIhAJAlEskfpGOYd/durjnzosuM+imSrqVIiIAIiIAKBEYhkj3Jhwwk/C4ygLiQCIiACIhBpApHsUUa6xVQ5ERABERCBQAlE0lFOntO4Yzwej2TdAv3r0MVEQAREQASiOevVcozLn22dcBYWRufvUspEQAREQAREoM8EItnrwqzX7UzHPHzKrMb9+0xGJ4qACIiACIgACETSUaJeG7HgwM6WaR2lVhYBERABERCBfAhE0lFioYGXASWG327+XD5wdK4IiIAIiIAIRNJR4snkCDYthmDb1MQiIAIiIAIikA+BSDpKzODZF1DaTNt4Mh84OlcEREAEREAEIuko0aw1jmladsx4VE0sAiIgAiIgAvkQCIujZDkG51ORbueaxlIsir7Usp19usVrRwREQAREQARyJBAGR3kayvwKdC/0OLQllGqHY+dN6D5Pp6Qmpgt3dho/xPDrGkzq3T5duuJEQAREQAREIFsCpV7rtR8KegVE57gaugGi4zwPStpOCFwHXQO1JyMzbatMYw9M5MFkHkeLomcCpTQREAEREIFeCZS6RzkWJfwQopOkvQXt4IY++28CgudCzdB8aDyU0TDrdQocZadZZS3MeKASRUAEREAERKAXAqVwlBNRpts90Smm9hLpMG0o1ZZh50fQSGglNBvKbKaxLmaav5p/4YylmQ9UqgiIgAiIgAhkJoBHeYHbHrjiFO+qt2L7GsSJPBwtPQeiXZzYuP/X4n/2JmkHQHdAYyGjvr4+bprm+Qyn2nFfOd6IWaWoWmopFBYBERABESg3AnV1daF0HosB8lBoGLQImgYNgg6EaM9CJ7shw6BTvMkLp900NDQ4aRPKLLKpqUn1CFGbqT1C1BgoitpD7VEMAn5/V6UYeu1Zvzgi7oGWQO9CCyA+l+TzSFoc+iX0JnQwdCUkEwEREAEREIFACJR61isrOQ+6H2KPcjlEew5yl6HDlhNyxkGjIE78kYmACIiACIhAYATC4ChZ2fWe/CreiQQ5ST86ihcBERABESgagTAMvRatcspYBERABERABPIlIEeZL0GdLwIiIAIiEGkCcpSRbl5VTgREQAREIF8CcpT5EtT5IiACIiACkSYgRxnp5lXlREAEREAE8iUgR5kvQZ0vAiIgAiIQaQJylJFuXlVOBERABEQgXwJylPkS1PkiIAIiIAKRJiBHGenmVeVEQAREQATyJSBHmS9BnS8CIiACIhBpAnKUkW5eVU4EREAERCBfAnKU+RLU+SIgAiIgApEmIEcZ6eZV5URABERABPIlIEeZL0GdLwIiIAIiEGkCcpSRbl5VTgREQAREIF8CcpT5EtT5IiACIiACkSYgRxnp5lXlREAEREAE8iUgR5kvQZ0vAiIgAiIQaQJylJFuXlVOBERABEQgXwJylPkS1PkiIAIiIAKRJiBHGenmVeVEQAREQATyJSBHmS9BnS8CIiACIhBpAnKUkW5eVU4EREAERCBfAnKU+RLU+SIgAiIgApEmIEcZ6eZV5URABERABPIlIEeZL0GdLwIiIAIiEGkCcpSRbl5VTgREQAREIF8CcpT5EtT5IiACIiACkSYQJkc5BKTNDLSHZkhTkgiIgAiIgAgUhUAYHOVWqNlUaCm0ZZpa7o+4V6F50GLoAEgmAiIgAiIgAoEQCIOjPBQ1paNkjzKdNSByDnQkdBHEfZkIiIAIiIAIBEIgDI7ydtT0+1CbT413QPwrXtpb2HJfJgIiIAIiIAKBEMj0TLBYBZiIjL/rZT4L2+VeeD2246EPvP3kZh0CO0M8bkfoEWg0ZNTX1z9mmuZ/MywTAREQAREQgXwI1NbWGj/+8Y9L4Rc3KfYeiDnH02YpqXSUfF7Z055CBJ9T0vh88iE35PNfQ0OD45NUVtGqR7iaS+2h9igGAf1dFYNq3/P0a4+qvmfZ5zNfwplUJhuERDrUJ6EXoEnQM9DJUHIYFkGZCIiACIiACBSXQCkcZaYaJXuDE3DQfGgEdBX0OMTh2g1QsneJoEwEREAEREAEiksgTI6SvcikPYcAnSTtdYjPJMdAS6GkM0VQJgIiIAIiIALFJRAmR5mpph1IXJLpgGSa4zjsfZa9oR4XlH0lUAHVI1ytqPZQexSDgP6uikFVeYqACIiACIiACIiACIiACIiACIhAmAlwMYXBvRQwLOvHZlPWXqoSmuSorNnbWz1CAzxDQfhoJfms3++wsHwG/MrH+Gzqken8sKQNQEEG9lKYcmiPbOrRSzUDTebfT7rlUZOFKAfmybIWdHsacuP6sFycgM8se0LizFmmPwiVev3Y3sp6OEchINoAAAbOSURBVMr4JnSfp1OwDaNthUJxKcJVEMM9LUzMe5Ytdb+3epRLe5yLSr0B3QU9AO0GpVq5tEdv9SiX9uDs/eehuRDbo+dNfLm0R2/1CGN7/Aa8081dKRfmKH7hrR+ybIOGe1nfgO0vvHByQ2hHeztfxfaJZELA22zKejrK9DOIx4bZyPG3UDu0VZqChoV5mqJ1i+qtHuXQHrzj5wS45JfxRQj/oVstE18cYfgM9ChWt91s6lEO7cF2eCGlZs8iPCNln8Fy+HxkU4+wtQc5/5/Hl5xTrRyYp5a3oOHxyO3tlBy5TN4dKfsMvgeN8+I+j+37XjjoTTZlvRaFWgXRAfHdUp4TZvNbYSkszLNl51ePcmmPkV5FOdS3CPqmt5/clEt79FaPcmkPct8TYu/mQ2gzKNXKpT1Y5kz1CFN7cMlTOsPDvS023Sxr5nw2FjXjsyU6laStRsBO7njb1GPSpfc4vGi7qeXgRdKVZRnifwTxC2MlNBsqR0uta7p6lkudlqGg5dAevLn6AsTeyzsQh/xSrVzao7d6LEOlyqE9yJ7ft7VQP2hfKNXKpT1Y5kz1WIb0MLRHDcrBDtKPoVR/gN0uKyfmXYUuVIB/iOsh08swua5sav5PYYfj07Re149NHFaU/7MpK49JGsu6LLkT0i3Zpxt6DQvzbLH51aNc2mMyKroC4rKP6axc2qO3epRDe2yHBjghpREuR/iPKfsMlkN7ZFOPsLTHNDBtgZ6H3oT4eb4FSrWsmfPOIGrWjAq9Ax0CDYOOg16BBkEHQrQXoElQDOIXCdNLYdmUlUMHyS87DiE8WoqC9vGaYWTel6qk1qMc2oM3ibdCX4FuTqlwaj3C8hlIKd4mwWzqUQ7twaU3r4W4uhi/c8dB5F9u7ZFNPcLSHvye3Bv6GvQL6D/QeVC5MUeRi2d8gLsW+gT6C8QPHKFxnzYB4nOCD6C3oFFQqay3sk5Bwej4eVf0MLQHFGbjnduWXgHDyjwbfn71KIf22AkVdCA+ckiKQ6/l1h7Z1KMc2gPo3R+cX4ktn4vxczwSKrf2QJF7rUcY2+MglJsOnFaOzBMlL9L/vHMYnSFvvluzPUQnWmrrrazs+SadT6nLms/1w8Q8n3qoPfKhV/hzy6U9OCy5WYbql8vno7d6lEt7sCnKhXmGPxsliYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI9EKA77vIREAESk9gLxThh9BS6NPSF0clEAEREAEREIFwEfgWisPVdA4JV7FUGhEQAREQAREoPQH+HBCXKKSjfAXi0m0LoFugu6DfQxdBL0LJRaf5G3t/hGg7Q9dBXJbxaYi/GtLTuM7xfRCXcXwUOgyiDYSuh5ZDXH+U62HSeB3GfwStgXju1hCtZ9m4HBvPa4Jehb4DyURABERABESgYAS4rNltEB3lVRDXHqbT4T7XLD4bSqYPRphGp/ikGzKMf2L7PnQBtAxaByWPQ9BdpvENbKmTocUQHXMV9D2I12FeXIOU4YOh073wn7H9LdQB3QjRepbtYsQx/RLoCYh5TIRkIiACIiACIlAwAj2HXumMVkP8BRyan6McgzQbeg66DHoKoqP6MpRqdJKdENPp0Oog2kPQBxDnKwyB+HNQXHifzncVZEE0OkA+Ox0A9SzbW4hbAfH68yBe/wpIJgKRIJD8EESiMqqECESMwDLUJzmxh86Qxl4gjUOmtB0gLuxPJ8gtHdz/Qu9AqXYUdthbZe+VPVQ6Vi7Gvy3En0/i+W0QnR4d5FCoGUpedz3C1VB/iLYMYtm4Pxbicbz+uxCvT4csE4FIEJCjjEQzqhIRIsCfAarx6pN0Utxd5sWdg+0NEJ0cjc80WyH2PBd620Ow/RhK2ggEnoF2hKZDHE6lI9wemu9tL8a2HloEHQw1QqOhX0CnQROhByAO69KSZduI8EsQHfAT0CfQoRCdrkwEREAEREAECkZgN+TEHhqHLfeFmiD2+pK2MwJ8Lsn01yD2+v4B0ej8XoaYxmeF50I9bTYiODzKY3gd9i5p46ElEOOpuyH2EreE6FyT8e8gvCdE61k2DtX+DWKvlMffAw2AZCIgAiIgAiJQUAJ0UCMhv/ebObRJB2ZB6Wx7RCZ7munSeT6fTfbMn/t0mFtAPY29Sg7v+l0z9XiWbVRqhMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUGIC/x9WAzmM/hB6wgAAAABJRU5ErkJggg=="
     },
     "metadata": {
      "jupyter-vega": "#1d18947e-20bb-4f91-8e3d-6d0284027ee1"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import altair as alt\n",
    "alt.renderers.enable('notebook')\n",
    "\n",
    "alt.Chart(plot_data.sample(1000)).encode(x=\"true score\", y=\"predicted score\").mark_point().interactive()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that the true scores are discrete, not continuous, hence the vertical lines on the graph at the integer values 0 to 4. Predicted scores are clearly tending towards the sample mean  - no predictions above 2 are made, despite the data holding true scores of 3 and 4. \n",
    "\n",
    "In the next notebook we look at better ways to evaluate models."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}