02-metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the Models\n",
    "\n",
    "In order to evaluate the performance of any model we need a distance metric. In this notebook we work through some commonly used distance metrics for evaluating learning to rank models. \n",
    "This notebook covers: \n",
    "\n",
    "1. mean squared error (MSE)\n",
    "2. pairwise difference (PD)\n",
    "3. Discounted Cumulative Gain (DCG)\n",
    "4. Normalised Discounted Cumulative Gain (nDCG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by loading in the data and fitting a basic linear regression model which we then use to make predictions for the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "### We go ahead and load in the data, transform it, and build a model. \n",
    "### We make predictions for the data in the test set.\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from learningtorank import process\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "train = pd.read_csv(\"data/train.txt\", header=None, sep=\" \")\n",
    "test = pd.read_csv(\"data/test.txt\", header=None, sep=\" \")\n",
    "\n",
    "train_df = process.df_transform(train)\n",
    "test_df = process.df_transform(test)\n",
    "\n",
    "X = train_df[train_df.columns[2:]]\n",
    "y = train_df[0]\n",
    "reg = LinearRegression().fit(X, y)\n",
    "\n",
    "X_test = test_df[test_df.columns[2:]]\n",
    "y_test = test_df[0]\n",
    "\n",
    "preds = reg.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mean Squared Error\n",
    "\n",
    "Mean squared error (MSE) is often used to comapare the difference between true numeric values and predicted values. \n",
    "\n",
    "In the case of the MSLR Web-10K data set, each query document pair has a true relevance $t$ score between 0 and four, and we can use a model to obtain a predicted relevance score $p$. As such, mean squared error can be computed as the mean of $(t-p)^2$, across document-query pairs in the testing set.\n",
    "\n",
    "We use the mean squared error function from the sci-kit learn module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6055521003389398"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "mean_squared_error(preds, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we think about a user of a search engine, they don't ever see that predicted value. They are instead presented with an ordered set of documents relating to their query. As such, mean squared error, which only takes into account the numeric values, is perhaps not the best distance metric for learning to rank."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pairwise difference\n",
    "\n",
    "Instead of considering the predicted and true numeric values associated with the query document pairs, pairwise difference considers only the ordering of the documents. Pariwise difference counts the number of \"swaps\" of neighbouring documents that have to be made to go from the predicted ordering to the true optimal ordering. The orderings are determined by ranking according to the predicted and true numeric values. \n",
    "\n",
    "Pairwise difference is computed for each query individually. The mean pariwise difference across all queries can then be computed and used to compare models. \n",
    "\n",
    "To compute the pairwsie difference, we use the bubble sort algorithm to transform the documents from the predicted order to the true optimal order, and count the nubmer of swaps implemented in the algorithm. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to determine the Pairwise Difference we work with each of the queries individually. \n",
    "\n",
    "Once the data is split by query we need to "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds_df = pd.DataFrame({'qid': test_df[1], 'truth' :test_df[0], 'predicted' : preds })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>qid</th>\n",
       "      <th>truth</th>\n",
       "      <th>predicted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>135905</th>\n",
       "      <td>16588</td>\n",
       "      <td>1</td>\n",
       "      <td>0.945433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15213</th>\n",
       "      <td>1993</td>\n",
       "      <td>1</td>\n",
       "      <td>0.964558</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199799</th>\n",
       "      <td>23323</td>\n",
       "      <td>0</td>\n",
       "      <td>0.869287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>140923</th>\n",
       "      <td>17188</td>\n",
       "      <td>0</td>\n",
       "      <td>0.686114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69636</th>\n",
       "      <td>8938</td>\n",
       "      <td>0</td>\n",
       "      <td>0.144627</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20160</th>\n",
       "      <td>2638</td>\n",
       "      <td>0</td>\n",
       "      <td>0.856185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105088</th>\n",
       "      <td>13078</td>\n",
       "      <td>0</td>\n",
       "      <td>1.065212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>163610</th>\n",
       "      <td>19483</td>\n",
       "      <td>0</td>\n",
       "      <td>0.686787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>184150</th>\n",
       "      <td>21718</td>\n",
       "      <td>0</td>\n",
       "      <td>0.623244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200571</th>\n",
       "      <td>23443</td>\n",
       "      <td>1</td>\n",
       "      <td>1.081383</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          qid  truth  predicted\n",
       "135905  16588      1   0.945433\n",
       "15213    1993      1   0.964558\n",
       "199799  23323      0   0.869287\n",
       "140923  17188      0   0.686114\n",
       "69636    8938      0   0.144627\n",
       "20160    2638      0   0.856185\n",
       "105088  13078      0   1.065212\n",
       "163610  19483      0   0.686787\n",
       "184150  21718      0   0.623244\n",
       "200571  23443      1   1.081383"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds_df.sample(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "groups = preds_df.groupby('qid')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "514\n"
     ]
    }
   ],
   "source": [
    "print(groups.size().min())\n",
    "print(groups.size().max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows that, for atleast one query, there is only 1 document for which a prediction was made. For such queries the list will always be in optimal order. We can either remove the contribution of these queries to the metric, or we can just leave them since they have the same effect on the metric of any odel considered. For now we keep them. \n",
    "\n",
    "Let's work with a specific query to compute the pairwise difference "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "qid_13 = preds_df[preds_df.qid =='13']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>qid</th>\n",
       "      <th>truth</th>\n",
       "      <th>predicted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "      <td>0.769230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>0.293494</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13</td>\n",
       "      <td>3</td>\n",
       "      <td>0.352380</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>0.634864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0.561271</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  qid  truth  predicted\n",
       "0  13      2   0.769230\n",
       "1  13      1   0.293494\n",
       "2  13      3   0.352380\n",
       "3  13      1   0.634864\n",
       "4  13      0   0.561271"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qid_13.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing we want to do is sort the list by \"predicted\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>qid</th>\n",
       "      <th>truth</th>\n",
       "      <th>predicted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0.140340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>0.293494</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129</th>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0.312027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>0.348857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13</td>\n",
       "      <td>3</td>\n",
       "      <td>0.352380</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    qid  truth  predicted\n",
       "20   13      0   0.140340\n",
       "1    13      1   0.293494\n",
       "129  13      0   0.312027\n",
       "6    13      1   0.348857\n",
       "2    13      3   0.352380"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qid_13.sort_values(by = \"predicted\").head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now want to count how many pairwise swaps have to be done by adjacently ordered. To do this we can consider simply the list of true values, in the order sorted by prediction like above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "truth_by_pred = qid_13.sort_values(by = \"predicted\").truth.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bubble_sort(vals_list):\n",
    "    swaps = 0\n",
    "    n=len(vals_list)\n",
    "    sorted = 0\n",
    "    while sorted == 0:\n",
    "        swaps_this_pass = 0\n",
    "        for i in range(0, n-1):\n",
    "\n",
    "            if vals_list[i]>vals_list[i+1]:\n",
    "                vals_list[i], vals_list[i+1] = vals_list[i+1], vals_list[i]\n",
    "                swaps_this_pass = swaps_this_pass + 1\n",
    "\n",
    "        if swaps_this_pass==0:\n",
    "            sorted=1\n",
    "        swaps = swaps + swaps_this_pass        \n",
    "        n = n-1 #ith pass of bubble sort puts the nth value in order.\n",
    "            #so there is no need to consider this. \n",
    "    return(swaps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2641"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bubble_sort(truth_by_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to write a function which computes this number of pairwise swaps for each for each of the queries. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bubble_swaps(group):\n",
    "    truth_by_pred = group.sort_values(by = \"predicted\").truth.tolist()\n",
    "    swaps = bubble_sort(truth_by_pred)\n",
    "    return swaps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "pairwise = groups.apply(bubble_swaps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "qid\n",
       "13633       0\n",
       "13543     571\n",
       "13618    1078\n",
       "7993     2184\n",
       "9718     2633\n",
       "14743       5\n",
       "7573     1056\n",
       "29968     199\n",
       "23728     177\n",
       "17173     972\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pairwise.sample(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2130.554"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pairwise.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two aspects of ranking which are not caputed by the pairwise difference metric: \n",
    "1. we care about ordering at the start of the document list more than at the end\n",
    "\n",
    "2. we care about ordering of documents less when the true values are similar.  \n",
    "\n",
    "\n",
    "Pairwise difference weights the contribution of every \"swap\" equally in the distance metric: An out of order pair of documents the end of the list contribute the same amount of weight to the distance metric as an out of order pair at the start. \n",
    "\n",
    "\n",
    "However, if we think about a user of a search engine, they care much more about the order of the list of documents on the first page of search results than they do about the order on the last page. \n",
    "\n",
    "Discounted Cumulative Gain, or dcg, downweights the contributions of documents to the distance metric as you move down the list logarithmically, thereby accounting for the fact that we care more about the start of the list than the end. \n",
    "\n",
    "\n",
    "If we consider the following two lists of three documents. Their true values are: \n",
    "{10,7,9} and {10,8,9}, their pairwise difference from the truth is 1 for both lists, since we must swap the second and third document to get the either list to optimal order. \n",
    "However, ideally we would want to penalise 10,7,9 more than 10,8,9. \n",
    "\n",
    "The metric that fits all our needs is discounted cumulative gain, or dcg. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "def dcg(ordered_data):\n",
    "    \"\"\"discounted cumulative gain\"\"\"\n",
    "    n = len(ordered_data)\n",
    "    if sum(ordered_data)==0:\n",
    "        return 0\n",
    "    else:\n",
    "        indexloop = range(0, n)\n",
    "        DCG = 0\n",
    "        for index in indexloop:\n",
    "            current_ratio=(2**(ordered_data[index])-1)*(math.log((float(index)+2), 2)**(-1))\n",
    "            DCG = DCG + current_ratio\n",
    "        return(DCG)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30.261924410467387"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dcg(truth_by_pred) #compute for one query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sort_dcg(unordered_group):\n",
    "    ordered_group = unordered_group.sort_values(by = \"predicted\").truth.tolist()\n",
    "    return dcg(ordered_group)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "dcg_score = groups.apply(sort_dcg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "qid\n",
       "29278    20.979624\n",
       "29803    33.537069\n",
       "27448     0.000000\n",
       "11218     1.896389\n",
       "3208     12.793771\n",
       "16378    18.896233\n",
       "3598     14.914728\n",
       "2068     14.234168\n",
       "4948     35.064786\n",
       "193      33.968572\n",
       "dtype: float64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dcg_score.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although this distance metric has a couple of features we are happy with, it's hard to compare the numerical values across queries, since they all have different numbers of documents in them. \n",
    "\n",
    "Normalised discounted cumulative gain, or ndcg, "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\" this returns 0 if all of the ordered data is undesirable\"\"\"\n",
    "def ndcg_p(ordered_data, p):\n",
    "    \"\"\"normalised discounted cumulative gain\"\"\"\n",
    "    if sum(ordered_data)==0:\n",
    "        return 0\n",
    "    else:\n",
    "        indexloop = range(0, p)\n",
    "        DCG_p = 0\n",
    "        for index in indexloop:\n",
    "            current_ratio=(2**(ordered_data[index])-1)*(math.log((float(index)+2), 2)**(-1))\n",
    "            DCG_p = DCG_p + current_ratio\n",
    "        sorted_data= sorted(ordered_data,reverse=True)\n",
    "        n = len(ordered_data)\n",
    "        indexloop = range(0, n)\n",
    "        iDCG_p = 0\n",
    "        for index in indexloop:\n",
    "            current_ratio=(2**(sorted_data[index])-1)*((math.log((index+2), 2))**(-1))\n",
    "            iDCG_p = iDCG_p + current_ratio\n",
    "        return(DCG_p/iDCG_p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "truth_by_pred = qid_13.sort_values(by = \"predicted\").truth.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "try_ndgc=ndcg_p(truth_by_pred, 120)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5495521971812585"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "try_ndgc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 3,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 3,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 2,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 2,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 3,\n",
       " 2,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 2,\n",
       " 1,\n",
       " 1,\n",
       " 2,\n",
       " 1,\n",
       " 0,\n",
       " 2,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 2,\n",
       " 1,\n",
       " 1,\n",
       " 2,\n",
       " 1,\n",
       " 2,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 3,\n",
       " 1,\n",
       " 2,\n",
       " 2,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 2,\n",
       " 1,\n",
       " 0,\n",
       " 2,\n",
       " 0,\n",
       " 2,\n",
       " 0,\n",
       " 1,\n",
       " 2,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 2,\n",
       " 2,\n",
       " 1,\n",
       " 2,\n",
       " 2,\n",
       " 3,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 2,\n",
       " 2,\n",
       " 0,\n",
       " 1,\n",
       " 2,\n",
       " 2,\n",
       " 1,\n",
       " 2,\n",
       " 0,\n",
       " 3,\n",
       " 1,\n",
       " 1,\n",
       " 2,\n",
       " 2,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 3,\n",
       " 2,\n",
       " 0,\n",
       " 1,\n",
       " 2,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 3,\n",
       " 1,\n",
       " 1,\n",
       " 2,\n",
       " 2,\n",
       " 1,\n",
       " 2,\n",
       " 0]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "truth_by_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_at_p(ordered_data, p): \n",
    "    \"\"\"\n",
    "    returns -1 if number of items in list is less than p\n",
    "    else returns ndcg@p\n",
    "    \"\"\"\n",
    "    n = len(ordered_data)\n",
    "    if n<p:\n",
    "        result = np.nan\n",
    "    else:\n",
    "        result = ndcg_p(ordered_data, p)\n",
    "    \n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sort_ndcg(unordered_group, p):\n",
    "    ordered_group = unordered_group.sort_values(by = \"predicted\").truth.tolist()\n",
    "    return compute_at_p(ordered_group, p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5495521971812585"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sort_ndcg(qid_13, 120)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4420443998743764"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "compute_at_p(truth_by_pred, 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "ndcg_score = groups.apply(sort_ndcg, p=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "qid\n",
       "27838         NaN\n",
       "3118          NaN\n",
       "13573    0.441229\n",
       "15748    0.428469\n",
       "5833     0.419832\n",
       "20938    0.298734\n",
       "16183         NaN\n",
       "29908    0.298554\n",
       "25903    0.476342\n",
       "13318    0.324499\n",
       "dtype: float64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ndcg_score.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NaN represents that there was not enough documents considered for that query to compute the ndcg at the requested level, p. \n",
    "\n",
    "In order to summaries all these numbers into one we can use the `np.nanmean()` function, which ignores any `NaN` entries when computing the mean: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.34106516621584887"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.nanmean(ndcg_score) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}