03-models-lambdamart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we train and compare three different learning to rank models. \n",
    "\n",
    "Each of the models are built using the [XGBoost](https://xgboost.readthedocs.io/en/latest/index.html) package, which is optimised for gradient boosting. \n",
    "\n",
    "We models we build are: RankNet, LambdaNet and LambdaRank. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xgboost as xgb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data processing\n",
    "\n",
    "In order to implement the XGBoost algorithms the data must be preprocessed in a particular way - XGBoost requires an file that indicates which query each entry in the data frame comes from. If your training file is `train.txt`, the same folder should contain a file named `train.txt.group`, holding the group information. `train.txt.group` will contain a list of numbers indicating the number of feature vectors for each query, in the same order as the appear in `train.txt`. For example, if `train.txt.group` contained [7,8,5] we would know that there are 20 feature vectors in the training set and that the first 7 correspond to one query, the next 8 correspond to a different query and the final five are from a third query. \n",
    "\n",
    "There's a bash script in the repo which should be run to compute the group information file for your `train.txt` file. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load in the data as a DMatrix, an xgboost data store which is optimised for efficiency and speed when implementing xgboost models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[16:11:30] 723412x137 matrix with 98384032 entries loaded from data/train_dat.txt\n",
      "[16:11:30] 6000 groups are loaded from data/train_dat.txt.group\n",
      "[16:11:31] 241521x137 matrix with 32846856 entries loaded from data/test_dat.txt\n",
      "[16:11:31] 2000 groups are loaded from data/test_dat.txt.group\n"
     ]
    }
   ],
   "source": [
    "training_data = xgb.DMatrix('data/train_dat.txt')\n",
    "testing_data = xgb.DMatrix('data/test_dat.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When XGBoost loads in the data, it also checks the folder containing the data for a file of the same name with file extension .txt.group. The .txt.group file contains query id information for the data in the .txt data set. We can see from the logging info printed when we load in the data that the train_dat file contains data from 6000 different queries, and the test_dat.txt contains data from 2000 queries. XGBoost uses this query information both for building models and for evaluating the goodness of the models built.\n",
    "We can also see from the logging that the data loaded in is of length 137 - that is the 136-long feature vector and the one \"score\" which is given by the human in the preprocessing stage. In this data set, the score is an integer from 0 to 4, with 0 denoting \"this article is not relevant\" and 4 denoting \"this article is extrememly relevent\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LambdaRank"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15.848957777023315\n"
     ]
    }
   ],
   "source": [
    "params_lm2 = [('objective','rank:ndcg'),('max_depth',2), ('eta',0.1), ('num_boost_round',4), ('seed',404)]\n",
    "\n",
    "start_lm2 = time.time()           \n",
    "model_lm2 = xgb.train(params_lm2, training_data)\n",
    "end_lm2 = time.time()\n",
    "print(end_lm2-start_lm2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.283322811126709\n"
     ]
    }
   ],
   "source": [
    "start_lm2p = time.time()\n",
    "pred_lm2 = model_lm2.predict(testing_data)\n",
    "end_lm2p = time.time()\n",
    "print(end_lm2p-start_lm2p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.61660594, 0.37398714, 0.37398714, ..., 0.61660594, 0.3965544 ,\n",
       "       0.61660594], dtype=float32)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_lm2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30.50355887413025\n"
     ]
    }
   ],
   "source": [
    "params_lm6 = [('objective','rank:ndcg'),('max_depth',6), ('eta',0.1), ('num_boost_round',4), ('seed',404)]\n",
    "\n",
    "start_lm6 = time.time()           \n",
    "model_lm6 = xgb.train(params_lm6, training_data)\n",
    "end_lm6 = time.time()\n",
    "print(end_lm6-start_lm6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.2156538963317871\n"
     ]
    }
   ],
   "source": [
    "start_lm6p = time.time()\n",
    "pred_lm6 = model_lm6.predict(testing_data)\n",
    "end_lm6p = time.time()\n",
    "print(end_lm6p-start_lm6p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We write the predictions to file. Later we will read them in and compare them across all the models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "pd.DataFrame(pred_lm6).to_csv(\"data/pred_lm6.txt\", header=None, sep=\" \")\n",
    "pd.DataFrame(pred_lm2).to_csv(\"data/pred_lm2.txt\", header=None, sep=\" \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = [('objective','rank:ndcg'),\n",
    "          ('max_depth',2), ('eta',0.1), ('num_boost_round',4)]\n",
    "\n",
    "model = xgb.train(params, training_data)\n",
    "\n",
    "predictions = model.predict(testing_data)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}