Implicit_ALS_books.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implicit ALS\n",
    "\n",
    "We usually consider using ALS on a set of user/product ratings. But what if the data isn't so self explanatory?\n",
    "\n",
    "### A day trip to the library\n",
    "Consider, for example, the data collected by a local library. The library records which users took out each books and how long they kept the books before returning them. \n",
    "\n",
    "As such, we have no explicit indication that a user liked or disliked the books they took out - Just because you borrowed a book does not mean that you enjoyed it, or even read it.\n",
    "Furthermore, the missing data is of interest - the fact that a user has not taken out a specific book could indicate that they dislike that genre, or that they haven't been to that section of the library.\n",
    "\n",
    "Furthermore the same user action could have many different causes. Suppose you withdraw a book three times. That might indicate that you loved the book, but it may also indicate that the book doesn't appeal to you as strongly as some other books you withdrew so you never got round to reading it the first two times.\n",
    "\n",
    "To make the situation even worse, implicit data is often dirty. For example, a user may withdraw a library book for their child using their account, or they may accidentally pick up a book that was sitting on the counter. \n",
    "\n",
    "### The solution\n",
    "Based on the standard ALS implementation, [Hu et al. (2008)](https://www.google.ca/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwi899eAu6baAhUurlkKHaVvB6UQFggsMAA&url=http%3A%2F%2Fyifanhu.net%2FPUB%2Fcf.pdf&usg=AOvVaw3WIcPGTpxR8m7C32F8whFx) presented a methodolgy for carrying out ALS when dealing with implicit data. \n",
    "\n",
    "The general idea is that we have some recorded observations $r_{u,i}$ denoting the level of interaction user $u$ had with product $i$. For example, if a user $1$ borrowed book $4$ once we may set $r_{1,4}=1$. Alternatively we may wish to allow $r_{u,i}$ to hold information about how many days the book was borrowed for. (There is a lot of freedom in this set up, so we need to make some data specific decisions regarding how we will select $r_{u,i}$).\n",
    "\n",
    "Given the set of observations $r_{u,i}$, a binary indicator $p_{u,i}$ is introduced where:\n",
    "\n",
    "$ p_{u,i} = \\begin{cases} 1 & \\mbox{if } r_{i,j}>0 \\\\\n",
    "0 & \\mbox{otherwise.} \\end{cases} $\n",
    "\n",
    "\n",
    "A confidence parameter $\\alpha$ lets the user determine how much importance they wish to place on the recorded $r_{u,i}$. This leads to the introduction of $c_{u,i}$ which we take to be the confidence we have in the strength of user $u$'s reaction to product $i$: \n",
    "$c_{u,i} = 1 + \\alpha r_{u,i}$.\n",
    "\n",
    "Let $N_u$ denote the number of users, and $N_p$ denote the number of products. Let $k\\in \\mathbb{R}^+$ be a user defined number of factors. \n",
    "Now, in implicit ALS the goal is to find matrices $X\\in \\mathbb{R}^{N_u \\times k}$ and $Y\\in \\mathbb{R}^{N_p \\times k}$ such that the following cost function is minimised:\n",
    "\n",
    "$\\sum_{u,i} c_{u,i}(p_{u,i}-X_u^T Y_i)^2 + \\lambda (\\sum_u \\| X_u\\|^2 + \\sum_{i} \\| y_u\\|^2), $\n",
    "\n",
    "\n",
    "where\n",
    "$X_u$ is the $u$th row of X, \n",
    "$Y_i$ is the $i$th row of Y,\n",
    "\\lambda is a user defined parameter which prevents overfitting. \n",
    "\n",
    "With this minimisation at hand, we are able to recover estimates of $c_{u,i}$, and thus of $r_{u,i}$ for interactions which have not yet occured. \n",
    "\n",
    "### Let's get going\n",
    "We are going to run implicit ALS using the implementation given in the pyspark.mllib.recommendation module. \n",
    "\n",
    "The data we will be using can be found at http://www2.informatik.uni-freiburg.de/~cziegler/BX/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Set up a spark context\n",
    "\n",
    "from pyspark import SparkContext,  SparkConf\n",
    "\n",
    "conf = SparkConf().setAppName(\"implicitALS\")\n",
    "sc = SparkContext(conf=conf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Data\n",
    "\n",
    "\n",
    "In the cell below, we download and unzip the data. The two files we are interested in are BX-Books.csv and BX-Book-Ratings.csv, which follow these schema: \n",
    "\n",
    "### BX-Books.csv\n",
    "| Field Name |  Type | Description |\n",
    "|------------|------|\n",
    "|ISBN |  String | length 10, alphanumeric |\n",
    "| Book-Title | String | Title of book |\n",
    "|Book-Author | String| Name of author |\n",
    "| Year-Of-Publication | String | yyyy|\n",
    "|Publisher| String |Name of publisher |\n",
    "|Image-URL-S | String| URL for small image on amazon.com |\n",
    "|Image-URL-M | String| URL for medium image on amazon.com |\n",
    "|Image-URL-L | String| URL for large image on amazon.com|\n",
    "\n",
    "\n",
    "### BX-Book-Ratings.csv\n",
    "| Field Name |  Type | Description |\n",
    "|------------|------|\n",
    "|User-ID |  Integer | Range from 2 to 278854 |\n",
    "| ISBN | String| length 10, alphanumeric |\n",
    "|Book-Rating| Integer | 1-10 denotes dislike-like. 0 denotes implicit interaction|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2018-04-09 15:29:29--  http://www2.informatik.uni-freiburg.de/~cziegler/BX/BX-CSV-Dump.zip\n",
      "Resolving www2.informatik.uni-freiburg.de (www2.informatik.uni-freiburg.de)... 132.230.105.133\n",
      "Connecting to www2.informatik.uni-freiburg.de (www2.informatik.uni-freiburg.de)|132.230.105.133|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 26085508 (25M) [application/zip]\n",
      "Saving to: ‘BX-CSV-Dump.zip’\n",
      "\n",
      "BX-CSV-Dump.zip     100%[===================>]  24.88M  3.17MB/s    in 8.0s    \n",
      "\n",
      "2018-04-09 15:29:37 (3.11 MB/s) - ‘BX-CSV-Dump.zip’ saved [26085508/26085508]\n",
      "\n",
      "Archive:  BX-CSV-Dump.zip\n",
      "  inflating: BX-Book-Ratings.csv     \n",
      "  inflating: BX-Books.csv            \n",
      "  inflating: BX-Users.csv            \n"
     ]
    }
   ],
   "source": [
    "#Downloading and unzipping the data\n",
    "!wget http://www2.informatik.uni-freiburg.de/~cziegler/BX/BX-CSV-Dump.zip\n",
    "!unzip BX-CSV-Dump.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cell above loads three .csv files into the working directory. We are interested in the files \"BX-Books-Ratings.csv\" and \"BX-Books.csv\". The first three columns of \"BX-Book-Ratings\" are the user id, an isbn which identifies the book, and a rating. A '0' in the rating column is used to denote that an implicit interaction occured between the user an the book. It is this data that we are interested in, and we extract such rows using the following grep command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!grep '\"0\"' BX-Book-Ratings.csv > implicit.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Load in the data\n",
    "#The data is csv, with ';' as a delimiter, hence the split command. \n",
    "#The data has quote marks around all info, so I remove these with a replace mapping. \n",
    "#The first bit of data is user id, the second is the book isbn number, \n",
    "# and the third is the observation. \n",
    "ratings = sc.textFile('implicit.csv').map(lambda x: x.replace('\"',\"\")) \\\n",
    "            .map(lambda x:x.split(\";\"))\\\n",
    "            .map(lambda x:(int(x[0]), str(x[1]), int(x[2])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the first 10 entries in the ratings file: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(276725, '034545104X', 0),\n",
       " (276727, '0446520802', 0),\n",
       " (276733, '2080674722', 0),\n",
       " (276746, '0425115801', 0),\n",
       " (276746, '0449006522', 0),\n",
       " (276746, '0553561618', 0),\n",
       " (276746, '055356451X', 0),\n",
       " (276746, '0786013990', 0),\n",
       " (276746, '0786014512', 0),\n",
       " (276747, '0451192001', 0)]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.take(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The implicit ALS function we are going to use requires that product ids are integers. At the moment we have unique ISBNs, which contain a mixture of numbers and letters, so we must convert to integers. This can be done using the zipWithIndex() function which takes an RDD and joins unique ids to each entry. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('0425103528', ((41455, 0), 103633)),\n",
       " ('0425103528', ((95991, 0), 103633)),\n",
       " ('0425103528', ((102967, 0), 103633)),\n",
       " ('0425103528', ((186570, 0), 103633)),\n",
       " ('0756401836', ((93047, 0), 30136)),\n",
       " ('0756401836', ((110483, 0), 30136)),\n",
       " ('0756401836', ((170415, 0), 30136)),\n",
       " ('0756401836', ((176875, 0), 30136)),\n",
       " ('321630681X', ((245839, 0), 61121)),\n",
       " ('0439110246', ((114414, 0), 2))]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Extract unique isbns.\n",
    "isbns=ratings.map(lambda x:x[1]).distinct()\n",
    "#Associates an integer with each unique isbn.\n",
    "isbns_with_indices=isbns.zipWithIndex() \n",
    "#sets isbn as the key\n",
    "reordered_ratings = ratings.map(lambda x:(x[1], (x[0], x[2]))) \n",
    "joined = reordered_ratings.join(isbns_with_indices) #joins with indexes \n",
    "joined.take(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(41455, 103633, 0),\n",
       " (95991, 103633, 0),\n",
       " (102967, 103633, 0),\n",
       " (186570, 103633, 0),\n",
       " (93047, 30136, 0),\n",
       " (110483, 30136, 0),\n",
       " (170415, 30136, 0),\n",
       " (176875, 30136, 0),\n",
       " (245839, 61121, 0),\n",
       " (114414, 2, 0)]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#The data above is of the form :\n",
    "    #(isbn, ((userid, rating), isbn-id-integer))\n",
    "#We use the map function to get to the form :\n",
    "    #(user id, isbn-id-integer, rating)\n",
    "#This is the form expected by the ALS function\n",
    "ratings_int_nice = joined.map(lambda x: (x[1][0][0], x[1][1], x[1][0][1]))\n",
    "ratings_int_nice.take(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Need 1s not 0s. since the matrix is singular if 0s. \n",
    "#i.e. we use '1' to indicate response, not 0.\n",
    "ratings_ones = ratings_int_nice.map(lambda x:(x[0], x[1], 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now import the ALS function from the mllib module, and build the model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pyspark.mllib.recommendation import ALS\n",
    "model=ALS.trainImplicit(ratings_ones, rank=5, iterations=3, alpha=0.99)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at user 8. We wish to make predictions on what books the user will like, based on their interactions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Filter out all the  id of all books rated by user id = 8. \n",
    "users_books = ratings_ones.filter(lambda x: x[0] is 8).map(lambda x:x[1])\n",
    "books_for_them = users_books.collect() #Collect this as a list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this next cell, we make a rdd of (user = 8, book ids) where there is an entry for every book they have not before interacted with. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(8, 0),\n",
       " (8, 1),\n",
       " (8, 2),\n",
       " (8, 3),\n",
       " (8, 4),\n",
       " (8, 5),\n",
       " (8, 6),\n",
       " (8, 7),\n",
       " (8, 8),\n",
       " (8, 9)]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unseen = isbns_with_indices.map(lambda x:x[1]) \\\n",
    "                            .filter(lambda x: x not in books_for_them) \\\n",
    "                            .map(lambda x: (8, int(x)))\n",
    "unseen.take(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Using the predict all function to give predictions for any unseens. \n",
    "predictions = model.predictAll(unseen)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now look at predictions for a range of user, product pairs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Rating(user=8, product=185544, rating=3.5499178936374144e-06),\n",
       " Rating(user=8, product=152288, rating=1.1140932904340358e-05),\n",
       " Rating(user=8, product=143464, rating=1.2945490545668097e-05),\n",
       " Rating(user=8, product=23776, rating=3.676099823410155e-05),\n",
       " Rating(user=8, product=155312, rating=1.923518143573988e-05),\n",
       " Rating(user=8, product=82512, rating=-2.2283889456646724e-06),\n",
       " Rating(user=8, product=170792, rating=-9.112801659288064e-09),\n",
       " Rating(user=8, product=103184, rating=4.111202468665078e-09),\n",
       " Rating(user=8, product=40888, rating=2.4160142131097027e-06),\n",
       " Rating(user=8, product=200376, rating=1.281169533897479e-05)]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions.take(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use .takeOrdered to view the 20 highest rated items for that user. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Rating(user=8, product=218987, rating=0.006400773040138297),\n",
       " Rating(user=8, product=37124, rating=0.004201500598023086),\n",
       " Rating(user=8, product=226501, rating=0.003013403659823166),\n",
       " Rating(user=8, product=222388, rating=0.0019098079964174936),\n",
       " Rating(user=8, product=224398, rating=0.001893200744164136),\n",
       " Rating(user=8, product=135249, rating=0.0018104361395196718),\n",
       " Rating(user=8, product=158268, rating=0.0017122310913895078),\n",
       " Rating(user=8, product=115200, rating=0.0015320753848580505),\n",
       " Rating(user=8, product=126542, rating=0.0015233736116273327),\n",
       " Rating(user=8, product=153367, rating=0.001396647908855291),\n",
       " Rating(user=8, product=104641, rating=0.0013685284571707751),\n",
       " Rating(user=8, product=56184, rating=0.0012524267983160201),\n",
       " Rating(user=8, product=61040, rating=0.0012331378504750984),\n",
       " Rating(user=8, product=215022, rating=0.0011415533435755764),\n",
       " Rating(user=8, product=170112, rating=0.0011322398700266108),\n",
       " Rating(user=8, product=194437, rating=0.0011117130375315337),\n",
       " Rating(user=8, product=156460, rating=0.0010892750649857627),\n",
       " Rating(user=8, product=134108, rating=0.0010762374172979586),\n",
       " Rating(user=8, product=223924, rating=0.001054600042183259),\n",
       " Rating(user=8, product=64700, rating=0.0010088824359691375)]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions.takeOrdered(20, lambda x: -x[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The .recommendProducts function allows us to view predicted ratings for specific user, item pairs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Rating(user=8, product=218987, rating=0.006400773040138297),\n",
       " Rating(user=8, product=37124, rating=0.004201500598023086),\n",
       " Rating(user=8, product=226501, rating=0.003013403659823166),\n",
       " Rating(user=8, product=222388, rating=0.0019098079964174936),\n",
       " Rating(user=8, product=224398, rating=0.001893200744164136),\n",
       " Rating(user=8, product=135249, rating=0.0018104361395196718),\n",
       " Rating(user=8, product=158268, rating=0.0017122310913895078),\n",
       " Rating(user=8, product=115200, rating=0.0015320753848580505),\n",
       " Rating(user=8, product=126542, rating=0.0015233736116273327),\n",
       " Rating(user=8, product=153367, rating=0.001396647908855291)]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.recommendProducts(8,10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion \n",
    "In this notebook we saw how to build a basic implicit ALS model in Spark. However, the data used was fairly plain, with \"0\"s being used for all implicit interactions. Furtherwork should consider a dataset more suited to implicit ALS. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}