day2/SGClassificationToFill.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "699a1ab5",
   "metadata": {},
   "source": [
    "This notebook guides you through an exercise in star/galaxy/quasar classification, using k Nearest Neighbors and Decision Trees algorithm.\n",
    "\n",
    "Written with ❤️ by Viviana Acquaviva for the LSST Data Science Fellowship lecture series.\n",
    "\n",
    "License: [BSD-3-clause](https://opensource.org/license/bsd-3-clause/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d7c35a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn import metrics \n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import cross_val_predict, cross_validate\n",
    "from sklearn.model_selection import KFold, StratifiedKFold"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d52431e4",
   "metadata": {},
   "source": [
    "We can read the data here. They were taken by the Sloan Digital Sky Survey and come from [Kaggle](https://www.kaggle.com/datasets/divyansh22/dummy-astronomy-data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c8779a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv('star_classification.csv', sep = ',')\n",
    "\n",
    "df = data.sample(frac = 0.2, random_state = 11)\n",
    "\n",
    "df.reset_index(drop = True, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0698ed69",
   "metadata": {},
   "source": [
    "Let's learn a few things about this data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35d86865",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4024fdef",
   "metadata": {},
   "source": [
    "We can get some basic info about the data (numerical columns only!) like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89a889e2",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd0394ca",
   "metadata": {},
   "source": [
    "What should be our features and targets?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2966bc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c844f607",
   "metadata": {},
   "source": [
    "We can select the following columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f34b2d04",
   "metadata": {},
   "outputs": [],
   "source": [
    "seldf = df[['u','g','r','i','z']]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1097580",
   "metadata": {},
   "source": [
    "### Feature exploration.\n",
    "\n",
    "What should we do?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b6b31ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2571d100",
   "metadata": {},
   "outputs": [],
   "source": [
    "for col in seldf.columns:\n",
    "    plt.hist(seldf[col], alpha = 0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1395173",
   "metadata": {},
   "source": [
    "When a histogram looks like this, what does it mean?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbb27ce2",
   "metadata": {},
   "outputs": [],
   "source": [
    "for col in seldf.columns:\n",
    "    plt.hist(seldf[col], alpha = 0.5, bins=100, range = (0,30))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "629f0fe9",
   "metadata": {},
   "outputs": [],
   "source": [
    "for col in seldf.columns:\n",
    "    print(len(np.where(seldf[col] < 0)[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98936b68",
   "metadata": {},
   "source": [
    "One possible (less than ideal) selection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce70af18",
   "metadata": {},
   "outputs": [],
   "source": [
    "for col in seldf.columns:\n",
    "    seldf = seldf[seldf[col] >= 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "658e31e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "seldf.shape "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f69d081a",
   "metadata": {},
   "outputs": [],
   "source": [
    "for col in seldf.columns:\n",
    "    plt.hist(seldf[col], alpha = 0.5, bins=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79e6f958",
   "metadata": {},
   "source": [
    "Let's now turn to the target column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99855784",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.unique(df['class'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2467d80e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "le = LabelEncoder() #turns categorical into 1 ... N"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2214f779",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = le.fit_transform(df['class'].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c376da17",
   "metadata": {},
   "outputs": [],
   "source": [
    "le.classes_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70bd2f44",
   "metadata": {},
   "source": [
    "It is also useful to know which class was mapped to which number!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93fde9c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "le_name_mapping = dict(zip(le.classes_, le.transform(le.classes_)))\n",
    "print(le_name_mapping)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1e58294",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.unique(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c01ff17",
   "metadata": {},
   "source": [
    "We can now plot the class distribution to see if something stands out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "876027b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hist(y);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f514d070",
   "metadata": {},
   "source": [
    "Verdict: Is there anything we should keep in mind?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29db7723",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63e3a03b",
   "metadata": {},
   "outputs": [],
   "source": [
    "target = y[seldf.index]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e2db9de",
   "metadata": {},
   "source": [
    "Now that we have our feature matrix and target vector, we could try and build a model, or try to get some more intel on the relationship between features and target. One way is to build a correlation matrix (easiest with seaborn); another one that I like when there are very few features like in this case, is to make scatter plots of the features and color the points by the label, to see if any obvious gradient shows up. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d734eac1",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(seldf.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7f0c99c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#It will take a little time.\n",
    "\n",
    "fig, axes = plt.subplots(ncols = len(seldf.columns), nrows = len(seldf.columns), figsize=(50,50))\n",
    "\n",
    "for i, col in enumerate(seldf.columns):\n",
    "    for j, col2 in enumerate(seldf.columns):\n",
    "        axes[i,j].scatter(seldf[col], seldf[col2], c = target)\n",
    "        axes[i,j].set_xlabel(str(seldf[col].name), fontsize = 35)\n",
    "        axes[i,j].set_ylabel(str(seldf[col2].name), fontsize = 35)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d809085d",
   "metadata": {},
   "source": [
    "Q: Is there anything special that we can learn from these plots? Do we expect our features to have enough discriminating power? Why or why not?\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28d3cc6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d79ea23",
   "metadata": {},
   "source": [
    "### Building models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e14f579b",
   "metadata": {},
   "source": [
    "Alright, time to build our first model - let's start with kNN."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec0fcf45",
   "metadata": {},
   "source": [
    "This line helps visualize the parameters of any algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b9231a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c97bfe3",
   "metadata": {},
   "source": [
    "We can set up a cross validation strategy like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8383c370",
   "metadata": {},
   "outputs": [],
   "source": [
    "cv = KFold(n_splits=5, random_state=10, shuffle= True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9eb8d9f8",
   "metadata": {},
   "source": [
    "This line does the cross validation for us by randomizing the data, creating five folds, building 5 models with the 5 possible training sets, applying each model to the objects in the test fold, and saving the 5 train and test scores in a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed312748",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = cross_validate(model, seldf, target, cv = cv, return_train_score = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3304513",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Time: %.3fs, Mean train score: %.3f, Mean test score: %.3f, std: %.3f\"%(scores['score_time'].mean(), scores['train_score'].mean(), scores['test_score'].mean(), scores['test_score'].std()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c8913ce",
   "metadata": {},
   "source": [
    "Let us now set up a decision tree classifier with default parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45d5db55",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = DecisionTreeClassifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "488d0792",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8149168",
   "metadata": {},
   "source": [
    "Decision trees are more complex than kNN and have a few more hyperparameters to play with. They include: \n",
    "- the minimum number of instances in a leaf node; \n",
    "- the minimum number of instances required in a split node;\n",
    "- the maximum depth of tree;\n",
    "- the minimum information gain required to decide whether a split is \"worth it\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "669f358b",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = cross_validate(model, seldf, target, cv = cv, return_train_score=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd98b150",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Time: %.3fs, Mean train score: %.3f, Mean test score: %.3f, std: %.3f\"%(scores['score_time'].mean(), scores['train_score'].mean(), scores['test_score'].mean(), scores['test_score'].std()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f33f1f5a",
   "metadata": {},
   "source": [
    "Before sending you on your way with optimizing these models, let's also talk about the great function cross_val_predict. It is used to generate test predictions in the cross validation framework. Just like cross_validate, this line will randomize the data, create five folds (if the random seed is fixed, they will be consistent across our notebooks!), build 5 models with the 5 possible training sets, apply each model to the objects in the test fold, and compile together the predicted values from the 5 test folds. The output will be this vector of predicted values, which has the same size as the target vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e8c1fca",
   "metadata": {},
   "outputs": [],
   "source": [
    "ypred = cross_val_predict(model, seldf, target, cv = cv)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9be12a2",
   "metadata": {},
   "source": [
    "Having these predictions is very useful because you can calculate more complex scores, or a confusion matrix (or a scatter plot of true vs predicted values in the case of regression)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "812284db",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(metrics.confusion_matrix(target, ypred))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb115831",
   "metadata": {},
   "source": [
    "Finally, I'm going to drop here this handy function to calculate and plot learning curves."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "363ec40e",
   "metadata": {},
   "source": [
    "Learning curves are helpful in order to visualize the training scores vs the test scores, and how they vary as a function of data set size. They allow us to determine whether we have enough learning data, AND whether we have a high bias or high variance problem.\n",
    "\n",
    "The source code below is a slight modification of [this code](https://scikit-learn.org/stable/auto_examples/model_selection/plot_learning_curve.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd1f5d18",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import learning_curve\n",
    "\n",
    "def plot_learning_curve(estimator, title, X, y, ylim=None, cv=5,\n",
    "                        n_jobs=-1, train_sizes=np.linspace(.1, 1.0, 5), scoring = 'accuracy'):\n",
    "    \"\"\"\n",
    "    Generate a simple plot of the test and training learning curve.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    estimator : object type that implements the \"fit\" and \"predict\" methods\n",
    "        An object of that type which is cloned for each validation.\n",
    "\n",
    "    title : string\n",
    "        Title for the chart.\n",
    "\n",
    "    X : array-like, shape (n_samples, n_features)\n",
    "        Training vector, where n_samples is the number of samples and\n",
    "        n_features is the number of features.\n",
    "\n",
    "    y : array-like, shape (n_samples) or (n_samples, n_features), optional\n",
    "        Target relative to X for classification or regression;\n",
    "        None for unsupervised learning.\n",
    "\n",
    "    ylim : tuple, shape (ymin, ymax), optional\n",
    "        Defines minimum and maximum yvalues plotted.\n",
    "\n",
    "    cv : int, cross-validation generator or an iterable, optional\n",
    "        Determines the cross-validation splitting strategy.\n",
    "        Possible inputs for cv are:\n",
    "          - None, to use the default 3-fold cross-validation,\n",
    "          - integer, to specify the number of folds.\n",
    "          - :term:`CV splitter`,\n",
    "          - An iterable yielding (train, test) splits as arrays of indices.\n",
    "\n",
    "        For integer/None inputs, if ``y`` is binary or multiclass,\n",
    "        :class:`StratifiedKFold` used. If the estimator is not a classifier\n",
    "        or if ``y`` is neither binary nor multiclass, :class:`KFold` is used.\n",
    "\n",
    "        Refer :ref:`User Guide <cross_validation>` for the various\n",
    "        cross-validators that can be used here.\n",
    "\n",
    "    n_jobs : int or None, optional (default=None)\n",
    "        Number of jobs to run in parallel.\n",
    "        ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context.\n",
    "        ``-1`` means using all processors. See :term:`Glossary <n_jobs>`\n",
    "        for more details.\n",
    "\n",
    "    train_sizes : array-like, shape (n_ticks,), dtype float or int\n",
    "        Relative or absolute numbers of training examples that will be used to\n",
    "        generate the learning curve. If the dtype is float, it is regarded as a\n",
    "        fraction of the maximum size of the training set (that is determined\n",
    "        by the selected validation method), i.e. it has to be within (0, 1].\n",
    "        Otherwise it is interpreted as absolute sizes of the training sets.\n",
    "        Note that for classification the number of samples usually have to\n",
    "        be big enough to contain at least one sample from each class.\n",
    "        (default: np.linspace(0.1, 1.0, 5))\n",
    "    \"\"\"\n",
    "    plt.figure(figsize=(10,6))\n",
    "    plt.title(title)\n",
    "    if ylim is not None:\n",
    "        plt.ylim(*ylim)\n",
    "    plt.xlabel(\"Training examples\")\n",
    "    plt.ylabel(str(scoring))\n",
    "    \n",
    "    train_sizes, train_scores, test_scores = learning_curve(\n",
    "        estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes, scoring = scoring)\n",
    "    train_scores_mean = np.mean(train_scores, axis=1)\n",
    "    train_scores_std = np.std(train_scores, axis=1)\n",
    "    test_scores_mean = np.mean(test_scores, axis=1)\n",
    "    test_scores_std = np.std(test_scores, axis=1)\n",
    "    plt.grid()\n",
    "\n",
    "    plt.fill_between(train_sizes, train_scores_mean - train_scores_std,\n",
    "                     train_scores_mean + train_scores_std, alpha=0.1,\n",
    "                     color=\"r\")\n",
    "    plt.fill_between(train_sizes, test_scores_mean - test_scores_std,\n",
    "                     test_scores_mean + test_scores_std, alpha=0.1, color=\"g\")\n",
    "    plt.plot(train_sizes, train_scores_mean, 'o-', color=\"r\",\n",
    "             label=\"Training score\")\n",
    "    plt.plot(train_sizes, test_scores_mean, 'o-', color=\"g\",\n",
    "             label=\"Test score from cross-validation\")\n",
    "\n",
    "    plt.legend(loc=\"best\")\n",
    "    return plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "903b7c3b",
   "metadata": {},
   "source": [
    "This is a demonstration of how to use the function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9184897",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_learning_curve(model, 'DT (default params)', seldf, target,  cv = cv);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cba64e8e",
   "metadata": {},
   "source": [
    "In the case, above, would we say that\n",
    "\n",
    "- We have high bias or high variance? Why?\n",
    "\n",
    "- We have enough training data or not? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9950e378",
   "metadata": {},
   "source": [
    "### Tasks\n",
    "\n",
    "Feel free to do the ones you prefer according to your expertise level and personal preference!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96219927",
   "metadata": {},
   "source": [
    "1. Consider (separately) the kNN and DT models with default parameters. Do they suffer from high variance or high bias? Why?\n",
    "\n",
    "2. Based on your answer above, do you anticipate better results from making the model(s) more complex or simpler? How could this be implemented, in terms of choosing hyperparameters and their range?\n",
    "\n",
    "3. Play around with hyperparameters for a model of your choice and see what test scores you can get! The highest score wins... a drink of their choice at Monday's dinner?\n",
    "\n",
    "4. What's something I should have done for kNN and didn't? :) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d05b270c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}