Regression Types.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "264a125e",
   "metadata": {},
   "source": [
    "# Feature Scaling\n",
    "Feature scaling provides many benifits to our machine learning process"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea613bce",
   "metadata": {},
   "source": [
    "1. Some machine learning models that rely on distance metrics require scaling to perform well.\n",
    "2. Benifits of feature scaling\n",
    "    1. Feature scaling improves the convergence of steepest descent algorithm which do not possess the property of scale invarience.\n",
    "    2. If features are on differente scales, certain weights may update faster than others since the feature values x(j) play a role in weight updates.\n",
    "    3. Critical benifits of feature scaling related to gradient descent.\n",
    "    4. There are some ML algos where scaling won't have an effect."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "493933a3",
   "metadata": {},
   "source": [
    "Two main ways to scale features:\n",
    "1. Standaradization\n",
    "    1. Rescales data to have a mean(mue) of 0 and standard deviation(sigma) of 1.\n",
    "2. Normalization\n",
    "    1. Rescales all data values to be between 0-1."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1cc96c2",
   "metadata": {},
   "source": [
    "# Standardization\n",
    "1. Rescale data to have a mean(mue) of 0 and standard deviation(sigma) of 1(unit variance).\n",
    "        X(changed) =(X - mue) / sigma.\n",
    "2. Nmaesake can be confusung since this is also reffered to as \"Z-score normalization\"."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a543b0f",
   "metadata": {},
   "source": [
    "# Normalization\n",
    "1. Scale all data values to be between 0 and 1.\n",
    "        X(changed) = (X-X(min))/(X(max)-X(min)).\n",
    "2. It is simple and esay to understand.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13738601",
   "metadata": {},
   "source": [
    "There are amny more methods of scaling features and Scikit Learn provides easy to use classes that \"fit\" and \"transform\" feature data for scaling."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d30c79c",
   "metadata": {},
   "source": [
    "1. A .fir() method call simply calculates the necessary statistics(Xmin, Xmax, mean, standard deviation).\n",
    "2. A .transform() call actually scalles data and returns the new scaled version of data. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe0a5c7c",
   "metadata": {},
   "source": [
    "1. we only fit to training data.\n",
    "2. Calculating statistical information should only come from traning data.\n",
    "3. Don't want to assume prior knowledge of the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d49be9f5",
   "metadata": {},
   "source": [
    "Using the full data set would cause data leakage:\n",
    "1. Claculating statistics from full data leads to some information of the test set leaking into the training process upon transform() conversion."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2abb48c",
   "metadata": {},
   "source": [
    "# Feature Scaling process:\n",
    "1. Perform train test split\n",
    "2. Fit to training feature data.\n",
    "3. Transform training feature data.\n",
    "4. Transform test feature data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e208b7f",
   "metadata": {},
   "source": [
    "# Do we need to scale the label?\n",
    "1. In general it is not necessary nor advice.\n",
    "2. Normalising the output distribution is altering the defination of the target.\n",
    "3. Predicting a distribution that doesn't mirror your real-world-target.\n",
    "4. Can negatively impact stochastic gradient descent."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61a465ce",
   "metadata": {},
   "source": [
    "# Cross validation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ca61a3b",
   "metadata": {},
   "source": [
    "It is a more advanced set of methods for splitting data into training and testing sets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49f54fd7",
   "metadata": {},
   "source": [
    "# Regularization for Linear Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "90c9655d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5f585991",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"Advertising.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ef41a64f",
   "metadata": {},
   "outputs": [],
   "source": [
    "X= df.drop('sales', axis =1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9a20fb99",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df['sales']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9e28d899",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3c7cc3cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "poly_convertor  = PolynomialFeatures(degree=3, include_bias=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8912b8e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "poly_features = poly_convertor.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "70eb9006",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(200, 19)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "poly_features.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9bc8c0c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "eee33c5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(poly_features, y, test_size=0.3, random_state=101)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "25f4d0d9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(140, 19)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b49f2a27",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ba423444",
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5167e931",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "StandardScaler()"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler.fit(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "97c392c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = scaler.transform(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "aae72f67",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "ae730cf7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.49300171, -0.33994238,  1.61586707,  0.28407363, -0.02568776,\n",
       "        1.49677566, -0.59023161,  0.41659155,  1.6137853 ,  0.08057172,\n",
       "       -0.05392229,  1.01524393, -0.36986163,  0.52457967,  1.48737034,\n",
       "       -0.66096022, -0.16360242,  0.54694754,  1.37075536])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "cb9288c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# poly_features[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ccba4d2",
   "metadata": {},
   "source": [
    "# Ridge Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f70c894",
   "metadata": {},
   "source": [
    "1. It is a regularization technique that works by helping reduce the potential for overfitting to the training data.\n",
    "2. It does this by adding in a penatly term to the error that is based on the squred value of the coefficents.\n",
    "3. Ridge regression is a regularization method for Linear Regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "416ab36f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Ridge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "8dcb87b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "ridg_model = Ridge(alpha=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "b4d48d3f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ridge(alpha=10)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridg_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ef44e861",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_pred = ridg_model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "12c91c0d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([13.7469803 , 19.17764   , 11.8623393 , 16.25574551,  8.92891155,\n",
       "        8.04595366, 20.13320851, 16.94847676, 10.29219174, 19.78365691,\n",
       "       10.31758361, 13.44077317, 12.24228062, 23.23864236, 19.12656362,\n",
       "        9.748809  , 12.84361521,  9.14248129,  8.82504966, 21.46989971,\n",
       "        8.25222201, 18.8110776 , 28.39227333, 25.06911066,  9.11912241,\n",
       "       11.8071711 , 20.85199462,  9.31962495, 12.25122303,  9.07811198,\n",
       "        9.25275556, 19.87154568, 10.55955497,  7.86818084, 17.67220132,\n",
       "        9.31442423, 10.1539168 ,  9.47802944,  8.63337302, 10.76209179,\n",
       "       11.63848891, 10.38883411, 10.90270005,  8.40019284, 11.35336224,\n",
       "       10.3514086 ,  8.00082147, 15.27139854, 12.94761469, 22.49762978,\n",
       "       10.80165407, 13.01337743, 14.82204895, 11.37354043, 11.69628953,\n",
       "        7.20576885, 25.53512232, 10.41926545, 17.38798022, 15.10158819])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "99e365c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_absolute_error, mean_squared_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "6f8cd088",
   "metadata": {},
   "outputs": [],
   "source": [
    "MAE = mean_absolute_error(y_test, test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "63392c2b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5774404204714183"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MAE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "da5cd3cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "RMSE = np.sqrt(mean_squared_error(y_test, test_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "3c26b9df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8946386461319685"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "RMSE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "c4b2affc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import RidgeCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "cca4eb6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "ridge_cv_model = RidgeCV(alphas=(0.1, 1.0, 10.0), scoring= 'neg_mean_absolute_error')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "c714a086",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RidgeCV(alphas=array([ 0.1,  1. , 10. ]), scoring='neg_mean_absolute_error')"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge_cv_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "e8290b49",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#  which alpha perameter perform best\n",
    "ridge_cv_model.alpha_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "80f776ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# from sklearn.metrics import SCORERS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "ba35117c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# SCORERS.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "93e76c7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_pred = ridge_cv_model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "d9a3fbcb",
   "metadata": {},
   "outputs": [],
   "source": [
    "MAE = mean_absolute_error(y_test, test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "f476077e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.42737748843352086"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MAE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "779f28ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "RMSE = np.sqrt(mean_squared_error(y_test, test_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "d9209ad4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6180719926924644"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# RMSE = Root Mean Squared Error\n",
    "RMSE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "e8e08385",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 5.40769392,  0.5885865 ,  0.40390395, -6.18263924,  4.59607939,\n",
       "       -1.18789654, -1.15200458,  0.57837796, -0.1261586 ,  2.5569777 ,\n",
       "       -1.38900471,  0.86059434,  0.72219553, -0.26129256,  0.17870787,\n",
       "        0.44353612, -0.21362436, -0.04622473, -0.06441449])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ridge_cv_model.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0333d662",
   "metadata": {},
   "source": [
    "# Lasso Regression (L1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1f06fbb",
   "metadata": {},
   "source": [
    "L1 regularization adds a penalty equal to the absolute value of the magnitude of coefficients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "6b991066",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LassoCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "7fbd96c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "lasso_cv_model = LassoCV(eps = 0.001, n_alphas = 100, cv = 5, max_iter=1000000)\n",
    "# lasso_cv_model = LassoCV(eps = 0.1, n_alphas = 100, cv = 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "2a7e01d7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LassoCV(cv=5, max_iter=1000000)"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso_cv_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "5d731514",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.004943070909225827"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso_cv_model.alpha_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "a2f25a4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_pred = lasso_cv_model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "012de550",
   "metadata": {},
   "outputs": [],
   "source": [
    "MAE = mean_absolute_error(y_test, test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "7f6e2fb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "RMSE = np.sqrt(mean_squared_error(y_test, test_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "18b9f350",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.43350346185900673"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MAE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "05af1da9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6063140748984039"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "RMSE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "852877cd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 4.86023329,  0.12544598,  0.20746872, -4.99250395,  4.38026519,\n",
       "       -0.22977201, -0.        ,  0.07267717, -0.        ,  1.77780246,\n",
       "       -0.69614918, -0.        ,  0.12044132, -0.        , -0.        ,\n",
       "       -0.        ,  0.        ,  0.        , -0.        ])"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lasso_cv_model.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86430e8e",
   "metadata": {},
   "source": [
    "# Elastic Net"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e939897d",
   "metadata": {},
   "source": [
    "We know Lasso is able to shrink coefficients to zero, but we have not  taken a deeper dive  into how or why that is. This ability becomes more clear when learning about elastic net which combines Lasso and Ridge together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "b37f6960",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import ElasticNetCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "2108aa90",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = ElasticNetCV(l1_ratio=[.1, .5, .7, .9, .95, .99, 1], eps = 0.001, n_alphas=100, max_iter=10000000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "25c9fe96",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ElasticNetCV(l1_ratio=[0.1, 0.5, 0.7, 0.9, 0.95, 0.99, 1], max_iter=10000000)"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "761862d7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.l1_ratio_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "7ca735bf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.004943070909225827"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.alpha_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "1100c2a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_pred = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "43124a95",
   "metadata": {},
   "outputs": [],
   "source": [
    "MAE = mean_absolute_error(y_test, test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "c1b193ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "RMSE = np.sqrt(mean_squared_error(y_test, test_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "7bc6942f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.43350346185900673"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MAE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "0ea82ce7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6063140748984039"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "RMSE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b887696c",
   "metadata": {},
   "source": [
    "# Linear Regression Project "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "424c3460",
   "metadata": {},
   "source": [
    "Most data sets require cleaning, analysis, and feature engineering before being used for machine learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "72c740c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "82277962",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('Ames_Housing_Data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "69ec8123",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PID</th>\n",
       "      <th>MS SubClass</th>\n",
       "      <th>MS Zoning</th>\n",
       "      <th>Lot Frontage</th>\n",
       "      <th>Lot Area</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>Lot Shape</th>\n",
       "      <th>Land Contour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>...</th>\n",
       "      <th>Pool Area</th>\n",
       "      <th>Pool QC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>Misc Feature</th>\n",
       "      <th>Misc Val</th>\n",
       "      <th>Mo Sold</th>\n",
       "      <th>Yr Sold</th>\n",
       "      <th>Sale Type</th>\n",
       "      <th>Sale Condition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>526301100</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>141.0</td>\n",
       "      <td>31770</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>215000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>526350040</td>\n",
       "      <td>20</td>\n",
       "      <td>RH</td>\n",
       "      <td>80.0</td>\n",
       "      <td>11622</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MnPrv</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>105000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>526351010</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>81.0</td>\n",
       "      <td>14267</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Gar2</td>\n",
       "      <td>12500</td>\n",
       "      <td>6</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>172000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>526353030</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>93.0</td>\n",
       "      <td>11160</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>244000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>527105010</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>74.0</td>\n",
       "      <td>13830</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MnPrv</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>189900</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 81 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         PID  MS SubClass MS Zoning  Lot Frontage  Lot Area Street Alley  \\\n",
       "0  526301100           20        RL         141.0     31770   Pave   NaN   \n",
       "1  526350040           20        RH          80.0     11622   Pave   NaN   \n",
       "2  526351010           20        RL          81.0     14267   Pave   NaN   \n",
       "3  526353030           20        RL          93.0     11160   Pave   NaN   \n",
       "4  527105010           60        RL          74.0     13830   Pave   NaN   \n",
       "\n",
       "  Lot Shape Land Contour Utilities  ... Pool Area Pool QC  Fence Misc Feature  \\\n",
       "0       IR1          Lvl    AllPub  ...         0     NaN    NaN          NaN   \n",
       "1       Reg          Lvl    AllPub  ...         0     NaN  MnPrv          NaN   \n",
       "2       IR1          Lvl    AllPub  ...         0     NaN    NaN         Gar2   \n",
       "3       Reg          Lvl    AllPub  ...         0     NaN    NaN          NaN   \n",
       "4       IR1          Lvl    AllPub  ...         0     NaN  MnPrv          NaN   \n",
       "\n",
       "  Misc Val Mo Sold Yr Sold  Sale Type  Sale Condition  SalePrice  \n",
       "0        0       5    2010        WD           Normal     215000  \n",
       "1        0       6    2010        WD           Normal     105000  \n",
       "2    12500       6    2010        WD           Normal     172000  \n",
       "3        0       4    2010        WD           Normal     244000  \n",
       "4        0       3    2010        WD           Normal     189900  \n",
       "\n",
       "[5 rows x 81 columns]"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "3a91b770",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 2930 entries, 0 to 2929\n",
      "Data columns (total 81 columns):\n",
      " #   Column           Non-Null Count  Dtype  \n",
      "---  ------           --------------  -----  \n",
      " 0   PID              2930 non-null   int64  \n",
      " 1   MS SubClass      2930 non-null   int64  \n",
      " 2   MS Zoning        2930 non-null   object \n",
      " 3   Lot Frontage     2440 non-null   float64\n",
      " 4   Lot Area         2930 non-null   int64  \n",
      " 5   Street           2930 non-null   object \n",
      " 6   Alley            198 non-null    object \n",
      " 7   Lot Shape        2930 non-null   object \n",
      " 8   Land Contour     2930 non-null   object \n",
      " 9   Utilities        2930 non-null   object \n",
      " 10  Lot Config       2930 non-null   object \n",
      " 11  Land Slope       2930 non-null   object \n",
      " 12  Neighborhood     2930 non-null   object \n",
      " 13  Condition 1      2930 non-null   object \n",
      " 14  Condition 2      2930 non-null   object \n",
      " 15  Bldg Type        2930 non-null   object \n",
      " 16  House Style      2930 non-null   object \n",
      " 17  Overall Qual     2930 non-null   int64  \n",
      " 18  Overall Cond     2930 non-null   int64  \n",
      " 19  Year Built       2930 non-null   int64  \n",
      " 20  Year Remod/Add   2930 non-null   int64  \n",
      " 21  Roof Style       2930 non-null   object \n",
      " 22  Roof Matl        2930 non-null   object \n",
      " 23  Exterior 1st     2930 non-null   object \n",
      " 24  Exterior 2nd     2930 non-null   object \n",
      " 25  Mas Vnr Type     2907 non-null   object \n",
      " 26  Mas Vnr Area     2907 non-null   float64\n",
      " 27  Exter Qual       2930 non-null   object \n",
      " 28  Exter Cond       2930 non-null   object \n",
      " 29  Foundation       2930 non-null   object \n",
      " 30  Bsmt Qual        2850 non-null   object \n",
      " 31  Bsmt Cond        2850 non-null   object \n",
      " 32  Bsmt Exposure    2847 non-null   object \n",
      " 33  BsmtFin Type 1   2850 non-null   object \n",
      " 34  BsmtFin SF 1     2929 non-null   float64\n",
      " 35  BsmtFin Type 2   2849 non-null   object \n",
      " 36  BsmtFin SF 2     2929 non-null   float64\n",
      " 37  Bsmt Unf SF      2929 non-null   float64\n",
      " 38  Total Bsmt SF    2929 non-null   float64\n",
      " 39  Heating          2930 non-null   object \n",
      " 40  Heating QC       2930 non-null   object \n",
      " 41  Central Air      2930 non-null   object \n",
      " 42  Electrical       2929 non-null   object \n",
      " 43  1st Flr SF       2930 non-null   int64  \n",
      " 44  2nd Flr SF       2930 non-null   int64  \n",
      " 45  Low Qual Fin SF  2930 non-null   int64  \n",
      " 46  Gr Liv Area      2930 non-null   int64  \n",
      " 47  Bsmt Full Bath   2928 non-null   float64\n",
      " 48  Bsmt Half Bath   2928 non-null   float64\n",
      " 49  Full Bath        2930 non-null   int64  \n",
      " 50  Half Bath        2930 non-null   int64  \n",
      " 51  Bedroom AbvGr    2930 non-null   int64  \n",
      " 52  Kitchen AbvGr    2930 non-null   int64  \n",
      " 53  Kitchen Qual     2930 non-null   object \n",
      " 54  TotRms AbvGrd    2930 non-null   int64  \n",
      " 55  Functional       2930 non-null   object \n",
      " 56  Fireplaces       2930 non-null   int64  \n",
      " 57  Fireplace Qu     1508 non-null   object \n",
      " 58  Garage Type      2773 non-null   object \n",
      " 59  Garage Yr Blt    2771 non-null   float64\n",
      " 60  Garage Finish    2771 non-null   object \n",
      " 61  Garage Cars      2929 non-null   float64\n",
      " 62  Garage Area      2929 non-null   float64\n",
      " 63  Garage Qual      2771 non-null   object \n",
      " 64  Garage Cond      2771 non-null   object \n",
      " 65  Paved Drive      2930 non-null   object \n",
      " 66  Wood Deck SF     2930 non-null   int64  \n",
      " 67  Open Porch SF    2930 non-null   int64  \n",
      " 68  Enclosed Porch   2930 non-null   int64  \n",
      " 69  3Ssn Porch       2930 non-null   int64  \n",
      " 70  Screen Porch     2930 non-null   int64  \n",
      " 71  Pool Area        2930 non-null   int64  \n",
      " 72  Pool QC          13 non-null     object \n",
      " 73  Fence            572 non-null    object \n",
      " 74  Misc Feature     106 non-null    object \n",
      " 75  Misc Val         2930 non-null   int64  \n",
      " 76  Mo Sold          2930 non-null   int64  \n",
      " 77  Yr Sold          2930 non-null   int64  \n",
      " 78  Sale Type        2930 non-null   object \n",
      " 79  Sale Condition   2930 non-null   object \n",
      " 80  SalePrice        2930 non-null   int64  \n",
      "dtypes: float64(11), int64(27), object(43)\n",
      "memory usage: 1.8+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "301bc963",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}