Feature Selection (Forward-Backward elemination).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "dfe83c8b",
   "metadata": {},
   "source": [
    "<b>The backward method is generally the preferred method, because the forward method produces so-called suppressor   effects.These suppressor effects occur when predictors are only significant when another predictor is held         constant.<br><b>\n",
    "    \n",
    "<b>Wrapper methods (forward, backward, and stepwise selection)<br><b>\n",
    " \n",
    "<b>It follows a greedy search approach by evaluating all the possible combinations of features against the evaluation criterion. The wrapper methods usually result in better predictive accuracy than filter methods.<b>\n",
    "    \n",
    "    https://www.analyticsvidhya.com/blog/2020/10/feature-selection-techniques-in-machine-learning/\n",
    "    https://www.youtube.com/watch?v=nh2IGC_kLWw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ecce9fbf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2000, 20)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "df=pd.read_csv('https://raw.githubusercontent.com/krishnaik06/Feature-Engineering-Live-sessions/master/mobile_dataset.csv')\n",
    "X=df.iloc[:,:-1]\n",
    "y=df['price_range']\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fee963e",
   "metadata": {},
   "source": [
    "### Forward Selection\n",
    "This is an iterative method wherein we start with the best performing variable against the target. Next, we select another variable that gives the best performance in combination with the first selected variable. This process continues until the preset criterion is achieved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8f2c6e2a",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done  20 out of  20 | elapsed:    2.1s finished\n",
      "\n",
      "[2022-10-08 15:44:54] Features: 1/20 -- score: 0.7174978576777677[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done  19 out of  19 | elapsed:    1.9s finished\n",
      "\n",
      "[2022-10-08 15:44:56] Features: 2/20 -- score: 0.7924956440698571[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done  18 out of  18 | elapsed:    1.8s finished\n",
      "\n",
      "[2022-10-08 15:44:58] Features: 3/20 -- score: 0.8854979417198307[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done  17 out of  17 | elapsed:    1.8s finished\n",
      "\n",
      "[2022-10-08 15:44:59] Features: 4/20 -- score: 0.9074989532260896[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done  16 out of  16 | elapsed:    1.7s finished\n",
      "\n",
      "[2022-10-08 15:45:01] Features: 5/20 -- score: 0.9079994537265902[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done  15 out of  15 | elapsed:    1.6s finished\n",
      "\n",
      "[2022-10-08 15:45:03] Features: 6/20 -- score: 0.9084984534759647[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done  14 out of  14 | elapsed:    1.7s finished\n",
      "\n",
      "[2022-10-08 15:45:04] Features: 7/20 -- score: 0.9084984534759647[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done  13 out of  13 | elapsed:    1.4s finished\n",
      "\n",
      "[2022-10-08 15:45:06] Features: 8/20 -- score: 0.9084984534759647[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done  12 out of  12 | elapsed:    1.4s finished\n",
      "\n",
      "[2022-10-08 15:45:07] Features: 9/20 -- score: 0.9084984534759647[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done  11 out of  11 | elapsed:    1.2s finished\n",
      "\n",
      "[2022-10-08 15:45:09] Features: 10/20 -- score: 0.9084984534759647[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    1.1s finished\n",
      "\n",
      "[2022-10-08 15:45:10] Features: 11/20 -- score: 0.9084984534759647[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   9 out of   9 | elapsed:    1.0s finished\n",
      "\n",
      "[2022-10-08 15:45:11] Features: 12/20 -- score: 0.9084984534759647[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   8 out of   8 | elapsed:    1.0s finished\n",
      "\n",
      "[2022-10-08 15:45:12] Features: 13/20 -- score: 0.9084984534759647[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.2s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   7 out of   7 | elapsed:    1.0s finished\n",
      "\n",
      "[2022-10-08 15:45:13] Features: 14/20 -- score: 0.9084984534759647[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   6 out of   6 | elapsed:    0.8s finished\n",
      "\n",
      "[2022-10-08 15:45:13] Features: 15/20 -- score: 0.9084984534759647[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.7s finished\n",
      "\n",
      "[2022-10-08 15:45:14] Features: 16/20 -- score: 0.9084984534759647[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   4 out of   4 | elapsed:    0.5s finished\n",
      "\n",
      "[2022-10-08 15:45:15] Features: 17/20 -- score: 0.9084984534759647[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:    0.3s finished\n",
      "\n",
      "[2022-10-08 15:45:15] Features: 18/20 -- score: 0.9084984534759647[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.2s finished\n",
      "\n",
      "[2022-10-08 15:45:15] Features: 19/20 -- score: 0.9079987033510272[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.1s finished\n",
      "\n",
      "[2022-10-08 15:45:15] Features: 20/20 -- score: 0.9064987026006516"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>SequentialFeatureSelector(cv=3, estimator=KNeighborsClassifier(n_neighbors=4),\n",
       "                          k_features=(1, 20), scoring=&#x27;accuracy&#x27;, verbose=2)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SequentialFeatureSelector</label><div class=\"sk-toggleable__content\"><pre>SequentialFeatureSelector(cv=3, estimator=KNeighborsClassifier(n_neighbors=4),\n",
       "                          k_features=(1, 20), scoring=&#x27;accuracy&#x27;, verbose=2)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: KNeighborsClassifier</label><div class=\"sk-toggleable__content\"><pre>KNeighborsClassifier(n_neighbors=4)</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">KNeighborsClassifier</label><div class=\"sk-toggleable__content\"><pre>KNeighborsClassifier(n_neighbors=4)</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "SequentialFeatureSelector(cv=3, estimator=KNeighborsClassifier(n_neighbors=4),\n",
       "                          k_features=(1, 20), scoring='accuracy', verbose=2)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from mlxtend.feature_selection import SequentialFeatureSelector as SFS\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "knn = KNeighborsClassifier(n_neighbors=4)\n",
    "\n",
    "fs=SFS(knn,k_features='best', \n",
    "           forward=True, \n",
    "           floating=False, \n",
    "           verbose=2, # it shows each steps\n",
    "           scoring='accuracy',\n",
    "           cv=3) #  By choosing cv=0, we don't perform any cross-validation, therefore, \n",
    "                 # the performance (here: 'accuracy') is computed entirely on the training set. \n",
    "fs.fit(X,y)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "13448730",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('battery_power', 'fc', 'int_memory', 'px_height', 'px_width', 'ram')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fs.k_feature_names_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4f8a7df7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9084984534759647"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fs.k_score_ # 0.9079994537265902 when k_features=5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b1fee6b",
   "metadata": {},
   "source": [
    "### Backward Selection\n",
    "This method works exactly opposite to the Forward Feature Selection method. Here, we start with all the features available and build a model. Next, we the variable from the model which gives the best evaluation measure value. This process is continued until the preset criterion is achieved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "53b3b81f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {color: black;background-color: white;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>SequentialFeatureSelector(cv=3, estimator=KNeighborsClassifier(n_neighbors=4),\n",
       "                          forward=False, k_features=(5, 5), n_jobs=-1,\n",
       "                          scoring=&#x27;accuracy&#x27;, verbose=False)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SequentialFeatureSelector</label><div class=\"sk-toggleable__content\"><pre>SequentialFeatureSelector(cv=3, estimator=KNeighborsClassifier(n_neighbors=4),\n",
       "                          forward=False, k_features=(5, 5), n_jobs=-1,\n",
       "                          scoring=&#x27;accuracy&#x27;, verbose=False)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: KNeighborsClassifier</label><div class=\"sk-toggleable__content\"><pre>KNeighborsClassifier(n_neighbors=4)</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-6\" type=\"checkbox\" ><label for=\"sk-estimator-id-6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">KNeighborsClassifier</label><div class=\"sk-toggleable__content\"><pre>KNeighborsClassifier(n_neighbors=4)</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "SequentialFeatureSelector(cv=3, estimator=KNeighborsClassifier(n_neighbors=4),\n",
       "                          forward=False, k_features=(5, 5), n_jobs=-1,\n",
       "                          scoring='accuracy', verbose=False)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fss=SFS(knn,k_features=5, \n",
    "           forward=False, \n",
    "           floating=False, \n",
    "           verbose=False, # it shows each steps\n",
    "           scoring='accuracy',\n",
    "           n_jobs=-1, # Controls the number of jobs that get dispatched during parallel execution if \n",
    "                      #n_jobs > 1 or n_jobs=-1. Reducing this number can be useful to avoid an explosion of memory \n",
    "                      #consumption when more jobs get dispatched than CPUs can process. This parameter can be: None,\n",
    "                     #in which case all the jobs are immediately created and spawned. Use this for lightweight and \n",
    "                     #fast-running jobs, to avoid delays due to on-demand spawning of the jobs An int, giving the \n",
    "                     #exact number of total jobs that are spawned \n",
    "           cv=3) #  By choosing cv=0, we don't perform any cross-validation, therefore, \n",
    "                 # the performance (here: 'accuracy') is computed entirely on the training set. \n",
    "fss.fit(X,y)   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2691ab42",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 4, 11, 12, 13)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fss.k_feature_idx_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab1e2489",
   "metadata": {},
   "source": [
    "### Exhaustive Feature Selection\n",
    "This is the most robust feature selection method covered so far. This is a brute-force evaluation of each feature subset. This means that it tries every possible combination of the variables and returns the best performing subset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "633bb6eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from mlxtend.feature_selection import ExhaustiveFeatureSelector as EXS\n",
    "es=EXS(knn,min_features=2,\n",
    "           max_features=8,\n",
    "           scoring='accuracy',\n",
    "           print_progress=False,\n",
    "           cv=5)\n",
    "es.fit(X,y)\n",
    "print('Best accuracy score: %.2f' % es.best_score_)\n",
    "print('Best subset (indices):', es.best_idx_)\n",
    "print('Best subset (corresponding names):', es.best_feature_names_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e24db8a",
   "metadata": {},
   "source": [
    "### Recursive Feature Elimination\n",
    "the goal of recursive feature elimination (RFE) is to select features by recursively considering smaller and smaller sets of features. First, the estimator is trained on the initial set of features and the importance of each feature is obtained either through a coef_ attribute or through a feature_importances_ attribute.\n",
    "\n",
    "Then, the least important features are pruned from the current set of features. That procedure is recursively repeated on the pruned set until the desired number of features to select is eventually reached."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "dc936c56",
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "when `importance_getter=='auto'`, the underlying estimator KNeighborsClassifier should have `coef_` or `feature_importances_` attribute. Either pass a fitted estimator to feature selector or call fit before calling transform.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "Input \u001b[0;32mIn [8]\u001b[0m, in \u001b[0;36m<cell line: 3>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01msklearn\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mfeature_selection\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m RFE\n\u001b[1;32m      2\u001b[0m rfe\u001b[38;5;241m=\u001b[39mRFE(knn,n_features_to_select\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m5\u001b[39m)\n\u001b[0;32m----> 3\u001b[0m \u001b[43mrfe\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfit_transform\u001b[49m\u001b[43m(\u001b[49m\u001b[43mX\u001b[49m\u001b[43m,\u001b[49m\u001b[43my\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m      4\u001b[0m rfe\u001b[38;5;241m.\u001b[39mget_feature_names_out\n",
      "File \u001b[0;32m~/.local/lib/python3.10/site-packages/sklearn/base.py:870\u001b[0m, in \u001b[0;36mTransformerMixin.fit_transform\u001b[0;34m(self, X, y, **fit_params)\u001b[0m\n\u001b[1;32m    867\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mfit(X, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mfit_params)\u001b[38;5;241m.\u001b[39mtransform(X)\n\u001b[1;32m    868\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    869\u001b[0m     \u001b[38;5;66;03m# fit method of arity 2 (supervised transformation)\u001b[39;00m\n\u001b[0;32m--> 870\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfit\u001b[49m\u001b[43m(\u001b[49m\u001b[43mX\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43my\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mfit_params\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241m.\u001b[39mtransform(X)\n",
      "File \u001b[0;32m~/.local/lib/python3.10/site-packages/sklearn/feature_selection/_rfe.py:235\u001b[0m, in \u001b[0;36mRFE.fit\u001b[0;34m(self, X, y, **fit_params)\u001b[0m\n\u001b[1;32m    215\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mfit\u001b[39m(\u001b[38;5;28mself\u001b[39m, X, y, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mfit_params):\n\u001b[1;32m    216\u001b[0m     \u001b[38;5;124;03m\"\"\"Fit the RFE model and then the underlying estimator on the selected features.\u001b[39;00m\n\u001b[1;32m    217\u001b[0m \n\u001b[1;32m    218\u001b[0m \u001b[38;5;124;03m    Parameters\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    233\u001b[0m \u001b[38;5;124;03m        Fitted estimator.\u001b[39;00m\n\u001b[1;32m    234\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[0;32m--> 235\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_fit\u001b[49m\u001b[43m(\u001b[49m\u001b[43mX\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43my\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mfit_params\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/.local/lib/python3.10/site-packages/sklearn/feature_selection/_rfe.py:299\u001b[0m, in \u001b[0;36mRFE._fit\u001b[0;34m(self, X, y, step_score, **fit_params)\u001b[0m\n\u001b[1;32m    296\u001b[0m estimator\u001b[38;5;241m.\u001b[39mfit(X[:, features], y, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mfit_params)\n\u001b[1;32m    298\u001b[0m \u001b[38;5;66;03m# Get importance and rank them\u001b[39;00m\n\u001b[0;32m--> 299\u001b[0m importances \u001b[38;5;241m=\u001b[39m \u001b[43m_get_feature_importances\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    300\u001b[0m \u001b[43m    \u001b[49m\u001b[43mestimator\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    301\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mimportance_getter\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    302\u001b[0m \u001b[43m    \u001b[49m\u001b[43mtransform_func\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43msquare\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m    303\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    304\u001b[0m ranks \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39margsort(importances)\n\u001b[1;32m    306\u001b[0m \u001b[38;5;66;03m# for sparse case ranks is matrix\u001b[39;00m\n",
      "File \u001b[0;32m~/.local/lib/python3.10/site-packages/sklearn/feature_selection/_base.py:208\u001b[0m, in \u001b[0;36m_get_feature_importances\u001b[0;34m(estimator, getter, transform_func, norm_order)\u001b[0m\n\u001b[1;32m    206\u001b[0m         getter \u001b[38;5;241m=\u001b[39m attrgetter(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mfeature_importances_\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m    207\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m--> 208\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m    209\u001b[0m             \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mwhen `importance_getter==\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mauto\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m`, the underlying \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    210\u001b[0m             \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mestimator \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mestimator\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__class__\u001b[39m\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__name__\u001b[39m\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m should have \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    211\u001b[0m             \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m`coef_` or `feature_importances_` attribute. Either \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    212\u001b[0m             \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mpass a fitted estimator to feature selector or call fit \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    213\u001b[0m             \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mbefore calling transform.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    214\u001b[0m         )\n\u001b[1;32m    215\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    216\u001b[0m     getter \u001b[38;5;241m=\u001b[39m attrgetter(getter)\n",
      "\u001b[0;31mValueError\u001b[0m: when `importance_getter=='auto'`, the underlying estimator KNeighborsClassifier should have `coef_` or `feature_importances_` attribute. Either pass a fitted estimator to feature selector or call fit before calling transform."
     ]
    }
   ],
   "source": [
    "from sklearn.feature_selection import RFE\n",
    "rfe=RFE(knn,n_features_to_select=5)\n",
    "rfe.fit(X,y)\n",
    "rfe.get_feature_names_out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1a724ef",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}