Feature Selection (Chi Square).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "843165a9",
   "metadata": {},
   "source": [
    "chi square test is calculated between feature that have no nogetive values and target variable\n",
    "     This score is used to evaluate <b>categorical variable in a classification task</b> This score can be used to select        the n_features features with the highest values for the test chi-squared statistic from X, which must contain        only non-negative features such as booleans or frequencies (e.g., term counts in document classification),          relative to the classes.\n",
    "     \n",
    "Recall that the chi-square test measures dependence between stochastic variables, so using this function “weeds      out” the features that are the most likely to be independent of class and therefore irrelevant for                  classification. The Chi Square statistic is commonly used for testing relationships between categorical              variables.\n",
    "\n",
    "It compares the observed distribution of the different classes of target Y among the different categories of        the feature, against the expected distribution of the target classes, regardless of the feature categories.\n",
    "     \n",
    "     Statistical test: https://www.youtube.com/watch?v=4-rxTA_5_xA\n",
    "                       https://www.youtube.com/watch?v=YrhlQB3mQFI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "aaadf10b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>embarked</th>\n",
       "      <th>class</th>\n",
       "      <th>who</th>\n",
       "      <th>adult_male</th>\n",
       "      <th>deck</th>\n",
       "      <th>embark_town</th>\n",
       "      <th>alive</th>\n",
       "      <th>alone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C</td>\n",
       "      <td>First</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>S</td>\n",
       "      <td>First</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   survived  pclass     sex   age  sibsp  parch     fare embarked  class  \\\n",
       "0         0       3    male  22.0      1      0   7.2500        S  Third   \n",
       "1         1       1  female  38.0      1      0  71.2833        C  First   \n",
       "2         1       3  female  26.0      0      0   7.9250        S  Third   \n",
       "3         1       1  female  35.0      1      0  53.1000        S  First   \n",
       "4         0       3    male  35.0      0      0   8.0500        S  Third   \n",
       "\n",
       "     who  adult_male deck  embark_town alive  alone  \n",
       "0    man        True  NaN  Southampton    no  False  \n",
       "1  woman       False    C    Cherbourg   yes  False  \n",
       "2  woman       False  NaN  Southampton   yes   True  \n",
       "3  woman       False    C  Southampton   yes  False  \n",
       "4    man        True  NaN  Southampton    no   True  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "df=sns.load_dataset('titanic')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d2ba8921",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['anagrams',\n",
       " 'anscombe',\n",
       " 'attention',\n",
       " 'brain_networks',\n",
       " 'car_crashes',\n",
       " 'diamonds',\n",
       " 'dots',\n",
       " 'dowjones',\n",
       " 'exercise',\n",
       " 'flights',\n",
       " 'fmri',\n",
       " 'geyser',\n",
       " 'glue',\n",
       " 'healthexp',\n",
       " 'iris',\n",
       " 'mpg',\n",
       " 'penguins',\n",
       " 'planets',\n",
       " 'seaice',\n",
       " 'taxis',\n",
       " 'tips',\n",
       " 'titanic']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sns.get_dataset_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0fc46dd2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sex</th>\n",
       "      <th>embarked</th>\n",
       "      <th>alone</th>\n",
       "      <th>pclass</th>\n",
       "      <th>survived</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>male</td>\n",
       "      <td>S</td>\n",
       "      <td>False</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>female</td>\n",
       "      <td>C</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>female</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>female</td>\n",
       "      <td>S</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>male</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      sex embarked  alone  pclass  survived\n",
       "0    male        S  False       3         0\n",
       "1  female        C  False       1         1\n",
       "2  female        S   True       3         1\n",
       "3  female        S  False       1         1\n",
       "4    male        S   True       3         0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "##['sex','embarked','alone','pclass','Survived']\n",
    "# before applying chi square test we have to first use label encoding\n",
    "df=df[['sex','embarked','alone','pclass','survived']]\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5bde459e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sex</th>\n",
       "      <th>embarked</th>\n",
       "      <th>alone</th>\n",
       "      <th>pclass</th>\n",
       "      <th>survived</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sex  embarked  alone  pclass  survived\n",
       "0    1         0      0       3         0\n",
       "1    0         1      0       1         1\n",
       "2    0         0      1       3         1\n",
       "3    0         0      0       1         1\n",
       "4    1         0      1       3         0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Let's perform label encoding on sex column\n",
    "import numpy as np\n",
    "df['sex']=np.where(df['sex']==\"male\",1,0)\n",
    "### let's perform label encoding on embarked\n",
    "ordinal_label = {k: i for i, k in enumerate(df['embarked'].unique(), 0)}\n",
    "df['embarked'] = df['embarked'].map(ordinal_label)\n",
    "### let's perform label encoding on alone\n",
    "df['alone']=np.where(df['alone']==True,1,0)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "70fe3358",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sex</th>\n",
       "      <th>embarked</th>\n",
       "      <th>alone</th>\n",
       "      <th>pclass</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>794</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>161</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>815</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     sex  embarked  alone  pclass\n",
       "69     1         0      0       3\n",
       "85     0         0      0       3\n",
       "794    1         0      1       3\n",
       "161    0         0      1       2\n",
       "815    1         0      1       1"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### train Test split is usually done to avaoid overfitting\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train,X_test,y_train,y_test=train_test_split(df[['sex','embarked','alone','pclass']],\n",
    "                                              df['survived'],test_size=0.3,random_state=100)\n",
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2046d060",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([65.67929505,  7.55053653, 10.88471585, 21.97994154]),\n",
       " array([5.30603805e-16, 5.99922095e-03, 9.69610546e-04, 2.75514881e-06]))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Perform chi2 test\n",
    "### chi2 returns 2 values\n",
    "### Fscore and the pvalue\n",
    "# more higher the f score more important the feature is, lower in case of p value\n",
    "from sklearn.feature_selection import chi2\n",
    "f_p_values=chi2(X_train,y_train)\n",
    "f_p_values # first array f score , p value "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "93aa3bc6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sex         5.306038e-16\n",
       "embarked    5.999221e-03\n",
       "alone       9.696105e-04\n",
       "pclass      2.755149e-06\n",
       "dtype: float64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "p_values=pd.Series(f_p_values[1],index=X_train.columns) # pandas has 2 data structure - series,dataframe\n",
    "p_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7757af85",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sex         5.306038e-16\n",
       "pclass      2.755149e-06\n",
       "embarked    5.999221e-03\n",
       "alone       9.696105e-04\n",
       "dtype: float64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_values.sort_index(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d4d6fc9e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sex         65.679295\n",
       "pclass      21.979942\n",
       "embarked     7.550537\n",
       "alone       10.884716\n",
       "dtype: float64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f_score=pd.Series(f_p_values[0],index=X_train.columns) # pandas has 2 data structure - series,dataframe\n",
    "f_score.sort_index(ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8771f217",
   "metadata": {},
   "source": [
    "## Observation\n",
    "sex column is the most important column when compared to the output feature"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "735b6cbf",
   "metadata": {},
   "source": [
    "### F-score ( Fisher score )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3419c7b",
   "metadata": {},
   "source": [
    "Fisher score is one of the most widely used supervised feature selection methods . However it selects each feature independently according to score under the feature criterion , which finds a suboptimal subset of features "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "423c676c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}