SOLUTION6.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 任务6 - 模型融合（2天）\n",
    "任务6：模型融合，模型融合方式任意，并结合Task5给出你的最优结果。时间：2天\n",
    "* 例如Stacking融合，用你目前评分最高的模型作为基准模型，和其他模型进行stacking融合，得到最终模型及评分结果。\n",
    "\n",
    "[参考答案](./../参考答案)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task6（模型融合）** - 模型融合，模型融合方式任意，并结合Task5给出你的最优结果。\n",
    "例如Stacking融合，用你目前评分最高的模型作为基准模型，和其他模型进行stacking融合，得到最终模型及评分结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据集划分"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Anaconda2\\lib\\site-packages\\sklearn\\preprocessing\\data.py:645: DataConversionWarning: Data with input dtype uint8, int64, float64 were all converted to float64 by StandardScaler.\n",
      "  return self.partial_fit(X, y)\n",
      "D:\\Anaconda2\\lib\\site-packages\\sklearn\\base.py:464: DataConversionWarning: Data with input dtype uint8, int64, float64 were all converted to float64 by StandardScaler.\n",
      "  return self.fit(X, **fit_params).transform(X)\n",
      "D:\\Anaconda2\\lib\\site-packages\\ipykernel_launcher.py:19: DataConversionWarning: Data with input dtype uint8, int64, float64 were all converted to float64 by StandardScaler.\n"
     ]
    }
   ],
   "source": [
    "import pickle\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# 载入数据\n",
    "with open('final.pkl', 'rb') as f:\n",
    "    final_data = pickle.load(f)\n",
    "# data = pd.read_csv(\"./data.csv\",encoding='gbk')\n",
    "# y=data.status\n",
    "# 划分训练集测试集\n",
    "X, y = final_data[final_data.columns.drop(\"status\")], final_data['status']\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=2018)\n",
    "\n",
    "\n",
    "# 特征归一化\n",
    "std = StandardScaler()\n",
    "X_train = std.fit_transform(X_train)\n",
    "X_test = std.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n",
    "from sklearn.metrics import roc_auc_score,roc_curve, auc\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "model_score_train = []\n",
    "roc_score_train = []\n",
    "model_score_test = []\n",
    "roc_score_test = []\n",
    "from sklearn.metrics import accuracy_score, roc_auc_score,precision_score\n",
    "\n",
    "def model_metrics(clf, X_train, X_test, y_train, y_test):\n",
    "    # 预测\n",
    "    y_train_pred = clf.predict(X_train)\n",
    "    y_test_pred = clf.predict(X_test)\n",
    "    \n",
    "    y_train_proba = clf.predict_proba(X_train)[:,1]\n",
    "    y_test_proba = clf.predict_proba(X_test)[:,1]\n",
    "    \n",
    "    \n",
    "    # 准确率\n",
    "    print(u'[准确率]')\n",
    "    print(u'训练集：'+ '%.4f'%accuracy_score(y_train, y_train_pred))\n",
    "    print(u'测试集：'+ '%.4f'%accuracy_score(y_test, y_test_pred))\n",
    "    \n",
    "    # 精准率\n",
    "    print(u'[精准率]')\n",
    "    print(u'训练集：'+ '%.4f'%precision_score(y_train, y_train_pred))\n",
    "    print(u'测试集：'+ '%.4f'%precision_score(y_test, y_test_pred))\n",
    "\n",
    "    # 召回率\n",
    "    print(u'[召回率]')\n",
    "    print(u'训练集：'+ '%.4f'%recall_score(y_train, y_train_pred))\n",
    "    print(u'测试集：'+ '%.4f'%recall_score(y_test, y_test_pred))\n",
    "    \n",
    "    # f1-score\n",
    "    print(u'[f1-score]')\n",
    "    print(u'训练集：'+ '%.4f'%f1_score(y_train, y_train_pred))\n",
    "    print(u'测试集：'+ '%.4f'%f1_score(y_test, y_test_pred))\n",
    "    \n",
    "    # auc取值：用roc_auc_score或auc\n",
    "    print(u'[auc值]')\n",
    "    print(u'训练集：'+ '%.4f'%roc_auc_score(y_train, y_train_proba))\n",
    "    print(u'测试集：'+ '%.4f'%roc_auc_score(y_test, y_test_proba))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型融合 - Stacking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn import svm\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from xgboost.sklearn import XGBClassifier\n",
    "from lightgbm.sklearn import LGBMClassifier\n",
    "\n",
    "from mlxtend.classifier import StackingClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 模型调优后得到的参数\n",
    "lr = LogisticRegression(C = 0.01, penalty = 'l2')\n",
    "svm_linear = svm.SVC(C = 0.01, kernel = 'linear', probability=True)\n",
    "svm_poly =  svm.SVC(C = 0.01, kernel = 'poly', probability=True)\n",
    "svm_rbf =  svm.SVC(gamma = 0.01, C =0.01 , probability=True)\n",
    "svm_sigmoid =  svm.SVC(C = 0.01, kernel = 'sigmoid',probability=True)\n",
    "dt = DecisionTreeClassifier(max_depth=5,min_samples_split=300,min_samples_leaf=90,max_features=7)\n",
    "xgb = XGBClassifier(learning_rate =0.01, n_estimators=180, max_depth=3, min_child_weight=5, \n",
    "                    gamma=0.4, subsample=0.5, colsample_bytree=0.9, reg_alpha=1, \n",
    "                    objective= 'binary:logistic', nthread=4,scale_pos_weight=1, seed=27)\n",
    "lgb = LGBMClassifier(learning_rate =0.1, n_estimators=40, max_depth=3, min_child_weight=11, \n",
    "                    gamma=0.1, subsample=0.5, colsample_bytree=0.8, reg_alpha=1e-5, \n",
    "                    nthread=4,scale_pos_weight=1, seed=27)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 调包实现\n",
    "\n",
    "> 使用4种SVM、决策树、XGBoost和LightGBM作为初级分类器，LR作为次级分类器。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1）将初级分类器产生的类别标签作为新特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sclf = StackingClassifier(classifiers=[svm_linear, svm_poly, svm_rbf, \n",
    "                                       svm_sigmoid, dt, xgb, lgb], meta_classifier=lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[准确率]\n",
      "训练集：0.8188\n",
      "测试集：0.7821\n",
      "[精准率]\n",
      "训练集：0.8387\n",
      "测试集：0.6644\n",
      "[召回率]\n",
      "训练集：0.3429\n",
      "测试集：0.2702\n",
      "[f1-score]\n",
      "训练集：0.4868\n",
      "测试集：0.3842\n",
      "[auc值]\n",
      "训练集：0.6993\n",
      "测试集：0.6528\n"
     ]
    }
   ],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "sclf.fit(X_train, y_train.values)\n",
    "model_metrics(sclf, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2）将初级分类器产生的输出类概率作为新特征"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对输出概率use_probas=True，有两种不同的处理方式。\n",
    "\n",
    ">假设有2个初级分类器和3个类别输出概率：p1=[0.2,0.5,0.3],p2=[0.3,0.4,0.4]。\n",
    "\n",
    ">如果average_probas=True，则对分类器的结果求平均，得到：p=[0.25,0.45,0.35]\n",
    "\n",
    ">[推荐]如果average_probas=False，则分类器的所有结果都保留作为新的特征：p=[0.2,0.5,0.3,0.3,0.4,0.4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sclf = StackingClassifier(classifiers=[svm_linear, svm_poly, svm_rbf, svm_sigmoid, dt, xgb, lgb], \n",
    "                            meta_classifier=lr, use_probas=True,average_probas=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[准确率]\n",
      "训练集：0.8130\n",
      "测试集：0.7807\n",
      "[精准率]\n",
      "训练集：0.7944\n",
      "测试集：0.6554\n",
      "[召回率]\n",
      "训练集：0.3429\n",
      "测试集：0.2702\n",
      "[f1-score]\n",
      "训练集：0.4791\n",
      "测试集：0.3826\n",
      "[auc值]\n",
      "训练集：0.8487\n",
      "测试集：0.7819\n"
     ]
    }
   ],
   "source": [
    "sclf.fit(X_train, y_train.values)\n",
    "model_metrics(sclf, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}