02_end_to_end_machine_learning_project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第二章 - 从零开始的端到端机器学习项目\n",
    "在本章中，我们要开始从零构建一个基于1990年加利福尼亚人口普查的数据预测该地区的房价情况的项目。\n",
    "\n",
    "在开始一个机器学习项目之前，一个合格的数据科学家需要考虑以下要求项目：\n",
    "- 1.纵览大局；\n",
    "- 2.获取数据；\n",
    "- 3.探索数据，发现问题；\n",
    "- 4.清洗数据以更适应接下来的学习算法；\n",
    "- 5.模型好中选优，开始训练；\n",
    "- 6.微调模型；\n",
    "- 7.展示自己的解决方案；\n",
    "- 8.启动、监视和维护自己的系统；\n",
    "\n",
    "**第二章的项目也会从这八个方面进行一一阐释和分解。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.纵览大局（Look at the Big Picture）\n",
    "概览：根据加州人口普查数据（多种指标，人口数量/收入中位数等）建立加州的房价模型。\n",
    "### 1.1 问题定义\n",
    "稍加分析容易得到这是一个多变量回归问题。\n",
    "### 1.2 性能指标选择\n",
    "- 均方根误差（RMSE）\n",
    "- 平均绝对误差（MAE）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.获取数据\n",
    "### 2.0定义常用函数\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import os\n",
    "\n",
    "# 随机数保证notebook全程输入稳定\n",
    "np.random.seed(22)\n",
    "\n",
    "# 导入绘图工具\n",
    "%matplotlib inline\n",
    "import matplotlib \n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams[\"axes.labelsize\"] = 14\n",
    "plt.rcParams['xtick.labelsize'] = 12\n",
    "plt.rcParams['ytick.labelsize'] = 12\n",
    "\n",
    "# 设置图片保存路径函数\n",
    "PROJECT_ROOT_DIR = \"G:/Jupyter Notebook/Hands-on-Machine-Learning-2.0\"\n",
    "CHAPTER_ID = \"02_End_to_End_project\"\n",
    "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format=fig_extension, dpi=resolution)\n",
    "\n",
    "# Ignore useless warnings (see SciPy issue #5998)\n",
    "import warnings\n",
    "warnings.filterwarnings(action=\"ignore\", message=\"^internal gelsd\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 下载数据到本地"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import tarfile \n",
    "from six.moves import urllib\n",
    "DOWNLOAD_ROOT = \"https://raw.githubusercontent.com/ageron/handson-ml2/master/\"\n",
    "HOUSING_PATH = os.path.join(\"datasets\", \"housing\")\n",
    "HOUSING_URL = DOWNLOAD_ROOT + \"datasets/housing/housing.tgz\"\n",
    "def fetch_housing_data(housing_url=HOUSING_URL,housing_path=HOUSING_PATH):\n",
    "    if not os.path.isdir(housing_path):\n",
    "        os.makedirs(housing_path)\n",
    "    tgz_path = os.path.join(housing_path,\"housing.tgz\")\n",
    "    urllib.request.urlretrieve(housing_url,tgz_path)\n",
    "    housing_tgz = tarfile.open(tgz_path)\n",
    "    housing_tgz.extractall(path=housing_path)\n",
    "    housing_tgz.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fetch_housing_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 探索数据，发现问题"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# 载入并读取数据\n",
    "def load_housing_data(housing_path=HOUSING_PATH):\n",
    "    csv_path = os.path.join(housing_path,\"housing.csv\")\n",
    "    return pd.read_csv(csv_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 查看数据大小和各种属性\n",
    "housing = load_housing_data()\n",
    "housing.head()\n",
    "# 每一行代表一个区，总共有10个属性。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# info()方法可以快速获取数据的简单描述，特别是总行数，每个属性的类型和非空值的数量，比如total_bedrooms属性只有20433个非空值，要小心。\n",
    "housing.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# value_counts()方法查看有多少种分类存在，每个分类下有多少数目。\n",
    "housing[\"ocean_proximity\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 可以再试一下\n",
    "housing[\"latitude\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 显示数值属性的摘要（这里的计算空值会被忽略）\n",
    "housing.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 请注意，缺失值已经被自动忽略了（例如，total_bedrooms的计数为20,433，而不是20,640） \n",
    "* std行显示标准差（测量值的分散程度）。 \n",
    "* 25％，50％和75％的行显示相应的百分位数：百分位数表示一组观察中给定百分比观察值下降的值。 例如，\n",
    "  \n",
    "  i. **25％**的地区的**housing_median_age**低于18，而**50％**低于29，**75％**低于37.\n",
    "\n",
    "  ii. 这些通常被称为第25百分位数（或**第1四分位数**），中位数和 第75百分位数（或**第3个四分位数**）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 另外一种快速了解数据的方法就是直方图：直方图用来显示给定值范围（横轴）的实例数量（纵轴）。\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as pyplot\n",
    "housing.hist(bins=50,figsize=(20,15)) #hist()方法\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "请注意这些直方图中的一些内容：\n",
    "\n",
    "1.首先，收入中位数属性看起来不像以美元表示（USD）。在与收集数据的团队核实后，被告知数据已经缩放并且**上限为15**（实际为15.0001），以获得更高的中位数收入，低收入中位数为0.5（实际为0.4999）。使用预处理属性在机器学习中很常见，并不一定是问题，但你应该至少了解数据是如何计算的。\n",
    "\n",
    "2.**房屋年龄中位数**和**房屋价格中位数**也被限制了上界。而后者正是我们所要计算的目标属性（标签）。你的机器学习算法可能会发现价格永远不会超过这个限制。您需要与您的客户团队核实，看看这是否是一个问题。如果他们告诉你他们需要精确的预测，甚至超过50万美元的部分也需要，那么你会有两种选择：\n",
    "\n",
    "* 被那些标签值设置了上限的地区，重新收集标签值。\n",
    "* 从训练集中删除这部分数据（以及从测试集中删除,因为如果预测超出$ 500,000，系统不应被评价为bad。） \n",
    "\n",
    "\n",
    "3.这些属性的尺度差别很大。我们将在本章后面我们**探索特征缩放**时讨论这些。\n",
    "\n",
    "4.最后，许多直方图都是**重尾**的：它们在**中位数右侧比在左侧延伸得更远**。这可能会使某些机器学习算法难以检测模式。稍后我们将尝试转换这些属性以获得更多钟形分布。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 通过采样创建测试集\n",
    "在这个阶段就分割数据有些很奇怪。毕竟，你只是简单快速地查看了数据而已，你需要再仔细调查下数据以决定使用什么算法。这么想是对的，但是人类的大脑是一个神奇的发现规律的系统，这意味着大脑非常容易发生过拟合：如果你查看了测试集，就会不经意地按照测试集中的规律来选择某个特定的机器学习模型。再当你使用测试集来评估误差率时，就会导致评估过于乐观，(相当于作弊提前看答案而没有真正掌握知识。)而实际部署的系统表现就会很差。这称为**数据透视偏差**。\n",
    "\n",
    "通常随机选一些实例，构建训练集。通常是数据集的20%(如果数据集很大的话，则可以少一些)；\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使得notebook的输出在每一层运行的时候都保持稳定\n",
    "import numpy as np \n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_train_test(data,test_ratio):\n",
    "    shuffled_indices = np.random.permutation(len(data))\n",
    "    test_set_size = int(len(data)*test_ratio)\n",
    "    test_indices = shuffled_indices[:test_set_size]\n",
    "    train_indices = shuffled_indices[test_set_size:]\n",
    "    return data.iloc[train_indices],data.iloc[test_indices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# ok，我们调用一下上面定义的函数\n",
    "train_set,test_set = split_train_test(housing,0.2)\n",
    "print(len(train_set),\"train +\",len(test_set),\"test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个方法是可行的，但是并不是完美的：如果我们再次运行程序，就会产生一个不同的测试集！多次运行之后，我们的模型就会“看到”整个数据集，这点我们需要避免。\n",
    "解决的办法\n",
    "\n",
    "1. 保存第一次运行得到的测试集，并在随后的过程加载。\n",
    "\n",
    "2. 在调用np.random.permutation()之前，设置随机数生成器的种子（比如np.random.seed(42)），以产生总是相同的随机索引（shuffled indices）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但是如果数据集更新了，上述的两个方法都会失效。一个通常的解决办法是使用每个实例的ID来判定这个实例是否应该放入测试集（假设每个实例都有唯一并且不变的ID）。\n",
    "\n",
    "例如，我们可以计算每个实例的标识符的哈希值，如果哈希值低于或等于最大哈希值的20%，就把该实例放入测试集。这样可以确保测试集在多次运行中保持一致，即使你更像数据集。新的测试集将包含20%的新实例，但不会包含之前在训练集中的任何实例。下面是一种可用的方法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from zlib import crc32\n",
    "def test_set_check(identifier,test_ratio):\n",
    "    return crc32(np.int64(identifier)) & 0xffffffff < test_ratio * 2**32\n",
    "\n",
    "def split_train_test_by_id(data, test_ratio, id_column):\n",
    "    ids = data[id_column]\n",
    "    in_test_set = ids.apply(lambda id_: test_set_check(id_, test_ratio))\n",
    "    return data.loc[~in_test_set], data.loc[in_test_set]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "不过，房产数据集没有ID这一列。最简单的方法是使用行索引作为 ID："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing_with_id = housing.reset_index()\n",
    "train_set, test_set = split_train_test_by_id(housing_with_id,0.2,\"index\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果使用行索引作为唯一识别码，你需要保证新数据都放到现有数据的尾部，且没有行被删除。如何达不到这样的要求，则可以用最stable的特征来创建唯一识别码。例如，一个区的维度和经度在几百万年之内是不变的，所以可以将两者结合成一个ID："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing_with_id[\"id\"] = housing[\"longitude\"]*1000+housing[\"latitude\"]\n",
    "train_set,test_set = split_train_test_by_id(housing_with_id,0.2,\"id\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_set.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scikit-Learn 提供了一些函数，可以用多种方式将数据集分割成多个子集。最简单的函数是`\n",
    "train_test_split`，它的作用和之前的函数`split_train_test`很像，并带有一些额外的参数。\n",
    "- 首先，它有一个`random_state`参数，可以设定前面讲过的随机生成器种子；\n",
    "- 其次，你可以将种子传递给多个行数相同的数据集，可以在相同的索引上分割数据集（这个功能非常有用，比如你的标签值是放在另一个DataFrame里的,好用谁用谁知道）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "train_set,test_set = train_test_split(housing,test_size=0.2,random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "目前为止，我们所采用的都是完全随机的取样方法。当你的数据集很大时（尤其是和属性数相比），这种做法非常好；\n",
    "\n",
    "但如果数据集不大，就会有**采样偏差**的风险。\n",
    "\n",
    "当一个调查公司想要对 1000 个人进行调查，它们不是在电话亭里随机选 1000 个人出来。调查公司要保证这 1000 个人对人群整体有代表性。例如，美国人口的 51.3% 是女性，48.7% 是男性。所以在美国，严谨的调查需要保证样本也是这个比例：513 名女性，487 名男性。\n",
    "\n",
    "这称作分层采样（stratified sampling）：将人群分成均匀的子分组，称为分层，从每个分层去取合适数量的实例，以保证测试集对总人数有代表性。如果调查公司采用纯随机采样，会有 12% 的概率导致采样偏差：女性人数少于 49%，或多于 54%。不管发生那种情况，调查结果都会严重偏差。\n",
    "\n",
    "假设专家告诉你，收入中位数是预测房价中位数非常重要的属性。这个时候我们可能需要保证测试集可以代表整体数据集中的各种收入分类。因为收入中位数是一个连续的数值属性，你首先需要创建一个收入类别属性。再仔细地看一下收入中位数的柱状图（图 2-9）（译注：该图是对收入中位数处理过后的图）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing[\"median_income\"].hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从上图我们可以看到，\n",
    "- 大多数收入中位数集中在2~5（万美元）\n",
    "- 但是也有一部分远远超过了6(万美元)\n",
    "\n",
    "在你的数据集中，每个分层都要有足够数量的实例，这点很重要，否则对分层的重要性的评估可能会有偏差。这意味着你不应该有太多的分层，而且每个分层应该足够大。\n",
    "\n",
    "下面的代码使用`pd.cut()`函数创建了一个收入类别属性，有5个类别（标注从1到5）：类别1的范围是0到1.5（即小于15000美元），类别2的范围是1.5到3，以此类推，最后将所有的大于5的分类归于分类5："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing[\"income_cat\"] = pd.cut(housing[\"median_income\"],bins=[0.,1.5,3.0,4.5,6.,np.inf],labels=[1,2,3,4,5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 对收入中位数按照1万美元，2万美元，3万美元，4万美元，5万及5万以上进行统计\n",
    "housing[\"income_cat\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 对中位数做直方图\n",
    "housing[\"income_cat\"].hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在，您可以根据收入类别进行分层抽样了。为此，您可以使用Scikit-Learn的StratifiedShuffleSplit类（分层随机采样）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedShuffleSplit\n",
    "\n",
    "split = StratifiedShuffleSplit(n_splits=1,test_size=0.2,random_state=42)\n",
    "for train_index, test_index in split.split(housing, housing[\"income_cat\"]):\n",
    "    start_train_set = housing.loc[train_index]\n",
    "    start_test_set = housing.loc[test_index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 我们可以查看测试集的收入类别比例\n",
    "start_test_set[\"income_cat\"].value_counts()/ len(start_test_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def income_cat_proportions(data):\n",
    "    return data[\"income_cat\"].value_counts() / len(data)\n",
    "\n",
    "train_set, test_set = train_test_split(housing, test_size=0.2, random_state=42)\n",
    "\n",
    "compare_props = pd.DataFrame({\n",
    "    \"Overall\": income_cat_proportions(housing),\n",
    "    \"Stratified\": income_cat_proportions(start_test_set),\n",
    "    \"Random\": income_cat_proportions(test_set),\n",
    "}).sort_index()\n",
    "compare_props[\"Rand. %error\"] = 100 * compare_props[\"Random\"] / compare_props[\"Overall\"] - 100\n",
    "compare_props[\"Strat. %error\"] = 100 * compare_props[\"Stratified\"] / compare_props[\"Overall\"] - 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "compare_props"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下图对比了总数据集、分层采样的测试集、纯随机采样测试集的收入分类比例。可以看到，分层采样测试集的收入分类比例与总数据集几乎相同，而随机采样数据集偏差严重。\n",
    "\n",
    "![image-20200723195139607](https://cdn.jsdelivr.net/gh/chenhaishun/test_pic@master/typora202007/23/195140-651702.png)\n",
    "\n",
    "图 分层采样和纯随机采样的样本偏差比较"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for set_ in (start_train_set, start_test_set):\n",
    "    set_.drop(\"income_cat\", axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "测试集通常被忽略，所以我们花了大量的时间来生成测试集。实际测试集是机器学习非常重要的一部分。还有，生成测试集过程中的许多思路对于后面的交叉验证讨论是非常有帮助的。接下来进入下一阶段：探索数据，发现问题；"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.探索数据，发现问题。\n",
    "目前为止，我们只是快速查看了数据，对要处理的数据有了整体了解。现在的目标是进一步探索数据。\n",
    "\n",
    "首先，保证你将测试集放在了一旁，只是研究训练集。\n",
    "\n",
    "另外，如果训练集非常大，你可以再采样一个探索集，保证操作方便快速。在我们的案例中，数据集很小，所以可以在全集上直接工作。我们创建一个副本，以免损伤训练集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing = start_train_set.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 地理数据可视化\n",
    "因为存在地理信息（纬度和经度），创建一个所有街区的散点图来数据可视化很有意思。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing.plot(kind=\"scatter\",x=\"longitude\",y=\"latitude\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这张图看起来很像加州，但是看不出什么特别的规律。将alpha设为 0.1，可以更容易看出数据点的密度："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing.plot(kind=\"scatter\", x=\"longitude\", y=\"latitude\", alpha=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在看起来好多了：可以非常清楚地看到高密度区域，即湾区、洛杉矶和圣地亚哥，以及中央谷，特别是从萨克拉门托和弗雷斯诺周围。通常来讲，人类的大脑非常善于发现图片中的规律，但是需要调整可视化参数使规律显现出来。\n",
    "\n",
    "下图中我们可以很好地看到房价分布。\n",
    "\n",
    "每个圈的半径表示街区的人口（选项s），\n",
    "\n",
    "颜色代表价格（选项c）。\n",
    "\n",
    "我们用预先定义的名为jet的颜色图（选项cmap），它的范围是从蓝色（低价）到红色（高价）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "housing.plot(kind=\"scatter\", x=\"longitude\", y=\"latitude\", alpha=0.4,\n",
    "    s=housing[\"population\"]/100, label=\"population\", figsize=(10,7),\n",
    "    c=\"median_house_value\", cmap=plt.get_cmap(\"jet\"), colorbar=True,\n",
    "    sharex=False)\n",
    "plt.legend()\n",
    "save_fig(\"housing_prices_scatterplot\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import matplotlib.image as mpimg\n",
    "# 读取加州地图，下面的代码将上图放在了加利福尼亚的地图之上，运行之前需要在图片保存路径中提前保存一张加州地图，以便读取。\n",
    "california_img=mpimg.imread(PROJECT_ROOT_DIR + '/images/02_end_to_end_project/california.png')\n",
    "# 在加州地图上绘制matplotlib subplot 对象。\n",
    "ax = housing.plot(kind=\"scatter\", x=\"longitude\", y=\"latitude\", figsize=(10,7),\n",
    "                       s=housing['population']/100, label=\"Population\",\n",
    "                       c=\"median_house_value\", cmap=plt.get_cmap(\"jet\"),\n",
    "                       colorbar=False, alpha=0.4,\n",
    "                      )\n",
    "plt.imshow(california_img, extent=[-124.55, -113.80, 32.45, 42.05], alpha=0.5,\n",
    "           cmap=plt.get_cmap(\"jet\"))\n",
    "plt.ylabel(\"Latitude\", fontsize=14)\n",
    "plt.xlabel(\"Longitude\", fontsize=14)\n",
    "\n",
    "prices = housing[\"median_house_value\"]\n",
    "tick_values = np.linspace(prices.min(), prices.max(), 11)\n",
    "cbar = plt.colorbar()\n",
    "cbar.ax.set_yticklabels([\"$%dk\"%(round(v/1000)) for v in tick_values], fontsize=14)\n",
    "cbar.set_label('Median House Value', fontsize=16)\n",
    "\n",
    "plt.legend(fontsize=16)\n",
    "save_fig(\"california_housing_prices_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从上图我们能够看出\n",
    "- 房价和**位置**有很大的关系，同时也和人口密度有很大的关系。因此我们就可以使用聚类算法检测主要的聚集\n",
    "- 海洋距离属性属性也可能有用，尽管在北加利福尼亚州沿海地区的房价不是太高，因此这不是一个简单的规则。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Looking for Correlations 查找相关性"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据集并不是非常大，可以很容易地使用`corr()`方法计算出每对属性间的**标准相关系数**（standard correlation coefficient，也称作皮尔逊相关系数）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corr_matrix = housing.corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看每个属性与房屋中值得相关程度："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corr_matrix[\"median_house_value\"].sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "相关系数的范围是 [[-1,1]].\n",
    "* 当它**接近1**时，就意味着有很强的**正相关**; 例如，当收入**中位数上升**时，**房屋中位数**价值往往会**上升**。 \n",
    "\n",
    "* 当系数**接近-1**时，意味着有强烈的**负相关**; 你可以在**纬度**和**房价中位数**之间看到一个轻微的**负相关**（即,当你往北时价格略有**下跌趋势**)。\n",
    "\n",
    "* 最后，接近零的系数意味着没有线性相关。\n",
    "\n",
    " 下图显示了各种图以及它们的水平轴和垂直轴之间的相关系数。\n",
    "\n",
    " ![image-20200723221657571](https://cdn.jsdelivr.net/gh/chenhaishun/test_pic@master/typora202007/23/221659-865472.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "还有一种检测属性间相关系数的方法是使用 Pandas 的`scatter_matrix`函数，它能画出每个数值属性对每个其它数值属性的图。因为现在共有 11 个数值属性，你可以得到11^2 = 121张图，在一页上画不下，所以只关注几个和房价中位数最有可能相关的属性。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pandas.plotting import scatter_matrix\n",
    "\n",
    "attributes = [\"median_house_value\", \"median_income\", \"total_rooms\",\n",
    "              \"housing_median_age\"]\n",
    "scatter_matrix(housing[attributes], figsize=(12, 8))\n",
    "save_fig(\"scatter_matrix_plot\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果 pandas 将每个变量对自己作图，主对角线（左上到右下）都会是直线图。所以 Pandas 展示的是每个属性的柱状图（也可以是其它的，请参考 Pandas 文档）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "housing.plot(kind=\"scatter\",x=\"median_income\",y=\"median_house_value\",alpha=0.1)\n",
    "plt.axis([0,16,0,550000])\n",
    "save_fig(\"income_vs_house_value_scatterplot\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从上图我们可以看到这样几点：\n",
    "- 首先，相关性确实很强；我们可以很清楚地看到上升趋势，并且点不是很分散。\n",
    "- 其次，我们之前看到的最高价，清晰地呈现为一条位于 500000 美元的水平线。这张图也呈现了一些不是那么明显的直线：一条位于 450000 美元的直线，一条位于 350000 美元的直线，一条在 280000 美元的线，和一些更靠下的线。\n",
    "\n",
    "为了避免我们的算法学习之后出现这些怪异的数据，可以尝试删除这些相应的区域。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Experimenting with Attribute Combinations-试验不同的属性组合\n",
    "到目前为止，\n",
    "- 我们已经了解了一些探索数据的方法，并且对数据也有了一定的了解，\n",
    "- 还对各属性之间的相关性进行了分析，特别是与目标属性之间的相关性。\n",
    "- 由于一些属性具有尾重的分布，所以我们可以对他们进行转换（比如对数转换），每个不同的项目历程都会不一样，但是大致思路相似。\n",
    "\n",
    "在将数据送进机器学习算法之前，我们可以想做的最后一件事是尝试各种组合属性，例如，\n",
    "- 在一个地区，如果我们不知道某个街区有多少户家庭，那么房间总数就不是很有用。因为我们真正想要的是**每个家庭的房间数量**。\n",
    "- 同样，卧室总数本身并不是很有用：你可能想把它与房间数量进行比较。而且每户人口也是如此。\n",
    "\n",
    "我们可以尝试组合这些属性。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 每个家庭平均拥有的房间数 = 总房间数 / 总的家庭数\n",
    "housing[\"rooms_per_household\"] = housing[\"total_rooms\"]/housing[\"households\"]\n",
    "\n",
    "# 总卧室数/总房间数\n",
    "housing[\"bedrooms_per_room\"] = housing[\"total_bedrooms\"]/housing[\"total_rooms\"]\n",
    "\n",
    "# 平均家庭人口数量 = 总人口 / 总家庭数\n",
    "housing[\"population_per_household\"]=housing[\"population\"]/housing[\"households\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 查看新的属性的相关性系数\n",
    "corr_matrix = housing.corr()\n",
    "corr_matrix[\"median_house_value\"].sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "再得到新的属性之后，我们可以发现\n",
    "- 与总房间数或卧室数相比，新的bedrooms_per_room属性与房价中位数的关联更强。显然，卧室数/总房间数的比例越低，房价就越高。每户的房间数也比街区的总房间数的更有信息，很明显，房屋越大，房价就越高。\n",
    "\n",
    "这一步的数据探索不必非常完备，此处的目的是有一个正确的开始，快速发现规律，以得到一个合理的原型。但是这是一个交互过程：一旦你得到了一个原型，并运行起来，你就可以分析它的输出，进而发现更多的规律，然后再回到数据探索这步。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 绘制散点图再看看\n",
    "housing.plot(kind=\"scatter\", x=\"rooms_per_household\", y=\"median_house_value\",\n",
    "             alpha=0.2)\n",
    "plt.axis([0, 5, 0, 520000])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 再次查看（多了三个属性）\n",
    "housing.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.清洗数据以更适应接下来的学习算法（Prepare the Data for Machine Learning Algorithms）\n",
    "现在来为机器学习算法准备数据。不要手工来做，你需要写一些函数，理由如下：\n",
    "\n",
    "- 函数可以让你在任何数据集上（比如，你下一次获取的是一个新的数据集）方便地进行重复数据转换。\n",
    "\n",
    "- 你会慢慢积累并建立一个转换函数库，可以在未来的数据挖掘项目中复用。\n",
    "\n",
    "- 在将数据传给算法之前，你可以在实时系统中使用这些函数。\n",
    "\n",
    "- 这可以让你方便地尝试多种数据转换，查看哪些转换方法结合起来效果最好。\n",
    "\n",
    "但是，还是先回到干净的训练集（通过再次复制strat_train_set），将预测变量和标签分开，因为我们不想对预测变量和目标值应用相同的转换（注意drop()创建了一份数据的备份，而不影响strat_train_set）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将我们的预测变量去除，同时创建数据副本，作为训练集。\n",
    "housing = start_train_set.drop(\"median_house_value\", axis=1) \n",
    "\n",
    "# 创建数据集标签，也就是我们的预测变量。\n",
    "housing_labels = start_train_set[\"median_house_value\"].copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以下代码**抽样取出部分新创建的训练集中的实例**，查看表头信息，验证\"median_house_value\"是否被去掉，如果被去掉，每个实例的属性应该只剩下9个，原来是10个"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_incomplete_rows = housing[housing.isnull().any(axis=1)].head()\n",
    "sample_incomplete_rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 数据清洗 Data Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "大多机器学习算法不能处理**缺失的特征**，因此先创建一些函数来处理特征缺失的问题。我们应该记得并且注意到了属性total_bedrooms有一些缺失值。有三个解决选项：\n",
    "\n",
    "- 去掉对应的街区；\n",
    "\n",
    "- 删除整个属性；\n",
    "\n",
    "- 进行赋值（0、平均值、中位数等等）\n",
    "\n",
    "我们可以使用DataFrame的`dropna()`，`drop()`和`fillna()`轻松完成这些操作：\n",
    "\n",
    "**注意：以下三个cell是对三个选择的使用演示，所以均使用上面刚刚抽样取出的新训练集的实例，这样真实的训练集并没有受到影响。**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 删除相应的区域\n",
    "sample_incomplete_rows.dropna(subset=[\"total_bedrooms\"]) \n",
    "# 从下面的结果也能看出来，所有的行都被删除。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 去掉“total_bedrooms”这个属性\n",
    "sample_incomplete_rows.drop(\"total_bedrooms\",axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 计算中值\n",
    "median = housing[\"total_bedrooms\"].median()\n",
    "sample_incomplete_rows[\"total_bedrooms\"].fillna(median, inplace=True) #将缺失值设置为某个值（零，均值，中位数等）\n",
    "sample_incomplete_rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果选择第三种办法，我们则需要计算**训练集**的中位数，用中位数填充训练集的缺失值，不要忘记保存该中位数。\n",
    "\n",
    "后面用测试集评估系统时，需要替换测试集中的缺失值，也可以用来实时(系统上线)替换新数据中的缺失值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此外，Scikit-Learn 提供了一个方便的类来处理缺失值：`Imputer`。下面是其使用方法：\n",
    "\n",
    "首先，需要创建一个Imputer实例，指定用某属性的中位数来替换该属性所有的缺失值："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.impute import SimpleImputer\n",
    "# 创建实例，并指定用中位数替换缺失值\n",
    "# “median”，也就是 “total_bedrooms” 的中值。\n",
    "imputer = SimpleImputer(strategy = \"median\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为只有数值属性才能算出中位数，我们需要创建一份不包括文本属性`ocean_proximity`的数据副本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing_num = housing.drop(\"ocean_proximity\",axis = 1)\n",
    "# 或者我们也可以这样写\n",
    "# housing_num = housing.select_dtypes(include=[np.number])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 现在，我们就可以用fit()方法将imputer实例拟合到训练数据\n",
    "imputer.fit(housing_num)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`imputer`计算出了每个属性的中位数，并将结果保存在了实例变量`statistics_`中。虽然此时只有属性`total_bedrooms`存在缺失值，但我们不能确定在以后的新的数据中会不会有其他属性也存在缺失值，所以安全的做法是将imputer应用到每个数值："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "imputer.statistics_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 检查计算的中值和前面手工技术按的中值是否相等\n",
    "housing_num.median().values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在，你就可以使用这个“训练过的”imputer来对训练集进行转换，将缺失值替换为中位数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = imputer.transform(housing_num)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "结果是一个包含转换后特征的普通的 Numpy 数组。如果你想将其放回到 PandasDataFrame中，也很简单："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing_tr = pd.DataFrame(X, columns=housing_num.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scikit-Learn 设计\n",
    "\n",
    "Scikit-Learn 设计的 API 设计的非常好。它的主要设计原则是：\n",
    "\n",
    "- 一致性：所有对象的接口一致且简单：\n",
    "\n",
    "  - 估计器（estimator）。任何可以基于数据集对一些参数进行估计的对象都被称为估计器（比如，imputer就是个估计器）。估计本身是通过fit()方法，只需要一个数据集作为参数（对于监督学习算法，需要两个数据集；第二个数据集包含标签）。任何其它用来指导估计过程的参数都被当做超参数（比如imputer的strategy），并且超参数要被设置成实例变量（通常通过构造器参数设置）。\n",
    "  - 转换器（transformer）。一些估计器（比如imputer）也可以转换数据集，这些估计器被称为转换器。API也是相当简单：转换是通过transform()方法，被转换的数据集作为参数。返回的是经过转换的数据集。转换过程依赖学习到的参数，比如imputer的例子。所有的转换都有一个便捷的方法fit_transform()，等同于调用fit()再transform()（但有时fit_transform()经过优化，运行的更快）。\n",
    "  -  预测器（predictor）。最后，一些估计器可以根据给出的数据集做预测，这些估计器称为预测器。例如，上一章的LinearRegression模型就是一个预测器：它根据一个国家的人均 GDP 预测生活满意度。预测器有一个predict()方法，可以用新实例的数据集做出相应的预测。预测器还有一个score()方法，可用于评估测试集（如果是监督学习算法的话，还要给出相应的标签）的预测质量。\n",
    "可检验。所有估计器的超参数都可以通过实例的public变量直接访问（比如，imputer.strategy），并且所有估计器学习到的参数也可以通过在实例变量名后加下划线来访问（比如，imputer.statistics_）。\n",
    "\n",
    "- 类不可扩散。数据集被表示成 NumPy 数组或 SciPy 稀疏矩阵，而不是自制的类。超参数只是普通的 Python 字符串或数字。\n",
    "\n",
    "- 可组合。尽可能使用现存的模块。例如，用任意的转换器序列加上一个估计器，就可以做成一个流水线，后面会看到例子。\n",
    "\n",
    "- 合理的默认值。Scikit-Learn 给大多数参数提供了合理的默认值，很容易就能创建一个系统。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing_tr = pd.DataFrame(X, columns=housing_num.columns,index=housing.index)\n",
    "housing_tr.loc[sample_incomplete_rows.index.values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "imputer.strategy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing_tr = pd.DataFrame(X, columns=housing_num.columns,index=housing_num.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing_tr.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 处理文本和类别属性"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在前面，我们丢弃了类别属性`ocean_proximity`，因为它是一个文本属性，我们不能计算出中位数。大多数机器学习算法更喜欢和数字打交道，所以让我们把这些文本标签转换为数字。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing_cat = housing[[\"ocean_proximity\"]]\n",
    "housing_cat.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scikit-Learn 为这个任务提供了一个转换器LabelEncoder："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "\n",
    "ordinal_encoder = OrdinalEncoder() # 实例化\n",
    "housing_cat_encoded = ordinal_encoder.fit_transform(housing_cat) #导入数据\n",
    "housing_cat_encoded[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 查看类别属性\n",
    "ordinal_encoder.categories_ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这种表示的一个问题是ML算法将假定两个离得近的值比两个离得远的值更相似。 显然情况并非如此（例如，类别0和类别4比类别0和类别1更相似）。\n",
    "\n",
    " 要解决此问题，常见的解决方案是为每个类别创建一个二进制属性：\n",
    "* 当类别为“<1H OCEAN”时，一个属性等于1（否则为0）\n",
    "* 当类别为“INLAND”时，另一个属性等于1（ 否则为0，等等。 \n",
    "\n",
    "这称为**one-hot encoding**,，因为只有一个属性等于1（**hot**），而其他属性将为0（**cold**）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "cat_encoder = OneHotEncoder()\n",
    "housing_cat_1hot = cat_encoder.fit_transform(housing_cat)\n",
    "housing_cat_1hot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "默认情况下，**OneHotEncoder**类返回一个**稀疏数组**，但如果需要，我们可以通过调用**toarray（）**方法将其转换为**密集数组**："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing_cat_1hot.toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "或者我们可以选择设置 `sparse=False`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_encoder = OneHotEncoder(sparse=False)\n",
    "housing_cat_1hot = cat_encoder.fit_transform(housing_cat)\n",
    "housing_cat_1hot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_encoder.categories_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**注意：**\n",
    "\n",
    "如果一个分类属性有较多的类别（如国家代码、职业、物种等），那么one-hot编码会导致大量的输入特征维度（高维）。这可能会减慢训练速度并降低性能。如果发生这种情况，您可能希望用与类别相关的有用的数字特征来替换分类输入：例如，您可以用到海洋的距离来替换海洋_近似特征（类似地，国家代码可以用国家的人口和人均GDP来替换）。或者，你可以用一个称为嵌入的可学习的低维向量来替换每个类别。每个类别的表征将在训练期间被学习：这就是表征学习。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 自定义转换器 Custom Transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "虽然Scikit-Learn提供了许多有用的transformer，但我们需要编写我们自己的transformer来执行诸如自定义清理操作或特定组合属性等任务。 \n",
    "\n",
    "我们希望自定义的transformers与Scikit-Learn的功能（例如pipline）无缝协作，并且由于Scikit-Learn依赖于鸭子类型的编译 duck typing（不是继承），所以你需要的只是创建一个类并实现三个函数\n",
    "\n",
    "我们需要创建一个类并执行三个方法：\n",
    "- fit()（返回自身）\n",
    "- transform()\n",
    "- fit_transform()\n",
    "\n",
    "\n",
    "如果添加TransfomerMixin作为基类，就可以直接得到最后一个方法。\n",
    "同时如果添加BaseEstimator作为基类（并在构造函数中避免* args和** kargs），我们还可以额外获取两个非常有用的自动化调整超参数的方法(`get_params()`和`set_params()`)\n",
    "\n",
    "例如，这是一个小型transformer类，它添加了我们前面讨论过的组合属性：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator, TransformerMixin\n",
    "\n",
    "# column index - 列索引\n",
    "rooms_ix, bedrooms_ix, population_ix, households_ix = 3, 4, 5, 6\n",
    "\n",
    "class CombinedAttributesAdder(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self, add_bedrooms_per_room = True): # no *args or **kargs\n",
    "        self.add_bedrooms_per_room = add_bedrooms_per_room\n",
    "    def fit(self, X, y=None):\n",
    "        return self  # nothing else to do\n",
    "    def transform(self, X, y=None):\n",
    "        rooms_per_household = X[:, rooms_ix] / X[:, households_ix]\n",
    "        population_per_household = X[:, population_ix] / X[:, households_ix]\n",
    "        if self.add_bedrooms_per_room:\n",
    "            bedrooms_per_room = X[:, bedrooms_ix] / X[:, rooms_ix]\n",
    "            return np.c_[X, rooms_per_household, population_per_household,\n",
    "                         bedrooms_per_room]\n",
    "        else:\n",
    "            return np.c_[X, rooms_per_household, population_per_household]\n",
    "\n",
    "attr_adder = CombinedAttributesAdder(add_bedrooms_per_room=False)\n",
    "housing_extra_attribs = attr_adder.transform(housing.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing_extra_attribs = pd.DataFrame(\n",
    "    housing_extra_attribs,\n",
    "    columns=list(housing.columns)+[\"rooms_per_household\", \"population_per_household\"],\n",
    "    index=housing.index)\n",
    "housing_extra_attribs.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在这个例子中，transformer有一个超参数*add_bedrooms_per_room*，默认情况下设置为True（提供合理的默认值通常很有帮助）。此超参数将允许您轻松找出添加此属性是否有助于机器学习算法。\n",
    "\n",
    "更一般地，我们可以为每个不能完全确保的数据准备步骤添加一个超参数。数据准备步骤越自动化，可以自动化的操作组合就越多，越容易发现更好用的组合（并能节省大量时间）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4 特征缩放 Feature Scaling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "预处理数据要做的最重要的转换之一是**特征缩放**。除了极个别情况，当输入的数值属性量度不同时，机器学习算法的性能都不会好。我们所处理的住房数据就是这样：总房间数分布范围是 6 到 39320，而收入中位数只分布在 0 到 15。通常情况下我们不需要对目标值进行缩放。\n",
    "\n",
    "通常情况下，我们不对目标值（target values）进行特征缩放。\n",
    "\n",
    "有两种常见的方法可以让所有的属性有相同的量度：\n",
    "\n",
    "- 最小-最大缩放(Min-Max scaling)\n",
    "- 标准化(standardization)\n",
    "\n",
    "最小-最大缩放（大多数人称其为归一化（normalization））：值被转变、重新缩放，直到范围变成 0 到 1。我们通过减去最小值，然后再除以最大值与最小值的差值，来进行归一化。Scikit-Learn 提供了一个转换器MinMaxScaler来实现这个功能。它有一个超参数feature_range，可以让你改变范围，如果我们希望范围不是 0 到 1。\n",
    "\n",
    "标准化与最小-最大缩放不同：首先减去平均值（所以标准化值的平均值总是 0），然后除以方差，使得到的分布具有单位方差。与归一化不同，标准化不会限定值到某个特定的范围，这对某些算法可能构成问题（比如，神经网络常需要输入值的范围是 0 到 1）。但是，标准化受到异常值的影响很小。\n",
    "\n",
    "例如，假设一个街区的收入中位数由于某种错误变成了100，归一化会将其它范围是 0 到 15 的值变为 0-0.15，但是标准化不会受什么影响。Scikit-Learn 提供了一个转换器StandardScaler来进行标准化\n",
    "\n",
    "注意：**特征缩放只在训练集上进行**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5 转换流水线 Transformation Pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从上面的一系列操作我们可以看出，许多数据存在转换步骤，需要按一定的顺序执行。幸运的是，Scikit-Learn 提供了类Pipeline，来进行这一系列的转换。下面是一个数值属性的流水线例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "num_pipeline = Pipeline([\n",
    "        ('imputer', SimpleImputer(strategy=\"median\")),\n",
    "        ('attribs_adder', CombinedAttributesAdder()),\n",
    "        ('std_scaler', StandardScaler()),\n",
    "    ])\n",
    "\n",
    "housing_num_tr = num_pipeline.fit_transform(housing_num)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing_num_tr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pipeline构造函数会通过一系列名称/估算器来配对来定义步骤的序列。除了最后一个是估算器之外，前面都必须是转换器（也就是必须有fit_transform()方法）,命名可以自己按照喜好定义。\n",
    "\n",
    "\n",
    "当你调用Pipeline的**fit（）方法**时，它在所有transformer上顺序调用**fit_transform（）**，将每个调用的输出作为参数传递给下一个调用，直到它到达最后一个estimator，在最后一个这里它只是调用**fit（）方法**。\n",
    "\n",
    "Pipeline显示了它与最终estimator相同的方法。 在这个例子中，最后一个estimator是一个**StandardScaler**，它是一个transformer，所以Pipeline有一个**transform（）方法**，它按顺序将所有变换应用于数据（它还有一个**fit_transform**方法，我们可以使用它而不是先调用**fit（）**然后调用**transform()**）\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "\n",
    "num_attribs = list(housing_num)\n",
    "cat_attribs = [\"ocean_proximity\"]\n",
    "\n",
    "full_pipeline = ColumnTransformer([\n",
    "        (\"num\", num_pipeline, num_attribs),\n",
    "        (\"cat\", OneHotEncoder(), cat_attribs),\n",
    "    ])\n",
    "\n",
    "housing_prepared = full_pipeline.fit_transform(housing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing_prepared"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing_prepared.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以上是基于Scikit-Learn 0.20做出的结果，早期版本使用基于DataFrameSelector转换器和FeatureUnion的解决方案将不同的转换应用于不同的列。作为参考，下面是基于DataFrameSelector transformer （仅选择Pandas DataFrame列的子集）和FeatureUnion的旧版解决方案："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator, TransformerMixin\n",
    "\n",
    "# Create a class to select numerical or categorical columns \n",
    "class OldDataFrameSelector(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self, attribute_names):\n",
    "        self.attribute_names = attribute_names\n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    def transform(self, X):\n",
    "        return X[self.attribute_names].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们现在有一个数值Pipeline，我们还需要在分类值上应用**LabelBi narizer**：如何将这些转换连接到单个pipeline？ \n",
    "\n",
    "Scikit-Learn为此提供了一个**FeatureUnion类**:\n",
    "\n",
    "1. 我们给它一个transformers列表（可以是整个transformer pipelines），\n",
    "2. 当它的transform（）方法被调用时，它并行运行每个transformer的**transform()** 方法，\n",
    "3. 等待它们的输出，然后连接它们并返回结果（当然，调用**fit（）方法**会调用所有transformer的**fit（）方法**）。 \n",
    "\n",
    "处理数字和分类属性的完整Pipeline可能如下所示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator, TransformerMixin\n",
    "\n",
    "#Create a class to select numerical or categorical columns \n",
    "class OldDataFrameSelector(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self, attribute_names):\n",
    "        self.attribute_names = attribute_names\n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    def transform(self, X):\n",
    "        return X[self.attribute_names].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import FeatureUnion\n",
    "\n",
    "num_attribs = list(housing_num)# 数字属性\n",
    "cat_attribs = [\"ocean_proximity\"]# 分类属性\n",
    "\n",
    "# 定义数字属性的 Pipeline\n",
    "old_num_pipeline = Pipeline([\n",
    "        ('selector', OldDataFrameSelector(num_attribs)),\n",
    "        ('imputer', SimpleImputer(strategy=\"median\")),\n",
    "        ('attribs_adder', CombinedAttributesAdder()),\n",
    "        ('std_scaler', StandardScaler()),\n",
    "    ])\n",
    "\n",
    "# 定义分类属性的Pipeline\n",
    "old_cat_pipeline = Pipeline([\n",
    "        ('selector', OldDataFrameSelector(cat_attribs)),\n",
    "        ('cat_encoder', OneHotEncoder(sparse=False)),\n",
    "    ])\n",
    "\n",
    "\n",
    "old_full_pipeline = FeatureUnion(transformer_list=[\n",
    "        (\"num_pipeline\", old_num_pipeline),\n",
    "        (\"cat_pipeline\", old_cat_pipeline),\n",
    "    ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "old_housing_prepared = old_full_pipeline.fit_transform(housing)\n",
    "old_housing_prepared"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 我们可以看到结果和ColumnTransformer一样\n",
    "np.allclose(housing_prepared, old_housing_prepared)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.选择和训练模型 Select and train a model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "终于到这一步了！你在前面限定了问题、获得了数据、探索了数据、采样了一个测试集、写了自动化的转换流水线来清理和为算法准备数据。现在，你已经准备好选择并训练一个机器学习模型了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 在训练集上训练和评估\n",
    "\n",
    "好消息是，由于所有这些先前的步骤，现在情况将会比你想象的要简单得多。 让我们首先训练一个**线性回归模型**："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "lin_reg = LinearRegression()\n",
    "lin_reg.fit(housing_prepared, housing_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "完毕！你现在就有了一个可用的线性回归模型。用一些训练集中的实例做下验证："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "some_data = housing.iloc[:5]\n",
    "some_labels = housing_labels.iloc[:5]\n",
    "some_data_prepared = full_pipeline.transform(some_data)\n",
    "\n",
    "print(\"Predictions:\", lin_reg.predict(some_data_prepared))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "与实际值比较："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(\"Labels:\", list(some_labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "some_data_prepared"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "虽然行的通，但是预测并不怎么准确（比如，第二个预测偏离了 50%！）。让我们使用 Scikit-Learn 的mean_squared_error函数，用全部训练集来计算下这个回归模型的 RMSE："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "housing_predictions = lin_reg.predict(housing_prepared)\n",
    "lin_mse = mean_squared_error(housing_labels, housing_predictions)\n",
    "lin_rmse = np.sqrt(lin_mse)\n",
    "lin_rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK，有总比没有强，但显然结果并不好：大多数街区的median_housing_values位于 120000 到 265000 美元之间，因此预测误差 68628 美元不能让人满意。这是一个模型欠拟合训练数据的例子。当这种情况发生时，意味着特征没有提供足够多的信息来做出一个好的预测，或者模型并不强大。\n",
    "\n",
    "就像前一章看到的，修复欠拟合的主要方法：\n",
    "- 选择一个更强大的模型，\n",
    "- 训练算法提供更好的特征，\n",
    "- 或去掉模型上的限制。\n",
    "\n",
    "这个模型还没有正则化，所以排除了最后一个选项。你可以尝试添加更多特征（比如，人口的对数值），但是首先让我们尝试一个更为复杂的模型，看看效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "tree_reg = DecisionTreeRegressor()\n",
    "tree_reg.fit(housing_prepared, housing_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing_predictions = tree_reg.predict(housing_prepared)\n",
    "tree_mse = mean_squared_error(housing_labels, housing_predictions)\n",
    "tree_rmse = np.sqrt(tree_mse)\n",
    "tree_rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "等一下，发生了什么？没有误差？这个模型可能是绝对完美的吗？当然，更大可能性是这个模型严重过拟合数据。如何确定呢？\n",
    "\n",
    "如前所述，直到你准备运行一个具备足够信心的模型，都不要碰测试集，因此你需要使用训练集的部分数据来做训练，用一部分来做模型验证。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 使用交叉验证进行评估 Better Evaluation Using Cross-Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "评估决策树模型的一种方法是使用train_test_split函数将训练集拆分为较小的训练集和验证集，然后针对较小的训练集训练模型并根据验证集对其进行评估。 这是一项工作，但没有什么太困难，它会运作得相当好。\n",
    "\n",
    "一个很好的选择是使用Scikit-Learn的交叉验证功能。 以下代码执行K-fold交叉验证：它将训练集随机分成10个不同的子集称为folds，然后它训练和评估决策树模型10次，每次选择不同的fold进行评估，在另外9个folds上进行训练。结果是一个包含10个评估分数的数组："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "scores = cross_val_score(tree_reg, housing_prepared, housing_labels,\n",
    "                         scoring=\"neg_mean_squared_error\", cv=10)\n",
    "\n",
    "tree_rmse_scores = np.sqrt(-scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 显示结果\n",
    "def display_scores(scores):\n",
    "    print(\"Scores:\", scores)\n",
    "    print(\"Mean:\", scores.mean())\n",
    "    print(\"Standard deviation:\", scores.std())\n",
    "\n",
    "display_scores(tree_rmse_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "决策树的结果看起来不像以前那么好。 事实上，它似乎是比线性回归模型更差！请注意，交叉验证不仅可以获得模型性能的估计值，还可以衡量此估算值的精确程度（即其标准偏差）。决策树的得分约为70666，标准偏差为±2928。 如果您只使用了一个验证集，则不会有此信息。 但交叉验证的代价是多次训练模型，因此并非总是可行。 我们可以为线性回归模型计算相同的指标，以进行确认："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "lin_scores = cross_val_score(lin_reg, housing_prepared, housing_labels,\n",
    "                             scoring=\"neg_mean_squared_error\", cv=10)\n",
    "lin_rmse_scores = np.sqrt(-lin_scores)\n",
    "display_scores(lin_rmse_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "判断没错：决策树模型过拟合很严重，它的性能比线性回归模型还差。\n",
    "\n",
    "现在再尝试最后一个模型：RandomForestRegressor。第7章我们会看到，随机森林是通过用特征的随机子集训练许多决策树。在其它多个模型之上建立模型称为集成学习（Ensemble Learning），它是推进 ML 算法的一种好方法。我们会跳过大部分的代码，因为代码本质上和其它模型一样："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "\n",
    "forest_reg = RandomForestRegressor(n_estimators=100, random_state=42)\n",
    "forest_reg.fit(housing_prepared, housing_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing_predictions = forest_reg.predict(housing_prepared)\n",
    "forest_mse = mean_squared_error(housing_labels, housing_predictions)\n",
    "forest_rmse = np.sqrt(forest_mse)\n",
    "forest_rmse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "forest_scores = cross_val_score(forest_reg, housing_prepared, housing_labels,\n",
    "                                scoring=\"neg_mean_squared_error\", cv=10)\n",
    "forest_rmse_scores = np.sqrt(-forest_scores)\n",
    "display_scores(forest_rmse_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "这次的结果要好得多：随机森林看起来很有希望。 但请注意，训练集上的得分仍然远低于验证集，这意味着该模型仍然**过拟合**训练集。 过拟合的可能解决方案是：\n",
    "* 简化模型，\n",
    "* 约束模型（即，正则化），\n",
    "* 或获得更多的训练数据。\n",
    "\n",
    "**然而**，在你深入了解随机森林之前，你应该尝试很多来自各种机器学习算法的其他模型（比如可能是具有不同内核的支持向量机，神经网络等），而不需要花太多时间调整超参数。 目标是将一些（两到五个）有希望的模型列入候选名单。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = cross_val_score(lin_reg, housing_prepared, housing_labels, scoring=\"neg_mean_squared_error\", cv=10)\n",
    "pd.Series(np.sqrt(-scores)).describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 支持向量机\n",
    "from sklearn.svm import SVR\n",
    "\n",
    "svm_reg = SVR(kernel=\"linear\")\n",
    "svm_reg.fit(housing_prepared, housing_labels)\n",
    "housing_predictions = svm_reg.predict(housing_prepared)\n",
    "svm_mse = mean_squared_error(housing_labels, housing_predictions)\n",
    "svm_rmse = np.sqrt(svm_mse)\n",
    "svm_rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.微调我们的模型 Fine-Tune Your Model\n",
    "假设你现在有了一个列表，列表里有几个有希望的模型。你现在需要对它们进行微调。让我们来看几种微调的方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1 网络搜索 Grid Search\n",
    "微调的一种方法是手工调整超参数，直到找到一个好的超参数组合。这么做的话会非常冗长，你也可能没有时间探索多种组合。\n",
    "\n",
    "你应该使用 Scikit-Learn 的GridSearchCV来做这项搜索工作。你所需要做的是告诉GridSearchCV要试验有哪些超参数，要试验什么值，GridSearchCV就能用交叉验证试验所有可能超参数值的组合。例如，下面的代码搜索了RandomForestRegressor超参数值的最佳组合："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "param_grid = [\n",
    "    # try 12 (3×4) combinations of hyperparameters\n",
    "    {'n_estimators': [3, 10, 30], 'max_features': [2, 4, 6, 8]},\n",
    "    # then try 6 (2×3) combinations with bootstrap set as False\n",
    "    {'bootstrap': [False], 'n_estimators': [3, 10], 'max_features': [2, 3, 4]},\n",
    "  ]\n",
    "\n",
    "forest_reg = RandomForestRegressor(random_state=42)\n",
    "# train across 5 folds, that's a total of (12+6)*5=90 rounds of training \n",
    "grid_search = GridSearchCV(forest_reg, param_grid, cv=5,\n",
    "                           scoring='neg_mean_squared_error',\n",
    "                           return_train_score=True)\n",
    "grid_search.fit(housing_prepared, housing_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当您不知道超参数应该具有什么值时，一种简单的方法是尝试**10的连续幂**（如果您想要更精细的搜索，则尝试更小的数字，如本例中使用n_estimators超参数所示"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个**param_grid**告诉Scikit-Learn\n",
    "* 首先评估第一个dict中指定的*n_estimators*和*max_features*超参数值的所有3×4= 12个组合（现在不要担心这些超参数意味着什么;在第7章中我们会解释他们）\n",
    "* 然后在第二个dict中尝试所有2×3 = 6个超参数值组合，但这次将*bootstrap*超参数设置为False而不是True（这是该超参数的默认值）。\n",
    "\n",
    "总而言之，\n",
    "* 网格搜索将探索RandomForestRegressor超参数值的18种组合，\n",
    "* 并将对每个模型进行五次训练（因为我们使用five-fold交叉验证）。\n",
    "\n",
    "换句话说，总共会有18×5 = 90轮的训练！ 这可能需要相当长的时间，但完成后你可以获得最佳的参数组合，如下所示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "让我们看看在网格搜索期间测试的每个超参数组合的得分："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cvres = grid_search.cv_results_\n",
    "for mean_score, params in zip(cvres[\"mean_test_score\"], cvres[\"params\"]):\n",
    "    print(np.sqrt(-mean_score), params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在此示例中，我们通过设置max_features超参数为6，n_estimators超参数为30获得最佳解决方案。 此组合的RMSE得分为49,959，略高于您之前使用默认超参数值（即52583.72407377466）得到的得分。 恭喜，您已经成功调整了最佳模型！\n",
    "\n",
    "不要忘记，您可以将一些数据准备步骤视为超参数。 例如，网格搜索将自动确定是否添加您不确定的功能（例如，使用CombinedAttributesAdder transformeradd_bedrooms_per_room超参数）。 它可以类似地用于自动找到处理异常值，缺失特征，特征选择等的最佳方法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(grid_search.cv_results_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2 随机搜索 Randomized Search\n",
    "随机搜索\n",
    "\n",
    "当探索相对较少的组合时，就像前面的例子，网格搜索还可以。但是当超参数的搜索空间很大时，最好使用RandomizedSearchCV。这个类的使用方法和类GridSearchCV很相似，但它不是尝试所有可能的组合，而是通过选择每个超参数的一个随机值的特定数量的随机组合。这个方法有两个优点：\n",
    "\n",
    "如果我们让随机搜索运行，比如 1000 次，它会探索每个超参数的 1000 个不同的值（而不是像网格搜索那样，只搜索每个超参数的几个值）。\n",
    "\n",
    "你可以方便地通过设定搜索次数，控制超参数搜索的计算量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from scipy.stats import randint\n",
    "\n",
    "param_distribs = {\n",
    "        'n_estimators': randint(low=1, high=200),\n",
    "        'max_features': randint(low=1, high=8),\n",
    "    }\n",
    "\n",
    "forest_reg = RandomForestRegressor(random_state=42)\n",
    "rnd_search = RandomizedSearchCV(forest_reg, param_distributions=param_distribs,\n",
    "                                n_iter=10, cv=5, scoring='neg_mean_squared_error', random_state=42)\n",
    "rnd_search.fit(housing_prepared, housing_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cvres = rnd_search.cv_results_\n",
    "for mean_score, params in zip(cvres[\"mean_test_score\"], cvres[\"params\"]):\n",
    "    print(np.sqrt(-mean_score), params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.3 组合模型 Ensemble Methods\n",
    "微调模型的另一种方法是尝试表现最好的模型组合（或“组合”）通常比最佳单个模型表现更好（就像随机森林比他们依赖的单个决策树表现更好），特别是如果各个模型产生非常不同类型的错误。我们将在第7章中更详细地介绍这个主题。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. 分析最佳模型及其错误 Analyze the Best Models and Their Errors\n",
    "通过检查最佳模型，您将经常获得关于该问题的良好见解。 例如，RandomForestRegressor可以指示每个属性的相对重要性，以进行准确的预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_importances = grid_search.best_estimator_.feature_importances_\n",
    "feature_importances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在相应的属性名称旁边显示这些重要性分数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "extra_attribs = [\"rooms_per_hhold\", \"pop_per_hhold\", \"bedrooms_per_room\"]\n",
    "#cat_encoder = cat_pipeline.named_steps[\"cat_encoder\"] # old solution\n",
    "cat_encoder = full_pipeline.named_transformers_[\"cat\"]\n",
    "cat_one_hot_attribs = list(cat_encoder.categories_[0])\n",
    "attributes = num_attribs + extra_attribs + cat_one_hot_attribs\n",
    "sorted(zip(feature_importances, attributes), reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有了这些信息，我们可以想尝试删除一些不太有用的特征（例如，显然只有一个ocean_proximity类别真的很有用，所以你可以试试放弃其他特征）。\n",
    "\n",
    "我们或许还应该查看模型产生的具体错误，然后尝试说明错误原因和可以解决问题的方法（增加额外的特征，或者相反，摆脱没有信息的，清理异常值等）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_model = grid_search.best_estimator_\n",
    "\n",
    "X_test = start_test_set.drop(\"median_house_value\", axis=1)\n",
    "y_test = start_test_set[\"median_house_value\"].copy()\n",
    "\n",
    "X_test_prepared = full_pipeline.transform(X_test)\n",
    "final_predictions = final_model.predict(X_test_prepared)\n",
    "\n",
    "final_mse = mean_squared_error(y_test, final_predictions)\n",
    "final_rmse = np.sqrt(final_mse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.1 在测试集上测试模型 Evaluate Your System on the Test Set\n",
    "调节完系统之后，你终于有了一个性能足够好的系统。现在就可以用测试集评估最后的模型了。这个过程没有什么特殊的：从测试集得到预测值和标签，运行full_pipeline转换数据（调用transform()，而不是fit_transform()！），再用测试集评估最终模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_model = grid_search.best_estimator_\n",
    "\n",
    "X_test = start_test_set.drop(\"median_house_value\", axis=1)\n",
    "y_test = start_test_set[\"median_house_value\"].copy()\n",
    "\n",
    "X_test_prepared = full_pipeline.transform(X_test)\n",
    "final_predictions = final_model.predict(X_test_prepared)\n",
    "\n",
    "final_mse = mean_squared_error(y_test, final_predictions)\n",
    "final_rmse = np.sqrt(final_mse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "评估结果通常要比交叉验证的效果差一点，如果你之前做过很多超参数微调（因为你的系统在验证集上微调，得到了不错的性能，通常不会在未知的数据集上有同样好的效果）。这个例子不属于这种情况，但是当发生这种情况时，你一定要忍住不要调节超参数，使测试集的效果变好；这样的提升不能推广到新数据上。\n",
    "\n",
    "然后就是项目的预上线阶段：你需要展示你的方案（重点说明学到了什么、做了什么、没做什么、做过什么假设、系统的限制是什么，等等），记录下所有事情，用漂亮的图表和容易记住的表达（比如，“收入中位数是房价最重要的预测量”）做一次精彩的展示。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. 启动、监控、维护系统 Launch, Monitor, and Maintain Your System"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 我们可以计算测试RMSE的95％置信区间：\n",
    "from scipy import stats\n",
    "\n",
    "confidence = 0.95\n",
    "squared_errors = (final_predictions - y_test) ** 2\n",
    "np.sqrt(stats.t.interval(confidence, len(squared_errors) - 1,\n",
    "                         loc=squared_errors.mean(),\n",
    "                         scale=stats.sem(squared_errors)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 我们可以手动计算间隔，如下所示：\n",
    "m = len(squared_errors)\n",
    "mean = squared_errors.mean()\n",
    "tscore = stats.t.ppf((1 + confidence) / 2, df=m - 1)\n",
    "tmargin = tscore * squared_errors.std(ddof=1) / np.sqrt(m)\n",
    "np.sqrt(mean - tmargin), np.sqrt(mean + tmargin)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "或者，我们可以使用z分数而不是t分数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "zscore = stats.norm.ppf((1 + confidence) / 2)\n",
    "zmargin = zscore * squared_errors.std(ddof=1) / np.sqrt(m)\n",
    "np.sqrt(mean - zmargin), np.sqrt(mean + zmargin)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 额外的资料"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 完整的管道，包括准备和预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "full_pipeline_with_predictor = Pipeline([\n",
    "        (\"preparation\", full_pipeline),\n",
    "        (\"linear\", LinearRegression())\n",
    "    ])\n",
    "\n",
    "full_pipeline_with_predictor.fit(housing, housing_labels)\n",
    "full_pipeline_with_predictor.predict(some_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model persistence using joblib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_model = full_pipeline_with_predictor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import joblib\n",
    "joblib.dump(my_model, \"my_model.pkl\") # DIFF\n",
    "#...\n",
    "my_model_loaded = joblib.load(\"my_model.pkl\") # DIFF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example SciPy distributions for RandomizedSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import geom, expon\n",
    "geom_distrib=geom(0.5).rvs(10000, random_state=42)\n",
    "expon_distrib=expon(scale=1).rvs(10000, random_state=42)\n",
    "plt.hist(geom_distrib, bins=50)\n",
    "plt.show()\n",
    "plt.hist(expon_distrib, bins=50)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 练习留给大家去思考，这里不给出参考答案，具体可以参考\n",
    "https://github.com/ageron/handson-ml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "file_extension": ".py",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6-final"
  },
  "mimetype": "text/x-python",
  "name": "python",
  "npconvert_exporter": "python",
  "pygments_lexer": "ipython3",
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "version": 3
 },
 "nbformat": 4,
 "nbformat_minor": 2
}