codes/MMSelfSup_tutorials/【5】自监督预训练模型的评估：“检测”下游任务.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/mmselfsup_logo.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 自监督预训练模型的评估：“检测” 下游任务\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/open-mmlab/OpenMMLabCourse/blob/main/codes/MMSelfSup_tutorials/【5】自监督预训练模型的评估：“检测”下游任务.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**MMSelfSup Repo**：[https://github.com/open-mmlab/mmselfsup](https://github.com/open-mmlab/mmselfsup)\n",
    "\n",
    "**MMSelfSup 官方文档链接**：[https://mmselfsup.readthedocs.io/en/latest](https://mmselfsup.readthedocs.io/en/latest)\n",
    "\n",
    "**MMSelfSup 视频教学**：[https://space.bilibili.com/1293512903/channel/collectiondetail?sid=657287](https://space.bilibili.com/1293512903/channel/collectiondetail?sid=657287)\n",
    "\n",
    "**MMSelfSup 代码库介绍 PPT 获取方式**：关注 OpenMMLab 公众号，后台回复：mmselfsup，即可获取课程 PPT\n",
    "\n",
    "**加入微信社群方式**：关注公众号，选择 “加入我们” -> “微信社区”，即可获取入群二维码。非常期待你的到来呀~\n",
    "\n",
    "**作者**：OpenMMLab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. 任务介绍"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在使用 “检测” 下游任务对自监督预训练模型进行评估中，MMSelfSup 目前支持了以下几种评估配置：（详细请参考官方[文档](https://mmselfsup.readthedocs.io/zh_CN/latest/tutorials/6_benchmarks.html#id5)）\n",
    "+ 将自监督预训练的权值迁移到 `Faster R-CNN` 模型上，并在 `Pascal VOC 2007 + 2012` 数据集上进行评估\n",
    "+ 将自监督预训练的权值迁移到 `Mask R-CNN` 模型上，并在 `COCO2017` 数据集上进行评估\n",
    "\n",
    "本教程将演示第一种评估配置。\n",
    "\n",
    "**注意：下游任务的 backbone 必须和自监督预训练模型的 backbone 保持一致，才能进行自监督预训练权值的迁移！**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "“检测” 下游任务评估的 **`基本流程`** 如下：\n",
    "1. 将自监督预训练好的 backbone 权值迁移到下游 “检测” 任务的 backbone 上\n",
    "2. 使用下游任务（检测）的数据集对下游任务的模型进行微调，以此验证自监督预训练模型提取图片特征的效果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 环境配置"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 查看 Python、PyTorch 和 Torchvision 的版本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check nvcc version\n",
    "!nvcc -V"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check GCC version\n",
    "!gcc --version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check PyTorch installation\n",
    "import torch, torchvision\n",
    "print(torch.__version__)\n",
    "print(torch.cuda.is_available())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 安装 MMSelfSup 的依赖库：MMCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install openmim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mim install mmcv-full"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3  安装 MMSelfSup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/open-mmlab/mmselfsup.git\n",
    "%cd /content/mmselfsup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install MMSelfSup from source\n",
    "!pip install -e . "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 安装 MMDetection \n",
    "在这里，mmdet 是我们这里所需要用到的依赖库，所以直接用 `pip install mmdet` 命令安装即可。\n",
    "\n",
    "其他方式的安装详情请参考： [MMDetection 文档](https://github.com/open-mmlab/mmdetection/blob/master/docs/en/get_started.md)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install mmdet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5 检查安装是否正确"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mmselfsup\n",
    "print(mmselfsup.__version__)\n",
    "\n",
    "import mmdet\n",
    "print(mmdet.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 准备数据集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.0 数据集介绍"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本教程将在 `Pascal VOC 2012` 和 `Pascal VOC 2007` 数据集上对自监督预训练好的模型进行“检测”下游任务的评估。\n",
    "\n",
    "Pascal VOC 2007 数据集包含 20 个类别，Pascal VOC 2012 在此基础上进行无交集扩展（即类别相同，图片不同）。\n",
    "\n",
    "在本教程中，我们根据 MoCo 论文里的数据集配置进行训练和测试：使用 VOC 07 和 VOC 12 的训练集和验证集进行微调，在 VOC 07 数据集上进行测试。\n",
    "\n",
    "数据集官方网址：http://host.robots.ox.ac.uk/pascal/VOC/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 下载数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content/mmselfsup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir data\n",
    "%cd data\n",
    "\n",
    "!wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar\n",
    "\n",
    "!wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar\n",
    "!wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 解压数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xf VOCtrainval_11-May-2012.tar\n",
    "!tar -xf VOCtrainval_06-Nov-2007.tar\n",
    "!tar -xf VOCtest_06-Nov-2007.tar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 准备自监督预训练模型的 backbone 权值文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 针对自监督预训练过程中保存的 checkpoint 文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们使用第一个教程 `模型自监督预训练 之 SimCLR` 中训练保存下来的 `epoch_1.pth` 文件进行演示，该文件可以从 [这里](https://download.openmmlab.com/mmselfsup/tutorial/epoch_1.pth) 下载，存放在文件夹 `mmselfsup/work_dirs/selfsup/simclr/simclr_resnet50_1xb32-coslr-1e_tinyin200` 里。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content/mmselfsup\n",
    "!mkdir -p work_dirs/selfsup/simclr/simclr_resnet50_1xb32-coslr-1e_tinyin200\n",
    "!wget https://download.openmmlab.com/mmselfsup/tutorial/epoch_1.pth  -P  work_dirs/selfsup/simclr/simclr_resnet50_1xb32-coslr-1e_tinyin200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以使用命令 `tools/model_converters/extract_backbone_weights.py` 来提取自监督预训练模型的 backbone 权值，代码如下：\n",
    "\n",
    "```python\n",
    "python tools/model_converters/extract_backbone_weights.py {CHECKPOINT} {MODEL_FILE}\n",
    "```\n",
    "\n",
    "参数:\n",
    "- CHECKPOINT：自监督预训练过程中保存下来（名为 `epoch_*.pth`）的模型文件路径\n",
    "- MODEL_FILE：输出 backbone 权重文件的保存路径。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content/mmselfsup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python ./tools/model_converters/extract_backbone_weights.py \\\n",
    "work_dirs/selfsup/simclr/simclr_resnet50_1xb32-coslr-1e_tinyin200/epoch_1.pth \\\n",
    "work_dirs/selfsup/simclr/simclr_resnet50_1xb32-coslr-1e_tinyin200/backbone.pth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 针对 MMSelfSup 模型库里的模型文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**注意：MMSelfSup 的 [模型库](https://github.com/open-mmlab/mmselfsup/blob/master/docs/en/model_zoo.md) 中的模型文件都已经提取过 backbone 权值，不需要再次提取！**我们直接使用即可。\n",
    "\n",
    "在模型库中找到在 SimCLR 的预训练模型文件 `simclr_resnet50_8xb32-coslr-200e_in1k`，下载放在 `checkpoints` 文件夹里 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content/mmselfsup\n",
    "!mkdir checkpoints\n",
    "!wget https://download.openmmlab.com/mmselfsup/simclr/simclr_resnet50_8xb32-coslr-200e_in1k_20220428-46ef6bb9.pth -P checkpoints"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 自监督预训练模型的 “检测” 下游任务评估"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 写自监督预训练模型的 “检测” 评估配置文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 新建一个名为 `faster_rcnn_r50_c4_mstrain_3k_voc0712.py` 的配置文件。（配置文件命名要求 & 含义可参考[这里](https://mmsegmentation.readthedocs.io/zh_CN/latest/tutorials/config.html#id3))\n",
    "\n",
    "\n",
    "\n",
    "2. `faster_rcnn_r50_c4_mstrain_3k_voc0712.py` 训练配置文件的内容：\n",
    "    1. 继承 [faster_rcnn_r50_c4_mstrain_24k_voc0712.py](https://github.com/open-mmlab/mmselfsup/blob/master/configs/benchmarks/mmdetection/voc0712/faster_rcnn_r50_c4_mstrain_24k_voc0712.py) 配置文件\n",
    "    2. 根据 batch size 调整学习率（调整原则请参考：[这里](https://mmselfsup.readthedocs.io/zh_CN/latest/get_started.html#id2)）\n",
    "    3. 根据需求修改参数：模型训练多少个 iteration 评估一次、模型训练多少个 iteration 保存一次 checkpoint 文件 以及 总共训练多少个 iteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile /content/mmselfsup/configs/benchmarks/mmdetection/voc0712/faster_rcnn_r50_c4_mstrain_3k_voc0712.py\n",
    "_base_ = 'faster_rcnn_r50_c4_mstrain_24k_voc0712.py'\n",
    "\n",
    "optimizer = dict(\n",
    "    lr=0.02 * (1 / 8)\n",
    ")\n",
    "\n",
    "evaluation = dict(interval=1000, metric='mAP')\n",
    "\n",
    "checkpoint_config = dict(by_epoch=False, interval=1000)\n",
    "\n",
    "runner = dict(type='IterBasedRunner', max_iters=3000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 开始 “检测” 下游任务的评估"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在运行 “检测” 下游任务评估时，我们使用 [mim_dist_train_c4.sh](https://github.com/open-mmlab/mmselfsup/blob/master/tools/benchmarks/mmdetection/mim_dist_train_c4.sh) 脚本来启动训练。\n",
    "\n",
    "```shell\n",
    "bash tools/benchmarks/mmdetection/mim_dist_train_c4.sh ${CONFIG} ${PRETRAIN} ${GPUS}\n",
    "```\n",
    "\n",
    "参数：\n",
    "- CONFIG：“检测” 评估所使用的配置文件，位于 configs/benchmarks/mmdetection/ 里对应的数据集目录下\n",
    "- PRETRAIN: 自监督预训练模型的 backbone 权重文件所在的路径\n",
    "- GPUS: 所使用 GPU 的数量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /content/mmselfsup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!bash tools/benchmarks/mmdetection/mim_dist_train_c4.sh \\\n",
    "configs/benchmarks/mmdetection/voc0712/faster_rcnn_r50_c4_mstrain_3k_voc0712.py \\\n",
    "checkpoints/simclr_resnet50_8xb32-coslr-200e_in1k_20220428-46ef6bb9.pth \\\n",
    "1"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}