01-tensor_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# What is PyTorch?\n",
    "\n",
    "It’s a Python based scientific computing package targeted at two sets of audiences:\n",
    "\n",
    "-  Tensorial library that uses the power of GPUs\n",
    "-  A deep learning research platform that provides maximum flexibility and speed\n",
    "\n",
    "## Import the library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch  # <Ctrl> / <Shift> + <Return>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting help in Jupyter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-8-1e5cee51ddae>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-8-1e5cee51ddae>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    torch.*Tensor? # <Tab>\u001b[0m\n\u001b[0m          ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "torch.*Tensor? # <Tab>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.BFloat16Tensor\n",
       "torch.BoolTensor\n",
       "torch.ByteTensor\n",
       "torch.CharTensor\n",
       "torch.DoubleTensor\n",
       "torch.FloatTensor\n",
       "torch.HalfTensor\n",
       "torch.IntTensor\n",
       "torch.LongTensor\n",
       "torch.ShortTensor\n",
       "torch.Tensor"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# What about all `*Tensor`s?\n",
    "# Press <esc> to get out of help\n",
    "torch.*Tensor?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Module()"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.nn.Module()  # <Shift>+<Tab>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mInit signature:\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mModule\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m     \n",
       "Base class for all neural network modules.\n",
       "\n",
       "Your models should also subclass this class.\n",
       "\n",
       "Modules can also contain other Modules, allowing to nest them in\n",
       "a tree structure. You can assign the submodules as regular attributes::\n",
       "\n",
       "    import torch.nn as nn\n",
       "    import torch.nn.functional as F\n",
       "\n",
       "    class Model(nn.Module):\n",
       "        def __init__(self):\n",
       "            super(Model, self).__init__()\n",
       "            self.conv1 = nn.Conv2d(1, 20, 5)\n",
       "            self.conv2 = nn.Conv2d(20, 20, 5)\n",
       "\n",
       "        def forward(self, x):\n",
       "            x = F.relu(self.conv1(x))\n",
       "            return F.relu(self.conv2(x))\n",
       "\n",
       "Submodules assigned in this way will be registered, and will have their\n",
       "parameters converted too when you call :meth:`to`, etc.\n",
       "\n",
       ":ivar training: Boolean represents whether this module is in training or\n",
       "                evaluation mode.\n",
       ":vartype training: bool\n",
       "\u001b[0;31mInit docstring:\u001b[0m Initializes internal Module state, shared by both nn.Module and ScriptModule.\n",
       "\u001b[0;31mFile:\u001b[0m           ~/.local/lib/python3.7/site-packages/torch/nn/modules/module.py\n",
       "\u001b[0;31mType:\u001b[0m           type\n",
       "\u001b[0;31mSubclasses:\u001b[0m     Identity, Linear, Bilinear, _ConvNd, Threshold, ReLU, RReLU, Hardtanh, Sigmoid, Hardsigmoid, ...\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Annotate your functions / classes!\n",
    "torch.nn.Module?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mInit signature:\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mModule\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mSource:\u001b[0m        \n",
       "\u001b[0;32mclass\u001b[0m \u001b[0mModule\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34mr\"\"\"Base class for all neural network modules.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m    Your models should also subclass this class.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m    Modules can also contain other Modules, allowing to nest them in\u001b[0m\n",
       "\u001b[0;34m    a tree structure. You can assign the submodules as regular attributes::\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        import torch.nn as nn\u001b[0m\n",
       "\u001b[0;34m        import torch.nn.functional as F\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        class Model(nn.Module):\u001b[0m\n",
       "\u001b[0;34m            def __init__(self):\u001b[0m\n",
       "\u001b[0;34m                super(Model, self).__init__()\u001b[0m\n",
       "\u001b[0;34m                self.conv1 = nn.Conv2d(1, 20, 5)\u001b[0m\n",
       "\u001b[0;34m                self.conv2 = nn.Conv2d(20, 20, 5)\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            def forward(self, x):\u001b[0m\n",
       "\u001b[0;34m                x = F.relu(self.conv1(x))\u001b[0m\n",
       "\u001b[0;34m                return F.relu(self.conv2(x))\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m    Submodules assigned in this way will be registered, and will have their\u001b[0m\n",
       "\u001b[0;34m    parameters converted too when you call :meth:`to`, etc.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m    :ivar training: Boolean represents whether this module is in training or\u001b[0m\n",
       "\u001b[0;34m                    evaluation mode.\u001b[0m\n",
       "\u001b[0;34m    :vartype training: bool\u001b[0m\n",
       "\u001b[0;34m    \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mdump_patches\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34mr\"\"\"This allows better BC support for :meth:`load_state_dict`. In\u001b[0m\n",
       "\u001b[0;34m    :meth:`state_dict`, the version number will be saved as in the attribute\u001b[0m\n",
       "\u001b[0;34m    `_metadata` of the returned state dict, and thus pickled. `_metadata` is a\u001b[0m\n",
       "\u001b[0;34m    dictionary with keys that follow the naming convention of state dict. See\u001b[0m\n",
       "\u001b[0;34m    ``_load_from_state_dict`` on how to use this information in loading.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m    If new parameters/buffers are added/removed from a module, this number shall\u001b[0m\n",
       "\u001b[0;34m    be bumped, and the module's `_load_from_state_dict` method can compare the\u001b[0m\n",
       "\u001b[0;34m    version number and do appropriate changes if the state dict is from before\u001b[0m\n",
       "\u001b[0;34m    the change.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0m_version\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mint\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mtraining\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0m_is_full_backward_hook\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mbool\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m__init__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m\"\"\"\u001b[0m\n",
       "\u001b[0;34m        Initializes internal Module state, shared by both nn.Module and ScriptModule.\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_C\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_log_api_usage_once\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"python.nn_module\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtraining\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mOrderedDict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffers\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mOrderedDict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_non_persistent_buffers_set\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mOrderedDict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_is_full_backward_hook\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mOrderedDict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_pre_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mOrderedDict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_state_dict_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mOrderedDict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_load_state_dict_pre_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mOrderedDict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mOrderedDict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mforward\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mCallable\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m...\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_forward_unimplemented\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mregister_buffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtensor\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpersistent\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Adds a buffer to the module.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        This is typically used to register a buffer that should not to be\u001b[0m\n",
       "\u001b[0;34m        considered a model parameter. For example, BatchNorm's ``running_mean``\u001b[0m\n",
       "\u001b[0;34m        is not a parameter, but is part of the module's state. Buffers, by\u001b[0m\n",
       "\u001b[0;34m        default, are persistent and will be saved alongside parameters. This\u001b[0m\n",
       "\u001b[0;34m        behavior can be changed by setting :attr:`persistent` to ``False``. The\u001b[0m\n",
       "\u001b[0;34m        only difference between a persistent buffer and a non-persistent buffer\u001b[0m\n",
       "\u001b[0;34m        is that the latter will not be a part of this module's\u001b[0m\n",
       "\u001b[0;34m        :attr:`state_dict`.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Buffers can be accessed as attributes using given names.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Args:\u001b[0m\n",
       "\u001b[0;34m            name (string): name of the buffer. The buffer can be accessed\u001b[0m\n",
       "\u001b[0;34m                from this module using the given name\u001b[0m\n",
       "\u001b[0;34m            tensor (Tensor): buffer to be registered.\u001b[0m\n",
       "\u001b[0;34m            persistent (bool): whether the buffer is part of this module's\u001b[0m\n",
       "\u001b[0;34m                :attr:`state_dict`.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Example::\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            >>> self.register_buffer('running_mean', torch.zeros(num_features))\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mpersistent\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mFalse\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjit\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mScriptModule\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"ScriptModule does not support non-persistent buffers\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;34m'_buffers'\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;34m\"cannot assign buffer before Module.__init__() call\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_six\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstring_classes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"buffer name should be a string. \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                            \u001b[0;34m\"Got {}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtypename\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0;34m'.'\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"buffer name can't contain \\\".\\\"\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"buffer name can't be empty string \\\"\\\"\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffers\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"attribute '{}' already exists\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0mtensor\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtensor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"cannot assign '{}' object to buffer '{}' \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                            \u001b[0;34m\"(torch Tensor or None required)\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                            \u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtypename\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffers\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtensor\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mpersistent\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_non_persistent_buffers_set\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdiscard\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_non_persistent_buffers_set\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mregister_parameter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparam\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mParameter\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Adds a parameter to the module.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        The parameter can be accessed as an attribute using given name.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Args:\u001b[0m\n",
       "\u001b[0;34m            name (string): name of the parameter. The parameter can be accessed\u001b[0m\n",
       "\u001b[0;34m                from this module using the given name\u001b[0m\n",
       "\u001b[0;34m            param (Parameter): parameter to be added to the module.\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;34m'_parameters'\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;34m\"cannot assign parameter before Module.__init__() call\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_six\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstring_classes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"parameter name should be a string. \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                            \u001b[0;34m\"Got {}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtypename\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0;34m'.'\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"parameter name can't contain \\\".\\\"\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"parameter name can't be empty string \\\"\\\"\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"attribute '{}' already exists\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mparam\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparam\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mParameter\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"cannot assign '{}' object to parameter '{}' \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                            \u001b[0;34m\"(torch.nn.Parameter or None required)\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                            \u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtypename\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparam\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad_fn\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;34m\"Cannot assign non-leaf Tensor to parameter '{0}'. Model \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;34m\"parameters must be created explicitly. To express '{0}' \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;34m\"as a function of another Tensor, compute the value in \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;34m\"the forward() method.\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparam\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0madd_module\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Module'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Adds a child module to the current module.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        The module can be accessed as an attribute using the given name.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Args:\u001b[0m\n",
       "\u001b[0;34m            name (string): name of the child module. The child module can be\u001b[0m\n",
       "\u001b[0;34m                accessed from this module using the given name\u001b[0m\n",
       "\u001b[0;34m            module (Module): child module to be added to the module.\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodule\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mModule\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{} is not a Module subclass\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtypename\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodule\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_six\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstring_classes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"module name should be a string. Got {}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtypename\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"attribute '{}' already exists\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0;34m'.'\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"module name can't contain \\\".\\\", got: {}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"module name can't be empty string \\\"\\\"\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchildren\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mdef\u001b[0m \u001b[0mcompute_should_use_set_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtensor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtensor_applied\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_has_compatible_shallow_copy_type\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtensor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtensor_applied\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;31m# If the new tensor has compatible tensor type as the existing tensor,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;31m# the current behavior is to change the tensor in-place using `.data =`,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;31m# and the future behavior is to overwrite the existing tensor. However,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;31m# changing the current behavior is a BC-breaking change, and we want it\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;31m# to happen in future releases. So for now we introduce the\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;31m# `torch.__future__.get_overwrite_module_params_on_conversion()`\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;31m# global flag to let the user control whether they want the future\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;31m# behavior of overwriting the existing tensor or not.\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mreturn\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__future__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_overwrite_module_params_on_conversion\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mreturn\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparam\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mparam\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;31m# Tensors stored in modules are graph leaves, and we don't want to\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;31m# track autograd history of `param_applied`, so we have to use\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;31m# `with torch.no_grad():`\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mno_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mparam_applied\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparam\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mshould_use_set_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompute_should_use_set_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparam\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparam_applied\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0mshould_use_set_data\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparam_applied\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32massert\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparam\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mParameter\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32massert\u001b[0m \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_leaf\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mParameter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparam_applied\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequires_grad\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mno_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mgrad_applied\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mshould_use_set_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompute_should_use_set_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad_applied\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mif\u001b[0m \u001b[0mshould_use_set_data\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrad_applied\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0;32massert\u001b[0m \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_leaf\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrad_applied\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequires_grad_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequires_grad\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuf\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mbuf\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffers\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbuf\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mapply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mCallable\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Module'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Applies ``fn`` recursively to every submodule (as returned by ``.children()``)\u001b[0m\n",
       "\u001b[0;34m        as well as self. Typical use includes initializing the parameters of a model\u001b[0m\n",
       "\u001b[0;34m        (see also :ref:`nn-init-doc`).\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Args:\u001b[0m\n",
       "\u001b[0;34m            fn (:class:`Module` -> None): function to be applied to each submodule\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            Module: self\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Example::\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            >>> @torch.no_grad()\u001b[0m\n",
       "\u001b[0;34m            >>> def init_weights(m):\u001b[0m\n",
       "\u001b[0;34m            >>>     print(m)\u001b[0m\n",
       "\u001b[0;34m            >>>     if type(m) == nn.Linear:\u001b[0m\n",
       "\u001b[0;34m            >>>         m.weight.fill_(1.0)\u001b[0m\n",
       "\u001b[0;34m            >>>         print(m.weight)\u001b[0m\n",
       "\u001b[0;34m            >>> net = nn.Sequential(nn.Linear(2, 2), nn.Linear(2, 2))\u001b[0m\n",
       "\u001b[0;34m            >>> net.apply(init_weights)\u001b[0m\n",
       "\u001b[0;34m            Linear(in_features=2, out_features=2, bias=True)\u001b[0m\n",
       "\u001b[0;34m            Parameter containing:\u001b[0m\n",
       "\u001b[0;34m            tensor([[ 1.,  1.],\u001b[0m\n",
       "\u001b[0;34m                    [ 1.,  1.]])\u001b[0m\n",
       "\u001b[0;34m            Linear(in_features=2, out_features=2, bias=True)\u001b[0m\n",
       "\u001b[0;34m            Parameter containing:\u001b[0m\n",
       "\u001b[0;34m            tensor([[ 1.,  1.],\u001b[0m\n",
       "\u001b[0;34m                    [ 1.,  1.]])\u001b[0m\n",
       "\u001b[0;34m            Sequential(\u001b[0m\n",
       "\u001b[0;34m              (0): Linear(in_features=2, out_features=2, bias=True)\u001b[0m\n",
       "\u001b[0;34m              (1): Linear(in_features=2, out_features=2, bias=True)\u001b[0m\n",
       "\u001b[0;34m            )\u001b[0m\n",
       "\u001b[0;34m            Sequential(\u001b[0m\n",
       "\u001b[0;34m              (0): Linear(in_features=2, out_features=2, bias=True)\u001b[0m\n",
       "\u001b[0;34m              (1): Linear(in_features=2, out_features=2, bias=True)\u001b[0m\n",
       "\u001b[0;34m            )\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchildren\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mapply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mcuda\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdevice\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdevice\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Moves all model parameters and buffers to the GPU.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        This also makes associated parameters and buffers different objects. So\u001b[0m\n",
       "\u001b[0;34m        it should be called before constructing optimizer if the module will\u001b[0m\n",
       "\u001b[0;34m        live on GPU while being optimized.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Args:\u001b[0m\n",
       "\u001b[0;34m            device (int, optional): if specified, all parameters will be\u001b[0m\n",
       "\u001b[0;34m                copied to that device\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            Module: self\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mxpu\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdevice\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdevice\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Moves all model parameters and buffers to the XPU.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        This also makes associated parameters and buffers different objects. So\u001b[0m\n",
       "\u001b[0;34m        it should be called before constructing optimizer if the module will\u001b[0m\n",
       "\u001b[0;34m        live on XPU while being optimized.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Arguments:\u001b[0m\n",
       "\u001b[0;34m            device (int, optional): if specified, all parameters will be\u001b[0m\n",
       "\u001b[0;34m                copied to that device\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            Module: self\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mxpu\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mcpu\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Moves all model parameters and buffers to the CPU.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            Module: self\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcpu\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdst_type\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Casts all parameters and buffers to :attr:`dst_type`.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Args:\u001b[0m\n",
       "\u001b[0;34m            dst_type (type or string): the desired type\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            Module: self\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdst_type\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mfloat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Casts all floating point parameters and buffers to float datatype.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            Module: self\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfloat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_floating_point\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mdouble\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Casts all floating point parameters and buffers to ``double`` datatype.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            Module: self\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdouble\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_floating_point\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mhalf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Casts all floating point parameters and buffers to ``half`` datatype.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            Module: self\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhalf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_floating_point\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mbfloat16\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Casts all floating point parameters and buffers to ``bfloat16`` datatype.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            Module: self\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbfloat16\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_floating_point\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m@\u001b[0m\u001b[0moverload\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdevice\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdevice\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m...\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m...\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m           \u001b[0mnon_blocking\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m...\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m...\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m@\u001b[0m\u001b[0moverload\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_blocking\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m...\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m...\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m@\u001b[0m\u001b[0moverload\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtensor\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_blocking\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m...\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m...\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Moves and/or casts the parameters and buffers.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        This can be called as\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        .. function:: to(device=None, dtype=None, non_blocking=False)\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        .. function:: to(dtype, non_blocking=False)\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        .. function:: to(tensor, non_blocking=False)\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        .. function:: to(memory_format=torch.channels_last)\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Its signature is similar to :meth:`torch.Tensor.to`, but only accepts\u001b[0m\n",
       "\u001b[0;34m        floating point or complex :attr:`dtype`s. In addition, this method will\u001b[0m\n",
       "\u001b[0;34m        only cast the floating point or complex parameters and buffers to :attr:`dtype`\u001b[0m\n",
       "\u001b[0;34m        (if given). The integral parameters and buffers will be moved\u001b[0m\n",
       "\u001b[0;34m        :attr:`device`, if that is given, but with dtypes unchanged. When\u001b[0m\n",
       "\u001b[0;34m        :attr:`non_blocking` is set, it tries to convert/move asynchronously\u001b[0m\n",
       "\u001b[0;34m        with respect to the host if possible, e.g., moving CPU Tensors with\u001b[0m\n",
       "\u001b[0;34m        pinned memory to CUDA devices.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        See below for examples.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        .. note::\u001b[0m\n",
       "\u001b[0;34m            This method modifies the module in-place.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Args:\u001b[0m\n",
       "\u001b[0;34m            device (:class:`torch.device`): the desired device of the parameters\u001b[0m\n",
       "\u001b[0;34m                and buffers in this module\u001b[0m\n",
       "\u001b[0;34m            dtype (:class:`torch.dtype`): the desired floating point or complex dtype of\u001b[0m\n",
       "\u001b[0;34m                the parameters and buffers in this module\u001b[0m\n",
       "\u001b[0;34m            tensor (torch.Tensor): Tensor whose dtype and device are the desired\u001b[0m\n",
       "\u001b[0;34m                dtype and device for all parameters and buffers in this module\u001b[0m\n",
       "\u001b[0;34m            memory_format (:class:`torch.memory_format`): the desired memory\u001b[0m\n",
       "\u001b[0;34m                format for 4D parameters and buffers in this module (keyword\u001b[0m\n",
       "\u001b[0;34m                only argument)\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            Module: self\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Examples::\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            >>> linear = nn.Linear(2, 2)\u001b[0m\n",
       "\u001b[0;34m            >>> linear.weight\u001b[0m\n",
       "\u001b[0;34m            Parameter containing:\u001b[0m\n",
       "\u001b[0;34m            tensor([[ 0.1913, -0.3420],\u001b[0m\n",
       "\u001b[0;34m                    [-0.5113, -0.2325]])\u001b[0m\n",
       "\u001b[0;34m            >>> linear.to(torch.double)\u001b[0m\n",
       "\u001b[0;34m            Linear(in_features=2, out_features=2, bias=True)\u001b[0m\n",
       "\u001b[0;34m            >>> linear.weight\u001b[0m\n",
       "\u001b[0;34m            Parameter containing:\u001b[0m\n",
       "\u001b[0;34m            tensor([[ 0.1913, -0.3420],\u001b[0m\n",
       "\u001b[0;34m                    [-0.5113, -0.2325]], dtype=torch.float64)\u001b[0m\n",
       "\u001b[0;34m            >>> gpu1 = torch.device(\"cuda:1\")\u001b[0m\n",
       "\u001b[0;34m            >>> linear.to(gpu1, dtype=torch.half, non_blocking=True)\u001b[0m\n",
       "\u001b[0;34m            Linear(in_features=2, out_features=2, bias=True)\u001b[0m\n",
       "\u001b[0;34m            >>> linear.weight\u001b[0m\n",
       "\u001b[0;34m            Parameter containing:\u001b[0m\n",
       "\u001b[0;34m            tensor([[ 0.1914, -0.3420],\u001b[0m\n",
       "\u001b[0;34m                    [-0.5112, -0.2324]], dtype=torch.float16, device='cuda:1')\u001b[0m\n",
       "\u001b[0;34m            >>> cpu = torch.device(\"cpu\")\u001b[0m\n",
       "\u001b[0;34m            >>> linear.to(cpu)\u001b[0m\n",
       "\u001b[0;34m            Linear(in_features=2, out_features=2, bias=True)\u001b[0m\n",
       "\u001b[0;34m            >>> linear.weight\u001b[0m\n",
       "\u001b[0;34m            Parameter containing:\u001b[0m\n",
       "\u001b[0;34m            tensor([[ 0.1914, -0.3420],\u001b[0m\n",
       "\u001b[0;34m                    [-0.5112, -0.2324]], dtype=torch.float16)\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            >>> linear = nn.Linear(2, 2, bias=None).to(torch.cdouble)\u001b[0m\n",
       "\u001b[0;34m            >>> linear.weight\u001b[0m\n",
       "\u001b[0;34m            Parameter containing:\u001b[0m\n",
       "\u001b[0;34m            tensor([[ 0.3741+0.j,  0.2382+0.j],\u001b[0m\n",
       "\u001b[0;34m                    [ 0.5593+0.j, -0.4443+0.j]], dtype=torch.complex128)\u001b[0m\n",
       "\u001b[0;34m            >>> linear(torch.ones(3, 2, dtype=torch.cdouble))\u001b[0m\n",
       "\u001b[0;34m            tensor([[0.6122+0.j, 0.1150+0.j],\u001b[0m\n",
       "\u001b[0;34m                    [0.6122+0.j, 0.1150+0.j],\u001b[0m\n",
       "\u001b[0;34m                    [0.6122+0.j, 0.1150+0.j]], dtype=torch.complex128)\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mdevice\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_blocking\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mconvert_to_format\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_C\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_nn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parse_to\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mdtype\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_floating_point\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_complex\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'nn.Module.to only accepts floating point or complex '\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                                \u001b[0;34m'dtypes, but got desired dtype={}'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_complex\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mwarnings\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"Complex modules are a new feature under active development whose design may change, \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"and some modules might not work as expected when using complex tensors as parameters or buffers. \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"Please file an issue at https://github.com/pytorch/pytorch/issues/new?template=bug-report.md \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"if a complex module does not work as expected.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mdef\u001b[0m \u001b[0mconvert\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mconvert_to_format\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdim\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m4\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_floating_point\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_complex\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                            \u001b[0mnon_blocking\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmemory_format\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mconvert_to_format\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_floating_point\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_complex\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_blocking\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconvert\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mregister_backward_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mCallable\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Module'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_grad_t\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_grad_t\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mRemovableHandle\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Registers a backward hook on the module.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        This function is deprecated in favor of :meth:`nn.Module.register_full_backward_hook` and\u001b[0m\n",
       "\u001b[0;34m        the behavior of this function will change in future versions.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            :class:`torch.utils.hooks.RemovableHandle`:\u001b[0m\n",
       "\u001b[0;34m                a handle that can be used to remove the added hook by calling\u001b[0m\n",
       "\u001b[0;34m                ``handle.remove()``\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_is_full_backward_hook\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Cannot use both regular backward hooks and full backward hooks on a \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                               \u001b[0;34m\"single Module. Please use only one of them.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_is_full_backward_hook\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mhandle\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRemovableHandle\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backward_hooks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backward_hooks\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mhandle\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mid\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mregister_full_backward_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mCallable\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Module'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_grad_t\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_grad_t\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mRemovableHandle\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Registers a backward hook on the module.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        The hook will be called every time the gradients with respect to module\u001b[0m\n",
       "\u001b[0;34m        inputs are computed. The hook should have the following signature::\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            hook(module, grad_input, grad_output) -> tuple(Tensor) or None\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        The :attr:`grad_input` and :attr:`grad_output` are tuples that contain the gradients\u001b[0m\n",
       "\u001b[0;34m        with respect to the inputs and outputs respectively. The hook should\u001b[0m\n",
       "\u001b[0;34m        not modify its arguments, but it can optionally return a new gradient with\u001b[0m\n",
       "\u001b[0;34m        respect to the input that will be used in place of :attr:`grad_input` in\u001b[0m\n",
       "\u001b[0;34m        subsequent computations. :attr:`grad_input` will only correspond to the inputs given\u001b[0m\n",
       "\u001b[0;34m        as positional arguments and all kwarg arguments are ignored. Entries\u001b[0m\n",
       "\u001b[0;34m        in :attr:`grad_input` and :attr:`grad_output` will be ``None`` for all non-Tensor\u001b[0m\n",
       "\u001b[0;34m        arguments.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        .. warning ::\u001b[0m\n",
       "\u001b[0;34m            Modifying inputs or outputs inplace is not allowed when using backward hooks and\u001b[0m\n",
       "\u001b[0;34m            will raise an error.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            :class:`torch.utils.hooks.RemovableHandle`:\u001b[0m\n",
       "\u001b[0;34m                a handle that can be used to remove the added hook by calling\u001b[0m\n",
       "\u001b[0;34m                ``handle.remove()``\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_is_full_backward_hook\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Cannot use both regular backward hooks and full backward hooks on a \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                               \u001b[0;34m\"single Module. Please use only one of them.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_is_full_backward_hook\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mhandle\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRemovableHandle\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backward_hooks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backward_hooks\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mhandle\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mid\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_get_backward_hooks\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Returns the backward hooks for use in the call function.\u001b[0m\n",
       "\u001b[0;34m        It returns two lists, one with the full backward hooks and one with the non-full\u001b[0m\n",
       "\u001b[0;34m        backward hooks.\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mCallable\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0m_global_is_full_backward_hook\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mfull_backward_hooks\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0m_global_backward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_is_full_backward_hook\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mfull_backward_hooks\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mnon_full_backward_hooks\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mCallable\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0m_global_is_full_backward_hook\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0m_global_backward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_is_full_backward_hook\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_maybe_warn_non_full_backward_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad_fn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mall\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mr\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mwarnings\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Using non-full backward hooks on a Module that does not return a \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                              \u001b[0;34m\"single Tensor or a tuple of Tensors is deprecated and will be removed \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                              \u001b[0;34m\"in future versions. This hook will be missing some of the grad_output. \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                              \u001b[0;34m\"Please use register_full_backward_hook to get the documented behavior.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mall\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mwarnings\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Using non-full backward hooks on a Module that does not take as input a \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                              \u001b[0;34m\"single Tensor or a tuple of Tensors is deprecated and will be removed \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                              \u001b[0;34m\"in future versions. This hook will be missing some of the grad_input. \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                              \u001b[0;34m\"Please use register_full_backward_hook to get the documented behavior.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0minputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;31m# At this point we are sure that inputs and result are tuple of Tensors\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mout_grad_fn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad_fn\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mr\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mresult\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad_fn\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mout_grad_fn\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mout_grad_fn\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mgrad_fn\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mout_grad_fn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mwarnings\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Using a non-full backward hook when outputs are nested in python data structure \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                          \u001b[0;34m\"is deprecated and will be removed in future versions. This hook will be missing \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                          \u001b[0;34m\"some grad_output.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mout_grad_fn\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mwarnings\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Using a non-full backward hook when outputs are generated by different autograd Nodes \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                          \u001b[0;34m\"is deprecated and will be removed in future versions. This hook will be missing \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                          \u001b[0;34m\"some grad_output. Please use register_full_backward_hook to get the documented behavior.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;31m# At this point the grad_ouput part of the hook will most likely be correct\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0minputs_grad_fn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad_fn\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0minputs\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad_fn\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mnext_functions\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mn\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mgrad_fn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnext_functions\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0minputs_grad_fn\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mnext_functions\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mwarnings\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Using a non-full backward hook when the forward contains multiple autograd Nodes \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                              \u001b[0;34m\"is deprecated and will be removed in future versions. This hook will be missing \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                              \u001b[0;34m\"some grad_input. Please use register_full_backward_hook to get the documented \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                              \u001b[0;34m\"behavior.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mregister_forward_pre_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mCallable\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m...\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mRemovableHandle\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Registers a forward pre-hook on the module.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        The hook will be called every time before :func:`forward` is invoked.\u001b[0m\n",
       "\u001b[0;34m        It should have the following signature::\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            hook(module, input) -> None or modified input\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        The input contains only the positional arguments given to the module.\u001b[0m\n",
       "\u001b[0;34m        Keyword arguments won't be passed to the hooks and only to the ``forward``.\u001b[0m\n",
       "\u001b[0;34m        The hook can modify the input. User can either return a tuple or a\u001b[0m\n",
       "\u001b[0;34m        single modified value in the hook. We will wrap the value into a tuple\u001b[0m\n",
       "\u001b[0;34m        if a single value is returned(unless that value is already a tuple).\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            :class:`torch.utils.hooks.RemovableHandle`:\u001b[0m\n",
       "\u001b[0;34m                a handle that can be used to remove the added hook by calling\u001b[0m\n",
       "\u001b[0;34m                ``handle.remove()``\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mhandle\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRemovableHandle\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_pre_hooks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_pre_hooks\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mhandle\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mid\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mregister_forward_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mCallable\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m...\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mRemovableHandle\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Registers a forward hook on the module.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        The hook will be called every time after :func:`forward` has computed an output.\u001b[0m\n",
       "\u001b[0;34m        It should have the following signature::\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            hook(module, input, output) -> None or modified output\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        The input contains only the positional arguments given to the module.\u001b[0m\n",
       "\u001b[0;34m        Keyword arguments won't be passed to the hooks and only to the ``forward``.\u001b[0m\n",
       "\u001b[0;34m        The hook can modify the output. It can modify the input inplace but\u001b[0m\n",
       "\u001b[0;34m        it will not have effect on forward since this is called after\u001b[0m\n",
       "\u001b[0;34m        :func:`forward` is called.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            :class:`torch.utils.hooks.RemovableHandle`:\u001b[0m\n",
       "\u001b[0;34m                a handle that can be used to remove the added hook by calling\u001b[0m\n",
       "\u001b[0;34m                ``handle.remove()``\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mhandle\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRemovableHandle\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mhandle\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mid\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mtracing_state\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_C\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_tracing_state\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mtracing_state\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_C\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mScriptMethod\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mrecording_scopes\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjit\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace_module_map\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mrecording_scopes\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;31m# type ignore was added because at this point one knows that\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;31m# torch.jit._trace._trace_module_map is not Optional and has type Dict[Any, Any] \u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mname\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjit\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace_module_map\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjit\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace_module_map\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;32mNone\u001b[0m  \u001b[0;31m# type: ignore\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mtracing_state\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpush_scope\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mrecording_scopes\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mrecording_scopes\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mtracing_state\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpop_scope\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backward_hooks\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_global_backward_hooks\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_backward_hooks\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mitertools\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0m_global_forward_pre_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_pre_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mresult\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0minput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mbw_hook\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mbw_hook\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mBackwardHook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0minput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbw_hook\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetup_input_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_C\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_tracing_state\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mitertools\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0m_global_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mhook_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mhook_result\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook_result\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mbw_hook\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbw_hook\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetup_output_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;31m# Handle the non-full backward hooks\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnon_full_backward_hooks\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mvar\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mwhile\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvar\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvar\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdict\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mvar\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mv\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mv\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mvar\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mv\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mvar\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mvar\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mgrad_fn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mvar\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad_fn\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mgrad_fn\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mwrapper\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunctools\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpartial\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhook\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mfunctools\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate_wrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrapper\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mgrad_fn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mregister_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrapper\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_maybe_warn_non_full_backward_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad_fn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0m__call__\u001b[0m \u001b[0;34m:\u001b[0m \u001b[0mCallable\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m...\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_call_impl\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m__setstate__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstate\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;31m# Support loading old checkpoints that don't have the following attrs:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;34m'_forward_pre_hooks'\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_pre_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mOrderedDict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;34m'_state_dict_hooks'\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_state_dict_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mOrderedDict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;34m'_load_state_dict_pre_hooks'\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_load_state_dict_pre_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mOrderedDict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;34m'_non_persistent_buffers_set'\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_non_persistent_buffers_set\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;34m'_is_full_backward_hook'\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_is_full_backward_hook\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m__getattr__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Module'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;34m'_parameters'\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0m_parameters\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'_parameters'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0m_parameters\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mreturn\u001b[0m \u001b[0m_parameters\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;34m'_buffers'\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0m_buffers\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'_buffers'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0m_buffers\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mreturn\u001b[0m \u001b[0m_buffers\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0;34m'_modules'\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mmodules\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'_modules'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmodules\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mreturn\u001b[0m \u001b[0mmodules\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mraise\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"'{}' object has no attribute '{}'\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m__setattr__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Module'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mdef\u001b[0m \u001b[0mremove_from\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mdicts_or_sets\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mfor\u001b[0m \u001b[0md\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mdicts_or_sets\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0md\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0md\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdict\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0;32mdel\u001b[0m \u001b[0md\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0md\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdiscard\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mparams\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'_parameters'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mParameter\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mparams\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mraise\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;34m\"cannot assign parameters before Module.__init__() call\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mremove_from\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffers\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_non_persistent_buffers_set\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mregister_parameter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0mparams\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mvalue\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"cannot assign '{}' as parameter '{}' \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                                \u001b[0;34m\"(torch.nn.Parameter or None expected)\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                                \u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtypename\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mregister_parameter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mmodules\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'_modules'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mModule\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0mmodules\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mraise\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0;34m\"cannot assign module before Module.__init__() call\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mremove_from\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffers\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_non_persistent_buffers_set\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mmodules\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32melif\u001b[0m \u001b[0mmodules\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmodules\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0mvalue\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"cannot assign '{}' as child module '{}' \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                                    \u001b[0;34m\"(torch.nn.Module or None expected)\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                                    \u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtypename\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mmodules\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mbuffers\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'_buffers'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0mbuffers\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mbuffers\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mif\u001b[0m \u001b[0mvalue\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"cannot assign '{}' as buffer '{}' \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                                        \u001b[0;34m\"(torch.Tensor or None expected)\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                                        \u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtypename\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mbuffers\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mobject\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__setattr__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m__delattr__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mdel\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffers\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mdel\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffers\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_non_persistent_buffers_set\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdiscard\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mdel\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mobject\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__delattr__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_register_state_dict_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"These hooks will be called with arguments: `self`, `state_dict`,\u001b[0m\n",
       "\u001b[0;34m        `prefix`, `local_metadata`, after the `state_dict` of `self` is set.\u001b[0m\n",
       "\u001b[0;34m        Note that only parameters and buffers of `self` or its children are\u001b[0m\n",
       "\u001b[0;34m        guaranteed to exist in `state_dict`. The hooks may modify `state_dict`\u001b[0m\n",
       "\u001b[0;34m        inplace or return a new one.\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mhandle\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRemovableHandle\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_state_dict_hooks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_state_dict_hooks\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mhandle\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mid\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_save_to_state_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdestination\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkeep_vars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Saves module state to `destination` dictionary, containing a state\u001b[0m\n",
       "\u001b[0;34m        of the module, but not its descendants. This is called on every\u001b[0m\n",
       "\u001b[0;34m        submodule in :meth:`~torch.nn.Module.state_dict`.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        In rare cases, subclasses can achieve class-specific behavior by\u001b[0m\n",
       "\u001b[0;34m        overriding this method with custom logic.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Args:\u001b[0m\n",
       "\u001b[0;34m            destination (dict): a dict where state will be stored\u001b[0m\n",
       "\u001b[0;34m            prefix (str): the prefix for parameters and buffers used in this\u001b[0m\n",
       "\u001b[0;34m                module\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparam\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mparam\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mdestination\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mprefix\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparam\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mkeep_vars\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdetach\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuf\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mbuf\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_non_persistent_buffers_set\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mdestination\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mprefix\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbuf\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mkeep_vars\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0mbuf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdetach\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;31m# The user can pass an optional arbitrary mappable object to `state_dict`, in which case `state_dict` returns\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;31m# back that same object. But if they pass nothing, an `OrederedDict` is created and returned.\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mT_destination\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTypeVar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'T_destination'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbound\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mMapping\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m@\u001b[0m\u001b[0moverload\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mstate_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdestination\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mT_destination\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m...\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkeep_vars\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m...\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mT_destination\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m...\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;31m# TODO: annotate with OrderedDict not Dict, but there is a problem:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;31m# https://docs.python.org/3/library/typing.html#typing.OrderedDict\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m@\u001b[0m\u001b[0moverload\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mstate_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m...\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkeep_vars\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m...\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mDict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34m...\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mstate_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdestination\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkeep_vars\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Returns a dictionary containing a whole state of the module.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Both parameters and persistent buffers (e.g. running averages) are\u001b[0m\n",
       "\u001b[0;34m        included. Keys are corresponding parameter and buffer names.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            dict:\u001b[0m\n",
       "\u001b[0;34m                a dictionary containing a whole state of the module\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Example::\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            >>> module.state_dict().keys()\u001b[0m\n",
       "\u001b[0;34m            ['bias', 'weight']\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mdestination\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mdestination\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mOrderedDict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mdestination\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_metadata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mOrderedDict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mdestination\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_metadata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mprefix\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlocal_metadata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mversion\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_version\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_save_to_state_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdestination\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkeep_vars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstate_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdestination\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mname\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'.'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkeep_vars\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkeep_vars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_state_dict_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mhook_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdestination\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlocal_metadata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mhook_result\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mdestination\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook_result\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mdestination\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_register_load_state_dict_pre_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"These hooks will be called with arguments: `state_dict`, `prefix`,\u001b[0m\n",
       "\u001b[0;34m        `local_metadata`, `strict`, `missing_keys`, `unexpected_keys`,\u001b[0m\n",
       "\u001b[0;34m        `error_msgs`, before loading `state_dict` into `self`. These arguments\u001b[0m\n",
       "\u001b[0;34m        are exactly the same as those of `_load_from_state_dict`.\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mhandle\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRemovableHandle\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_load_state_dict_pre_hooks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_load_state_dict_pre_hooks\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mhandle\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mid\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_load_from_state_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstate_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlocal_metadata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstrict\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                              \u001b[0mmissing_keys\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0munexpected_keys\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0merror_msgs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Copies parameters and buffers from :attr:`state_dict` into only\u001b[0m\n",
       "\u001b[0;34m        this module, but not its descendants. This is called on every submodule\u001b[0m\n",
       "\u001b[0;34m        in :meth:`~torch.nn.Module.load_state_dict`. Metadata saved for this\u001b[0m\n",
       "\u001b[0;34m        module in input :attr:`state_dict` is provided as :attr:`local_metadata`.\u001b[0m\n",
       "\u001b[0;34m        For state dicts without metadata, :attr:`local_metadata` is empty.\u001b[0m\n",
       "\u001b[0;34m        Subclasses can achieve class-specific backward compatible loading using\u001b[0m\n",
       "\u001b[0;34m        the version number at `local_metadata.get(\"version\", None)`.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        .. note::\u001b[0m\n",
       "\u001b[0;34m            :attr:`state_dict` is not the same object as the input\u001b[0m\n",
       "\u001b[0;34m            :attr:`state_dict` to :meth:`~torch.nn.Module.load_state_dict`. So\u001b[0m\n",
       "\u001b[0;34m            it can be modified.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Args:\u001b[0m\n",
       "\u001b[0;34m            state_dict (dict): a dict containing parameters and\u001b[0m\n",
       "\u001b[0;34m                persistent buffers.\u001b[0m\n",
       "\u001b[0;34m            prefix (str): the prefix for parameters and buffers used in this\u001b[0m\n",
       "\u001b[0;34m                module\u001b[0m\n",
       "\u001b[0;34m            local_metadata (dict): a dict containing the metadata for this module.\u001b[0m\n",
       "\u001b[0;34m                See\u001b[0m\n",
       "\u001b[0;34m            strict (bool): whether to strictly enforce that the keys in\u001b[0m\n",
       "\u001b[0;34m                :attr:`state_dict` with :attr:`prefix` match the names of\u001b[0m\n",
       "\u001b[0;34m                parameters and buffers in this module\u001b[0m\n",
       "\u001b[0;34m            missing_keys (list of str): if ``strict=True``, add missing keys to\u001b[0m\n",
       "\u001b[0;34m                this list\u001b[0m\n",
       "\u001b[0;34m            unexpected_keys (list of str): if ``strict=True``, add unexpected\u001b[0m\n",
       "\u001b[0;34m                keys to this list\u001b[0m\n",
       "\u001b[0;34m            error_msgs (list of str): error messages should be added to this\u001b[0m\n",
       "\u001b[0;34m                list, and will be reported together in\u001b[0m\n",
       "\u001b[0;34m                :meth:`~torch.nn.Module.load_state_dict`\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_load_state_dict_pre_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstate_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlocal_metadata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstrict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmissing_keys\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0munexpected_keys\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0merror_msgs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mpersistent_buffers\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mv\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mv\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mk\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_non_persistent_buffers_set\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mlocal_name_params\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mitertools\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpersistent_buffers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mlocal_state\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mv\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mv\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mlocal_name_params\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mv\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparam\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mlocal_state\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mkey\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mprefix\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mkey\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mstate_dict\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0minput_param\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstate_dict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;31m# This is used to avoid copying uninitialized parameters into\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;31m# non-lazy modules, since they dont have the hook to do the checks\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;31m# in such case, it will error when accessing the .shape attribute.\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mis_param_lazy\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparam\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparameter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mUninitializedParameter\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;31m# Backward compatibility: loading 1-dim tensor from 0.3.* to version 0.4+\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mis_param_lazy\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_param\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0minput_param\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minput_param\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mis_param_lazy\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0minput_param\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;31m# local shape should match the one in checkpoint\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0merror_msgs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'size mismatch for {}: copying a param with shape {} from checkpoint, '\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                                      \u001b[0;34m'the shape in current model is {}.'\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                                      \u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput_param\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mcontinue\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mno_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_param\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mex\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0merror_msgs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'While copying the parameter named \"{}\", '\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                                      \u001b[0;34m'whose dimensions in the model are {} and '\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                                      \u001b[0;34m'whose dimensions in the checkpoint are {}, '\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                                      \u001b[0;34m'an exception occurred : {}.'\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                                      \u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput_param\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mex\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32melif\u001b[0m \u001b[0mstrict\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mmissing_keys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mstrict\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mfor\u001b[0m \u001b[0mkey\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mstate_dict\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprefix\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0minput_name\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprefix\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0minput_name\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minput_name\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'.'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m  \u001b[0;31m# get the name of param/buffer/child\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mif\u001b[0m \u001b[0minput_name\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0minput_name\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mlocal_state\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0munexpected_keys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mload_state_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstate_dict\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'OrderedDict[str, Tensor]'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mstrict\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Copies parameters and buffers from :attr:`state_dict` into\u001b[0m\n",
       "\u001b[0;34m        this module and its descendants. If :attr:`strict` is ``True``, then\u001b[0m\n",
       "\u001b[0;34m        the keys of :attr:`state_dict` must exactly match the keys returned\u001b[0m\n",
       "\u001b[0;34m        by this module's :meth:`~torch.nn.Module.state_dict` function.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Args:\u001b[0m\n",
       "\u001b[0;34m            state_dict (dict): a dict containing parameters and\u001b[0m\n",
       "\u001b[0;34m                persistent buffers.\u001b[0m\n",
       "\u001b[0;34m            strict (bool, optional): whether to strictly enforce that the keys\u001b[0m\n",
       "\u001b[0;34m                in :attr:`state_dict` match the keys returned by this module's\u001b[0m\n",
       "\u001b[0;34m                :meth:`~torch.nn.Module.state_dict` function. Default: ``True``\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields:\u001b[0m\n",
       "\u001b[0;34m                * **missing_keys** is a list of str containing the missing keys\u001b[0m\n",
       "\u001b[0;34m                * **unexpected_keys** is a list of str containing the unexpected keys\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mmissing_keys\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0munexpected_keys\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0merror_msgs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;31m# copy state_dict so _load_from_state_dict can modify it\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mmetadata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstate_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'_metadata'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mstate_dict\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstate_dict\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mmetadata\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;31m# mypy isn't aware that \"_metadata\" exists in state_dict\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mstate_dict\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_metadata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmetadata\u001b[0m  \u001b[0;31m# type: ignore[attr-defined]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mdef\u001b[0m \u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodule\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m''\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mlocal_metadata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mmetadata\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0mmetadata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprefix\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_load_from_state_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mstate_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlocal_metadata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmissing_keys\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0munexpected_keys\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0merror_msgs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mfor\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mchild\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0mchild\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchild\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mname\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'.'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mdel\u001b[0m \u001b[0mload\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mstrict\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0munexpected_keys\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0merror_msgs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minsert\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Unexpected key(s) in state_dict: {}. '\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0;34m', '\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'\"{}\"'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m \u001b[0;32min\u001b[0m \u001b[0munexpected_keys\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmissing_keys\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0merror_msgs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minsert\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Missing key(s) in state_dict: {}. '\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0;34m', '\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'\"{}\"'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmissing_keys\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merror_msgs\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Error(s) in loading state_dict for {}:\\n\\t{}'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                               \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__class__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"\\n\\t\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merror_msgs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0m_IncompatibleKeys\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmissing_keys\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0munexpected_keys\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_named_members\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mget_members_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrecurse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Helper method for yielding various names + members of modules.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mmemo\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mmodules\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnamed_modules\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprefix\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mprefix\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mrecurse\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprefix\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mmodule_prefix\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmodules\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mmembers\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_members_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodule\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mv\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmembers\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0mv\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mv\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmemo\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mcontinue\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mmemo\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mv\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mname\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule_prefix\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m'.'\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mmodule_prefix\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32myield\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mv\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mparameters\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrecurse\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mIterator\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mParameter\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Returns an iterator over module parameters.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        This is typically passed to an optimizer.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Args:\u001b[0m\n",
       "\u001b[0;34m            recurse (bool): if True, then yields parameters of this module\u001b[0m\n",
       "\u001b[0;34m                and all submodules. Otherwise, yields only parameters that\u001b[0m\n",
       "\u001b[0;34m                are direct members of this module.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Yields:\u001b[0m\n",
       "\u001b[0;34m            Parameter: module parameter\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Example::\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            >>> for param in model.parameters():\u001b[0m\n",
       "\u001b[0;34m            >>>     print(type(param), param.size())\u001b[0m\n",
       "\u001b[0;34m            <class 'torch.Tensor'> (20L,)\u001b[0m\n",
       "\u001b[0;34m            <class 'torch.Tensor'> (20L, 1L, 5L, 5L)\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparam\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnamed_parameters\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrecurse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrecurse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32myield\u001b[0m \u001b[0mparam\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mnamed_parameters\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrecurse\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mIterator\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mTuple\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mParameter\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Returns an iterator over module parameters, yielding both the\u001b[0m\n",
       "\u001b[0;34m        name of the parameter as well as the parameter itself.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Args:\u001b[0m\n",
       "\u001b[0;34m            prefix (str): prefix to prepend to all parameter names.\u001b[0m\n",
       "\u001b[0;34m            recurse (bool): if True, then yields parameters of this module\u001b[0m\n",
       "\u001b[0;34m                and all submodules. Otherwise, yields only parameters that\u001b[0m\n",
       "\u001b[0;34m                are direct members of this module.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Yields:\u001b[0m\n",
       "\u001b[0;34m            (string, Parameter): Tuple containing the name and parameter\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Example::\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            >>> for name, param in self.named_parameters():\u001b[0m\n",
       "\u001b[0;34m            >>>    if name in ['bias']:\u001b[0m\n",
       "\u001b[0;34m            >>>        print(param.size())\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mgen\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_named_members\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mlambda\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mprefix\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mprefix\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrecurse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrecurse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0melem\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mgen\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32myield\u001b[0m \u001b[0melem\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mbuffers\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrecurse\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mIterator\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Returns an iterator over module buffers.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Args:\u001b[0m\n",
       "\u001b[0;34m            recurse (bool): if True, then yields buffers of this module\u001b[0m\n",
       "\u001b[0;34m                and all submodules. Otherwise, yields only buffers that\u001b[0m\n",
       "\u001b[0;34m                are direct members of this module.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Yields:\u001b[0m\n",
       "\u001b[0;34m            torch.Tensor: module buffer\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Example::\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            >>> for buf in model.buffers():\u001b[0m\n",
       "\u001b[0;34m            >>>     print(type(buf), buf.size())\u001b[0m\n",
       "\u001b[0;34m            <class 'torch.Tensor'> (20L,)\u001b[0m\n",
       "\u001b[0;34m            <class 'torch.Tensor'> (20L, 1L, 5L, 5L)\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuf\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnamed_buffers\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrecurse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrecurse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32myield\u001b[0m \u001b[0mbuf\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mnamed_buffers\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrecurse\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mIterator\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mTuple\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Returns an iterator over module buffers, yielding both the\u001b[0m\n",
       "\u001b[0;34m        name of the buffer as well as the buffer itself.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Args:\u001b[0m\n",
       "\u001b[0;34m            prefix (str): prefix to prepend to all buffer names.\u001b[0m\n",
       "\u001b[0;34m            recurse (bool): if True, then yields buffers of this module\u001b[0m\n",
       "\u001b[0;34m                and all submodules. Otherwise, yields only buffers that\u001b[0m\n",
       "\u001b[0;34m                are direct members of this module.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Yields:\u001b[0m\n",
       "\u001b[0;34m            (string, torch.Tensor): Tuple containing the name and buffer\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Example::\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            >>> for name, buf in self.named_buffers():\u001b[0m\n",
       "\u001b[0;34m            >>>    if name in ['running_var']:\u001b[0m\n",
       "\u001b[0;34m            >>>        print(buf.size())\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mgen\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_named_members\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mlambda\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mprefix\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mprefix\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrecurse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrecurse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0melem\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mgen\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32myield\u001b[0m \u001b[0melem\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mchildren\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mIterator\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Module'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Returns an iterator over immediate children modules.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Yields:\u001b[0m\n",
       "\u001b[0;34m            Module: a child module\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnamed_children\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32myield\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mnamed_children\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mIterator\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mTuple\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Module'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Returns an iterator over immediate children modules, yielding both\u001b[0m\n",
       "\u001b[0;34m        the name of the module as well as the module itself.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Yields:\u001b[0m\n",
       "\u001b[0;34m            (string, Module): Tuple containing a name and child module\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Example::\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            >>> for name, module in model.named_children():\u001b[0m\n",
       "\u001b[0;34m            >>>     if name in ['conv4', 'conv5']:\u001b[0m\n",
       "\u001b[0;34m            >>>         print(module)\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mmemo\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmemo\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mmemo\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodule\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32myield\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mmodules\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mIterator\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Module'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Returns an iterator over all modules in the network.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Yields:\u001b[0m\n",
       "\u001b[0;34m            Module: a module in the network\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Note:\u001b[0m\n",
       "\u001b[0;34m            Duplicate modules are returned only once. In the following\u001b[0m\n",
       "\u001b[0;34m            example, ``l`` will be returned only once.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Example::\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            >>> l = nn.Linear(2, 2)\u001b[0m\n",
       "\u001b[0;34m            >>> net = nn.Sequential(l, l)\u001b[0m\n",
       "\u001b[0;34m            >>> for idx, m in enumerate(net.modules()):\u001b[0m\n",
       "\u001b[0;34m                    print(idx, '->', m)\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            0 -> Sequential(\u001b[0m\n",
       "\u001b[0;34m              (0): Linear(in_features=2, out_features=2, bias=True)\u001b[0m\n",
       "\u001b[0;34m              (1): Linear(in_features=2, out_features=2, bias=True)\u001b[0m\n",
       "\u001b[0;34m            )\u001b[0m\n",
       "\u001b[0;34m            1 -> Linear(in_features=2, out_features=2, bias=True)\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnamed_modules\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32myield\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mnamed_modules\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmemo\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mSet\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Module'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Returns an iterator over all modules in the network, yielding\u001b[0m\n",
       "\u001b[0;34m        both the name of the module as well as the module itself.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Yields:\u001b[0m\n",
       "\u001b[0;34m            (string, Module): Tuple of name and module\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Note:\u001b[0m\n",
       "\u001b[0;34m            Duplicate modules are returned only once. In the following\u001b[0m\n",
       "\u001b[0;34m            example, ``l`` will be returned only once.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Example::\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            >>> l = nn.Linear(2, 2)\u001b[0m\n",
       "\u001b[0;34m            >>> net = nn.Sequential(l, l)\u001b[0m\n",
       "\u001b[0;34m            >>> for idx, m in enumerate(net.named_modules()):\u001b[0m\n",
       "\u001b[0;34m                    print(idx, '->', m)\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m            0 -> ('', Sequential(\u001b[0m\n",
       "\u001b[0;34m              (0): Linear(in_features=2, out_features=2, bias=True)\u001b[0m\n",
       "\u001b[0;34m              (1): Linear(in_features=2, out_features=2, bias=True)\u001b[0m\n",
       "\u001b[0;34m            ))\u001b[0m\n",
       "\u001b[0;34m            1 -> ('0', Linear(in_features=2, out_features=2, bias=True))\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mmemo\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mmemo\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmemo\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mmemo\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32myield\u001b[0m \u001b[0mprefix\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mfor\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mcontinue\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0msubmodule_prefix\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mprefix\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m'.'\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mprefix\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mfor\u001b[0m \u001b[0mm\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnamed_modules\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmemo\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msubmodule_prefix\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32myield\u001b[0m \u001b[0mm\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Sets the module in training mode.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        This has any effect only on certain modules. See documentations of\u001b[0m\n",
       "\u001b[0;34m        particular modules for details of their behaviors in training/evaluation\u001b[0m\n",
       "\u001b[0;34m        mode, if they are affected, e.g. :class:`Dropout`, :class:`BatchNorm`,\u001b[0m\n",
       "\u001b[0;34m        etc.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Args:\u001b[0m\n",
       "\u001b[0;34m            mode (bool): whether to set training mode (``True``) or evaluation\u001b[0m\n",
       "\u001b[0;34m                         mode (``False``). Default: ``True``.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            Module: self\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtraining\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchildren\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0meval\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Sets the module in evaluation mode.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        This has any effect only on certain modules. See documentations of\u001b[0m\n",
       "\u001b[0;34m        particular modules for details of their behaviors in training/evaluation\u001b[0m\n",
       "\u001b[0;34m        mode, if they are affected, e.g. :class:`Dropout`, :class:`BatchNorm`,\u001b[0m\n",
       "\u001b[0;34m        etc.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        This is equivalent with :meth:`self.train(False) <torch.nn.Module.train>`.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            Module: self\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mrequires_grad_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrequires_grad\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Change if autograd should record operations on parameters in this\u001b[0m\n",
       "\u001b[0;34m        module.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        This method sets the parameters' :attr:`requires_grad` attributes\u001b[0m\n",
       "\u001b[0;34m        in-place.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        This method is helpful for freezing part of the module for finetuning\u001b[0m\n",
       "\u001b[0;34m        or training parts of a model individually (e.g., GAN training).\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Args:\u001b[0m\n",
       "\u001b[0;34m            requires_grad (bool): whether autograd should record operations on\u001b[0m\n",
       "\u001b[0;34m                                  parameters in this module. Default: ``True``.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Returns:\u001b[0m\n",
       "\u001b[0;34m            Module: self\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mp\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparameters\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequires_grad_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrequires_grad\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mset_to_none\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Sets gradients of all model parameters to zero. See similar function\u001b[0m\n",
       "\u001b[0;34m        under :class:`torch.optim.Optimizer` for more context.\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        Args:\u001b[0m\n",
       "\u001b[0;34m            set_to_none (bool): instead of setting to zero, set the grads to None.\u001b[0m\n",
       "\u001b[0;34m                See :meth:`torch.optim.Optimizer.zero_grad` for details.\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'_is_replica'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mwarnings\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;34m\"Calling .zero_grad() from a module created with nn.DataParallel() has no effect. \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;34m\"The parameters are copied (in a differentiable manner) from the original module. \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;34m\"This means they are not leaf nodes in autograd and so don't accumulate gradients. \"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;34m\"If you need gradients in your forward method, consider using autograd.grad instead.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mp\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparameters\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32mif\u001b[0m \u001b[0mset_to_none\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32mif\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad_fn\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdetach_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                        \u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequires_grad_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                    \u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgrad\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mshare_memory\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshare_memory_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_get_name\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__class__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0mextra_repr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;34mr\"\"\"Set the extra representation of the module\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m        To print customized extra information, you should re-implement\u001b[0m\n",
       "\u001b[0;34m        this method in your own modules. Both single-line and multi-line\u001b[0m\n",
       "\u001b[0;34m        strings are acceptable.\u001b[0m\n",
       "\u001b[0;34m        \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m__repr__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;31m# We treat the extra repr like the sub-module, one item per line\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mextra_lines\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mextra_repr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mextra_repr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;31m# empty string will be split into list ['']\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mextra_repr\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mextra_lines\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mextra_repr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'\\n'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mchild_lines\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mfor\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mmod_str\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrepr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodule\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mmod_str\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_addindent\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmod_str\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0mchild_lines\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'('\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mkey\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'): '\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mmod_str\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mlines\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mextra_lines\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mchild_lines\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mmain_str\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_name\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'('\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mif\u001b[0m \u001b[0mlines\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;31m# simple one-liner info, which most builtin Modules will use\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mextra_lines\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mchild_lines\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mmain_str\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mextra_lines\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m            \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m                \u001b[0mmain_str\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;34m'\\n  '\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'\\n  '\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlines\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'\\n'\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mmain_str\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;34m')'\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mmain_str\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m__dir__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mmodule_attrs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__class__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mattrs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mparameters\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mmodules\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mbuffers\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mkeys\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule_attrs\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mattrs\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mparameters\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mmodules\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mbuffers\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;31m# Eliminate attrs that are not legal Python variable names\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mkeys\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mkey\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mkey\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mkeys\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0misdigit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0msorted\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;32mdef\u001b[0m \u001b[0m_replicate_for_data_parallel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mreplica\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__new__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mreplica\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;31m# replicas do not have parameters themselves, the replicas reference the original\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;31m# module.\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mreplica\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mOrderedDict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mreplica\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffers\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mreplica\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mreplica\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mreplica\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_modules\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0mreplica\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_is_replica\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m        \u001b[0;32mreturn\u001b[0m \u001b[0mreplica\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mFile:\u001b[0m           ~/.local/lib/python3.7/site-packages/torch/nn/modules/module.py\n",
       "\u001b[0;31mType:\u001b[0m           type\n",
       "\u001b[0;31mSubclasses:\u001b[0m     Identity, Linear, Bilinear, _ConvNd, Threshold, ReLU, RReLU, Hardtanh, Sigmoid, Hardsigmoid, ...\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "torch.nn.Module??"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropping to Bash: magic!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 932K\n",
      "-rw-r--r--  1 ahmed.sy users 8.6K May  2 12:47 00-logic_neuron_programming.ipynb\n",
      "-rw-r--r--  1 ahmed.sy users  22K May  2 13:04 01-tensor_tutorial.ipynb\n",
      "-rw-r--r--  1 ahmed.sy users 6.6K May  2 12:47 02-space_stretching.ipynb\n",
      "-rw-r--r--  1 ahmed.sy users 6.7K May  2 12:47 03-autograd_tutorial.ipynb\n",
      "-rw-r--r--  1 ahmed.sy users 7.3K May  2 12:47 04-spiral_classification.ipynb\n",
      "-rw-r--r--  1 ahmed.sy users 9.4K May  2 12:47 05-regression.ipynb\n",
      "-rw-r--r--  1 ahmed.sy users  14K May  2 12:47 06-convnet.ipynb\n",
      "-rw-r--r--  1 ahmed.sy users 8.6K May  2 12:47 07-listening_to_kernels.ipynb\n",
      "-rw-r--r--  1 ahmed.sy users  24K May  2 12:47 08-seq_classification.ipynb\n",
      "-rw-r--r--  1 ahmed.sy users 7.2K May  2 12:47 09-echo_data.ipynb\n",
      "-rw-r--r--  1 ahmed.sy users 7.3K May  2 12:47 10-autoencoder.ipynb\n",
      "-rw-r--r--  1 ahmed.sy users  11K May  2 12:47 11-VAE.ipynb\n",
      "-rw-r--r--  1 ahmed.sy users  20K May  2 12:47 12-regularization.ipynb\n",
      "-rw-r--r--  1 ahmed.sy users 4.4K May  2 12:47 13-bayesian_nn.ipynb\n",
      "-rw-r--r--  1 ahmed.sy users  13K May  2 12:47 14-truck_backer-upper.ipynb\n",
      "-rw-r--r--  1 ahmed.sy users  26K May  2 12:47 15-transformer.ipynb\n",
      "-rw-r--r--  1 ahmed.sy users 604K May  2 12:47 16-gated_GCN.ipynb\n",
      "-rw-r--r--  1 ahmed.sy users   16 May  2 12:47 apt.txt\n",
      "drwxr-xr-x 20 ahmed.sy users 4.0K May  2 12:47 docs\n",
      "-rw-r--r--  1 ahmed.sy users  274 May  2 12:47 environment.yml\n",
      "drwxr-xr-x  3 ahmed.sy users 4.0K May  2 12:47 extra\n",
      "-rw-r--r--  1 ahmed.sy users  281 May  2 12:47 LICENSE.md\n",
      "-rw-r--r--  1 ahmed.sy users 3.7K May  2 12:47 README.md\n",
      "drwxr-xr-x  2 ahmed.sy users 4.0K May  2 12:47 res\n",
      "drwxr-xr-x  2 ahmed.sy users 4.0K May  2 12:47 slides\n"
     ]
    }
   ],
   "source": [
    "# List all the files in the current directory\n",
    "!ls -lh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "442 00-logic_neuron_programming.ipynb\n",
      "1013 01-tensor_tutorial.ipynb\n",
      "273 02-space_stretching.ipynb\n",
      "331 03-autograd_tutorial.ipynb\n",
      "294 04-spiral_classification.ipynb\n",
      "362 05-regression.ipynb\n",
      "458 06-convnet.ipynb\n",
      "337 07-listening_to_kernels.ipynb\n",
      "660 08-seq_classification.ipynb\n",
      "255 09-echo_data.ipynb\n",
      "264 10-autoencoder.ipynb\n",
      "353 11-VAE.ipynb\n",
      "640 12-regularization.ipynb\n",
      "195 13-bayesian_nn.ipynb\n",
      "422 14-truck_backer-upper.ipynb\n",
      "772 15-transformer.ipynb\n",
      "1083 16-gated_GCN.ipynb\n",
      "1 apt.txt\n",
      "20 environment.yml\n",
      "4 LICENSE.md\n",
      "71 README.md\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# List all the files but with cleaner outputs for readability\n",
    "for f in $(ls *.*); do\n",
    "    echo $(wc -l $f)\n",
    "done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "IPython's 'magic' functions\n",
       "===========================\n",
       "\n",
       "The magic function system provides a series of functions which allow you to\n",
       "control the behavior of IPython itself, plus a lot of system-type\n",
       "features. There are two kinds of magics, line-oriented and cell-oriented.\n",
       "\n",
       "Line magics are prefixed with the % character and work much like OS\n",
       "command-line calls: they get as an argument the rest of the line, where\n",
       "arguments are passed without parentheses or quotes.  For example, this will\n",
       "time the given statement::\n",
       "\n",
       "        %timeit range(1000)\n",
       "\n",
       "Cell magics are prefixed with a double %%, and they are functions that get as\n",
       "an argument not only the rest of the line, but also the lines below it in a\n",
       "separate argument.  These magics are called with two arguments: the rest of the\n",
       "call line and the body of the cell, consisting of the lines below the first.\n",
       "For example::\n",
       "\n",
       "        %%timeit x = numpy.random.randn((100, 100))\n",
       "        numpy.linalg.svd(x)\n",
       "\n",
       "will time the execution of the numpy svd routine, running the assignment of x\n",
       "as part of the setup phase, which is not timed.\n",
       "\n",
       "In a line-oriented client (the terminal or Qt console IPython), starting a new\n",
       "input with %% will automatically enter cell mode, and IPython will continue\n",
       "reading input until a blank line is given.  In the notebook, simply type the\n",
       "whole cell as one entity, but keep in mind that the %% escape can only be at\n",
       "the very start of the cell.\n",
       "\n",
       "NOTE: If you have 'automagic' enabled (via the command line option or with the\n",
       "%automagic function), you don't need to type in the % explicitly for line\n",
       "magics; cell magics always require an explicit '%%' escape.  By default,\n",
       "IPython ships with automagic on, so you should only rarely need the % escape.\n",
       "\n",
       "Example: typing '%cd mydir' (without the quotes) changes your working directory\n",
       "to 'mydir', if it exists.\n",
       "\n",
       "For a list of the available magic functions, use %lsmagic. For a description\n",
       "of any of them, type %magic_name?, e.g. '%cd?'.\n",
       "\n",
       "Currently the magic system has the following functions:\n",
       "%alias:\n",
       "    Define an alias for a system command.\n",
       "    \n",
       "    '%alias alias_name cmd' defines 'alias_name' as an alias for 'cmd'\n",
       "    \n",
       "    Then, typing 'alias_name params' will execute the system command 'cmd\n",
       "    params' (from your underlying operating system).\n",
       "    \n",
       "    Aliases have lower precedence than magic functions and Python normal\n",
       "    variables, so if 'foo' is both a Python variable and an alias, the\n",
       "    alias can not be executed until 'del foo' removes the Python variable.\n",
       "    \n",
       "    You can use the %l specifier in an alias definition to represent the\n",
       "    whole line when the alias is called.  For example::\n",
       "    \n",
       "      In [2]: alias bracket echo \"Input in brackets: <%l>\"\n",
       "      In [3]: bracket hello world\n",
       "      Input in brackets: <hello world>\n",
       "    \n",
       "    You can also define aliases with parameters using %s specifiers (one\n",
       "    per parameter)::\n",
       "    \n",
       "      In [1]: alias parts echo first %s second %s\n",
       "      In [2]: %parts A B\n",
       "      first A second B\n",
       "      In [3]: %parts A\n",
       "      Incorrect number of arguments: 2 expected.\n",
       "      parts is an alias to: 'echo first %s second %s'\n",
       "    \n",
       "    Note that %l and %s are mutually exclusive.  You can only use one or\n",
       "    the other in your aliases.\n",
       "    \n",
       "    Aliases expand Python variables just like system calls using ! or !!\n",
       "    do: all expressions prefixed with '$' get expanded.  For details of\n",
       "    the semantic rules, see PEP-215:\n",
       "    http://www.python.org/peps/pep-0215.html.  This is the library used by\n",
       "    IPython for variable expansion.  If you want to access a true shell\n",
       "    variable, an extra $ is necessary to prevent its expansion by\n",
       "    IPython::\n",
       "    \n",
       "      In [6]: alias show echo\n",
       "      In [7]: PATH='A Python string'\n",
       "      In [8]: show $PATH\n",
       "      A Python string\n",
       "      In [9]: show $$PATH\n",
       "      /usr/local/lf9560/bin:/usr/local/intel/compiler70/ia32/bin:...\n",
       "    \n",
       "    You can use the alias facility to access all of $PATH.  See the %rehashx\n",
       "    function, which automatically creates aliases for the contents of your\n",
       "    $PATH.\n",
       "    \n",
       "    If called with no parameters, %alias prints the current alias table\n",
       "    for your system.  For posix systems, the default aliases are 'cat',\n",
       "    'cp', 'mv', 'rm', 'rmdir', and 'mkdir', and other platform-specific\n",
       "    aliases are added.  For windows-based systems, the default aliases are\n",
       "    'copy', 'ddir', 'echo', 'ls', 'ldir', 'mkdir', 'ren', and 'rmdir'.\n",
       "    \n",
       "    You can see the definition of alias by adding a question mark in the\n",
       "    end::\n",
       "    \n",
       "      In [1]: cat?\n",
       "      Repr: <alias cat for 'cat'>\n",
       "%alias_magic:\n",
       "    ::\n",
       "    \n",
       "      %alias_magic [-l] [-c] [-p PARAMS] name target\n",
       "    \n",
       "    Create an alias for an existing line or cell magic.\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    ::\n",
       "    \n",
       "      In [1]: %alias_magic t timeit\n",
       "      Created `%t` as an alias for `%timeit`.\n",
       "      Created `%%t` as an alias for `%%timeit`.\n",
       "    \n",
       "      In [2]: %t -n1 pass\n",
       "      1 loops, best of 3: 954 ns per loop\n",
       "    \n",
       "      In [3]: %%t -n1\n",
       "         ...: pass\n",
       "         ...:\n",
       "      1 loops, best of 3: 954 ns per loop\n",
       "    \n",
       "      In [4]: %alias_magic --cell whereami pwd\n",
       "      UsageError: Cell magic function `%%pwd` not found.\n",
       "      In [5]: %alias_magic --line whereami pwd\n",
       "      Created `%whereami` as an alias for `%pwd`.\n",
       "    \n",
       "      In [6]: %whereami\n",
       "      Out[6]: u'/home/testuser'\n",
       "    \n",
       "      In [7]: %alias_magic h history \"-p -l 30\" --line\n",
       "      Created `%h` as an alias for `%history -l 30`.\n",
       "    \n",
       "    positional arguments:\n",
       "      name                  Name of the magic to be created.\n",
       "      target                Name of the existing line or cell magic.\n",
       "    \n",
       "    optional arguments:\n",
       "      -l, --line            Create a line magic alias.\n",
       "      -c, --cell            Create a cell magic alias.\n",
       "      -p PARAMS, --params PARAMS\n",
       "                            Parameters passed to the magic function.\n",
       "%autoawait:\n",
       "    \n",
       "    Allow to change the status of the autoawait option.\n",
       "    \n",
       "    This allow you to set a specific asynchronous code runner.\n",
       "    \n",
       "    If no value is passed, print the currently used asynchronous integration\n",
       "    and whether it is activated.\n",
       "    \n",
       "    It can take a number of value evaluated in the following order:\n",
       "    \n",
       "    - False/false/off deactivate autoawait integration\n",
       "    - True/true/on activate autoawait integration using configured default\n",
       "      loop\n",
       "    - asyncio/curio/trio activate autoawait integration and use integration\n",
       "      with said library.\n",
       "    \n",
       "    - `sync` turn on the pseudo-sync integration (mostly used for\n",
       "      `IPython.embed()` which does not run IPython with a real eventloop and\n",
       "      deactivate running asynchronous code. Turning on Asynchronous code with\n",
       "      the pseudo sync loop is undefined behavior and may lead IPython to crash.\n",
       "    \n",
       "    If the passed parameter does not match any of the above and is a python\n",
       "    identifier, get said object from user namespace and set it as the\n",
       "    runner, and activate autoawait. \n",
       "    \n",
       "    If the object is a fully qualified object name, attempt to import it and\n",
       "    set it as the runner, and activate autoawait.\n",
       "    \n",
       "    \n",
       "    The exact behavior of autoawait is experimental and subject to change\n",
       "    across version of IPython and Python.\n",
       "%autocall:\n",
       "    Make functions callable without having to type parentheses.\n",
       "    \n",
       "    Usage:\n",
       "    \n",
       "       %autocall [mode]\n",
       "    \n",
       "    The mode can be one of: 0->Off, 1->Smart, 2->Full.  If not given, the\n",
       "    value is toggled on and off (remembering the previous state).\n",
       "    \n",
       "    In more detail, these values mean:\n",
       "    \n",
       "    0 -> fully disabled\n",
       "    \n",
       "    1 -> active, but do not apply if there are no arguments on the line.\n",
       "    \n",
       "    In this mode, you get::\n",
       "    \n",
       "      In [1]: callable\n",
       "      Out[1]: <built-in function callable>\n",
       "    \n",
       "      In [2]: callable 'hello'\n",
       "      ------> callable('hello')\n",
       "      Out[2]: False\n",
       "    \n",
       "    2 -> Active always.  Even if no arguments are present, the callable\n",
       "    object is called::\n",
       "    \n",
       "      In [2]: float\n",
       "      ------> float()\n",
       "      Out[2]: 0.0\n",
       "    \n",
       "    Note that even with autocall off, you can still use '/' at the start of\n",
       "    a line to treat the first argument on the command line as a function\n",
       "    and add parentheses to it::\n",
       "    \n",
       "      In [8]: /str 43\n",
       "      ------> str(43)\n",
       "      Out[8]: '43'\n",
       "    \n",
       "    # all-random (note for auto-testing)\n",
       "%automagic:\n",
       "    Make magic functions callable without having to type the initial %.\n",
       "    \n",
       "    Without arguments toggles on/off (when off, you must call it as\n",
       "    %automagic, of course).  With arguments it sets the value, and you can\n",
       "    use any of (case insensitive):\n",
       "    \n",
       "     - on, 1, True: to activate\n",
       "    \n",
       "     - off, 0, False: to deactivate.\n",
       "    \n",
       "    Note that magic functions have lowest priority, so if there's a\n",
       "    variable whose name collides with that of a magic fn, automagic won't\n",
       "    work for that function (you get the variable instead). However, if you\n",
       "    delete the variable (del var), the previously shadowed magic function\n",
       "    becomes visible to automagic again.\n",
       "%autosave:\n",
       "    Set the autosave interval in the notebook (in seconds).\n",
       "    \n",
       "    The default value is 120, or two minutes.\n",
       "    ``%autosave 0`` will disable autosave.\n",
       "    \n",
       "    This magic only has an effect when called from the notebook interface.\n",
       "    It has no effect when called in a startup file.\n",
       "%bookmark:\n",
       "    Manage IPython's bookmark system.\n",
       "    \n",
       "    %bookmark <name>       - set bookmark to current dir\n",
       "    %bookmark <name> <dir> - set bookmark to <dir>\n",
       "    %bookmark -l           - list all bookmarks\n",
       "    %bookmark -d <name>    - remove bookmark\n",
       "    %bookmark -r           - remove all bookmarks\n",
       "    \n",
       "    You can later on access a bookmarked folder with::\n",
       "    \n",
       "      %cd -b <name>\n",
       "    \n",
       "    or simply '%cd <name>' if there is no directory called <name> AND\n",
       "    there is such a bookmark defined.\n",
       "    \n",
       "    Your bookmarks persist through IPython sessions, but they are\n",
       "    associated with each profile.\n",
       "%cat:\n",
       "    Alias for `!cat`\n",
       "%cd:\n",
       "    Change the current working directory.\n",
       "    \n",
       "    This command automatically maintains an internal list of directories\n",
       "    you visit during your IPython session, in the variable _dh. The\n",
       "    command %dhist shows this history nicely formatted. You can also\n",
       "    do 'cd -<tab>' to see directory history conveniently.\n",
       "    \n",
       "    Usage:\n",
       "    \n",
       "      cd 'dir': changes to directory 'dir'.\n",
       "    \n",
       "      cd -: changes to the last visited directory.\n",
       "    \n",
       "      cd -<n>: changes to the n-th directory in the directory history.\n",
       "    \n",
       "      cd --foo: change to directory that matches 'foo' in history\n",
       "    \n",
       "      cd -b <bookmark_name>: jump to a bookmark set by %bookmark\n",
       "         (note: cd <bookmark_name> is enough if there is no\n",
       "          directory <bookmark_name>, but a bookmark with the name exists.)\n",
       "          'cd -b <tab>' allows you to tab-complete bookmark names.\n",
       "    \n",
       "    Options:\n",
       "    \n",
       "    -q: quiet.  Do not print the working directory after the cd command is\n",
       "    executed.  By default IPython's cd command does print this directory,\n",
       "    since the default prompts do not display path information.\n",
       "    \n",
       "    Note that !cd doesn't work for this purpose because the shell where\n",
       "    !command runs is immediately discarded after executing 'command'.\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    ::\n",
       "    \n",
       "      In [10]: cd parent/child\n",
       "      /home/tsuser/parent/child\n",
       "%clear:\n",
       "    Clear the terminal.\n",
       "%colors:\n",
       "    Switch color scheme for prompts, info system and exception handlers.\n",
       "    \n",
       "    Currently implemented schemes: NoColor, Linux, LightBG.\n",
       "    \n",
       "    Color scheme names are not case-sensitive.\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    To get a plain black and white terminal::\n",
       "    \n",
       "      %colors nocolor\n",
       "%conda:\n",
       "    Run the conda package manager within the current kernel.\n",
       "    \n",
       "    Usage:\n",
       "      %conda install [pkgs]\n",
       "%config:\n",
       "    configure IPython\n",
       "    \n",
       "        %config Class[.trait=value]\n",
       "    \n",
       "    This magic exposes most of the IPython config system. Any\n",
       "    Configurable class should be able to be configured with the simple\n",
       "    line::\n",
       "    \n",
       "        %config Class.trait=value\n",
       "    \n",
       "    Where `value` will be resolved in the user's namespace, if it is an\n",
       "    expression or variable name.\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    \n",
       "    To see what classes are available for config, pass no arguments::\n",
       "    \n",
       "        In [1]: %config\n",
       "        Available objects for config:\n",
       "            TerminalInteractiveShell\n",
       "            HistoryManager\n",
       "            PrefilterManager\n",
       "            AliasManager\n",
       "            IPCompleter\n",
       "            DisplayFormatter\n",
       "    \n",
       "    To view what is configurable on a given class, just pass the class\n",
       "    name::\n",
       "    \n",
       "        In [2]: %config IPCompleter\n",
       "        IPCompleter options\n",
       "        -----------------\n",
       "        IPCompleter.omit__names=<Enum>\n",
       "            Current: 2\n",
       "            Choices: (0, 1, 2)\n",
       "            Instruct the completer to omit private method names\n",
       "            Specifically, when completing on ``object.<tab>``.\n",
       "            When 2 [default]: all names that start with '_' will be excluded.\n",
       "            When 1: all 'magic' names (``__foo__``) will be excluded.\n",
       "            When 0: nothing will be excluded.\n",
       "        IPCompleter.merge_completions=<CBool>\n",
       "            Current: True\n",
       "            Whether to merge completion results into a single list\n",
       "            If False, only the completion results from the first non-empty\n",
       "            completer will be returned.\n",
       "        IPCompleter.limit_to__all__=<CBool>\n",
       "            Current: False\n",
       "            Instruct the completer to use __all__ for the completion\n",
       "            Specifically, when completing on ``object.<tab>``.\n",
       "            When True: only those names in obj.__all__ will be included.\n",
       "            When False [default]: the __all__ attribute is ignored\n",
       "        IPCompleter.greedy=<CBool>\n",
       "            Current: False\n",
       "            Activate greedy completion\n",
       "            This will enable completion on elements of lists, results of\n",
       "            function calls, etc., but can be unsafe because the code is\n",
       "            actually evaluated on TAB.\n",
       "    \n",
       "    but the real use is in setting values::\n",
       "    \n",
       "        In [3]: %config IPCompleter.greedy = True\n",
       "    \n",
       "    and these values are read from the user_ns if they are variables::\n",
       "    \n",
       "        In [4]: feeling_greedy=False\n",
       "    \n",
       "        In [5]: %config IPCompleter.greedy = feeling_greedy\n",
       "%connect_info:\n",
       "    Print information for connecting other clients to this kernel\n",
       "    \n",
       "    It will print the contents of this session's connection file, as well as\n",
       "    shortcuts for local clients.\n",
       "    \n",
       "    In the simplest case, when called from the most recently launched kernel,\n",
       "    secondary clients can be connected, simply with:\n",
       "    \n",
       "    $> jupyter <app> --existing\n",
       "%cp:\n",
       "    Alias for `!cp`\n",
       "%debug:\n",
       "    ::\n",
       "    \n",
       "      %debug [--breakpoint FILE:LINE] [statement [statement ...]]\n",
       "    \n",
       "    Activate the interactive debugger.\n",
       "    \n",
       "    This magic command support two ways of activating debugger.\n",
       "    One is to activate debugger before executing code.  This way, you\n",
       "    can set a break point, to step through the code from the point.\n",
       "    You can use this mode by giving statements to execute and optionally\n",
       "    a breakpoint.\n",
       "    \n",
       "    The other one is to activate debugger in post-mortem mode.  You can\n",
       "    activate this mode simply running %debug without any argument.\n",
       "    If an exception has just occurred, this lets you inspect its stack\n",
       "    frames interactively.  Note that this will always work only on the last\n",
       "    traceback that occurred, so you must call this quickly after an\n",
       "    exception that you wish to inspect has fired, because if another one\n",
       "    occurs, it clobbers the previous one.\n",
       "    \n",
       "    If you want IPython to automatically do this on every exception, see\n",
       "    the %pdb magic for more details.\n",
       "    \n",
       "    .. versionchanged:: 7.3\n",
       "        When running code, user variables are no longer expanded,\n",
       "        the magic line is always left unmodified.\n",
       "    \n",
       "    positional arguments:\n",
       "      statement             Code to run in debugger. You can omit this in cell\n",
       "                            magic mode.\n",
       "    \n",
       "    optional arguments:\n",
       "      --breakpoint <FILE:LINE>, -b <FILE:LINE>\n",
       "                            Set break point at LINE in FILE.\n",
       "%dhist:\n",
       "    Print your history of visited directories.\n",
       "    \n",
       "    %dhist       -> print full history\n",
       "    %dhist n     -> print last n entries only\n",
       "    %dhist n1 n2 -> print entries between n1 and n2 (n2 not included)\n",
       "    \n",
       "    This history is automatically maintained by the %cd command, and\n",
       "    always available as the global list variable _dh. You can use %cd -<n>\n",
       "    to go to directory number <n>.\n",
       "    \n",
       "    Note that most of time, you should view directory history by entering\n",
       "    cd -<TAB>.\n",
       "%dirs:\n",
       "    Return the current directory stack.\n",
       "%doctest_mode:\n",
       "    Toggle doctest mode on and off.\n",
       "    \n",
       "    This mode is intended to make IPython behave as much as possible like a\n",
       "    plain Python shell, from the perspective of how its prompts, exceptions\n",
       "    and output look.  This makes it easy to copy and paste parts of a\n",
       "    session into doctests.  It does so by:\n",
       "    \n",
       "    - Changing the prompts to the classic ``>>>`` ones.\n",
       "    - Changing the exception reporting mode to 'Plain'.\n",
       "    - Disabling pretty-printing of output.\n",
       "    \n",
       "    Note that IPython also supports the pasting of code snippets that have\n",
       "    leading '>>>' and '...' prompts in them.  This means that you can paste\n",
       "    doctests from files or docstrings (even if they have leading\n",
       "    whitespace), and the code will execute correctly.  You can then use\n",
       "    '%history -t' to see the translated history; this will give you the\n",
       "    input after removal of all the leading prompts and whitespace, which\n",
       "    can be pasted back into an editor.\n",
       "    \n",
       "    With these features, you can switch into this mode easily whenever you\n",
       "    need to do testing and changes to doctests, without having to leave\n",
       "    your existing IPython session.\n",
       "%ed:\n",
       "    Alias for `%edit`.\n",
       "%edit:\n",
       "    Bring up an editor and execute the resulting code.\n",
       "    \n",
       "    Usage:\n",
       "      %edit [options] [args]\n",
       "    \n",
       "    %edit runs an external text editor. You will need to set the command for\n",
       "    this editor via the ``TerminalInteractiveShell.editor`` option in your\n",
       "    configuration file before it will work.\n",
       "    \n",
       "    This command allows you to conveniently edit multi-line code right in\n",
       "    your IPython session.\n",
       "    \n",
       "    If called without arguments, %edit opens up an empty editor with a\n",
       "    temporary file and will execute the contents of this file when you\n",
       "    close it (don't forget to save it!).\n",
       "    \n",
       "    Options:\n",
       "    \n",
       "    -n <number>\n",
       "      Open the editor at a specified line number. By default, the IPython\n",
       "      editor hook uses the unix syntax 'editor +N filename', but you can\n",
       "      configure this by providing your own modified hook if your favorite\n",
       "      editor supports line-number specifications with a different syntax.\n",
       "    \n",
       "    -p\n",
       "      Call the editor with the same data as the previous time it was used,\n",
       "      regardless of how long ago (in your current session) it was.\n",
       "    \n",
       "    -r\n",
       "      Use 'raw' input. This option only applies to input taken from the\n",
       "      user's history.  By default, the 'processed' history is used, so that\n",
       "      magics are loaded in their transformed version to valid Python.  If\n",
       "      this option is given, the raw input as typed as the command line is\n",
       "      used instead.  When you exit the editor, it will be executed by\n",
       "      IPython's own processor.\n",
       "    \n",
       "    Arguments:\n",
       "    \n",
       "    If arguments are given, the following possibilities exist:\n",
       "    \n",
       "    - The arguments are numbers or pairs of colon-separated numbers (like\n",
       "      1 4:8 9). These are interpreted as lines of previous input to be\n",
       "      loaded into the editor. The syntax is the same of the %macro command.\n",
       "    \n",
       "    - If the argument doesn't start with a number, it is evaluated as a\n",
       "      variable and its contents loaded into the editor. You can thus edit\n",
       "      any string which contains python code (including the result of\n",
       "      previous edits).\n",
       "    \n",
       "    - If the argument is the name of an object (other than a string),\n",
       "      IPython will try to locate the file where it was defined and open the\n",
       "      editor at the point where it is defined. You can use ``%edit function``\n",
       "      to load an editor exactly at the point where 'function' is defined,\n",
       "      edit it and have the file be executed automatically.\n",
       "    \n",
       "      If the object is a macro (see %macro for details), this opens up your\n",
       "      specified editor with a temporary file containing the macro's data.\n",
       "      Upon exit, the macro is reloaded with the contents of the file.\n",
       "    \n",
       "      Note: opening at an exact line is only supported under Unix, and some\n",
       "      editors (like kedit and gedit up to Gnome 2.8) do not understand the\n",
       "      '+NUMBER' parameter necessary for this feature. Good editors like\n",
       "      (X)Emacs, vi, jed, pico and joe all do.\n",
       "    \n",
       "    - If the argument is not found as a variable, IPython will look for a\n",
       "      file with that name (adding .py if necessary) and load it into the\n",
       "      editor. It will execute its contents with execfile() when you exit,\n",
       "      loading any code in the file into your interactive namespace.\n",
       "    \n",
       "    Unlike in the terminal, this is designed to use a GUI editor, and we do\n",
       "    not know when it has closed. So the file you edit will not be\n",
       "    automatically executed or printed.\n",
       "    \n",
       "    Note that %edit is also available through the alias %ed.\n",
       "%env:\n",
       "    Get, set, or list environment variables.\n",
       "    \n",
       "    Usage:\n",
       "    \n",
       "      %env: lists all environment variables/values\n",
       "      %env var: get value for var\n",
       "      %env var val: set value for var\n",
       "      %env var=val: set value for var\n",
       "      %env var=$val: set value for var, using python expansion if possible\n",
       "%gui:\n",
       "    Enable or disable IPython GUI event loop integration.\n",
       "    \n",
       "    %gui [GUINAME]\n",
       "    \n",
       "    This magic replaces IPython's threaded shells that were activated\n",
       "    using the (pylab/wthread/etc.) command line flags.  GUI toolkits\n",
       "    can now be enabled at runtime and keyboard\n",
       "    interrupts should work without any problems.  The following toolkits\n",
       "    are supported:  wxPython, PyQt4, PyGTK, Tk and Cocoa (OSX)::\n",
       "    \n",
       "        %gui wx      # enable wxPython event loop integration\n",
       "        %gui qt4|qt  # enable PyQt4 event loop integration\n",
       "        %gui qt5     # enable PyQt5 event loop integration\n",
       "        %gui gtk     # enable PyGTK event loop integration\n",
       "        %gui gtk3    # enable Gtk3 event loop integration\n",
       "        %gui tk      # enable Tk event loop integration\n",
       "        %gui osx     # enable Cocoa event loop integration\n",
       "                     # (requires %matplotlib 1.1)\n",
       "        %gui         # disable all event loop integration\n",
       "    \n",
       "    WARNING:  after any of these has been called you can simply create\n",
       "    an application object, but DO NOT start the event loop yourself, as\n",
       "    we have already handled that.\n",
       "%hist:\n",
       "    Alias for `%history`.\n",
       "%history:\n",
       "    ::\n",
       "    \n",
       "      %history [-n] [-o] [-p] [-t] [-f FILENAME] [-g [PATTERN [PATTERN ...]]]\n",
       "                   [-l [LIMIT]] [-u]\n",
       "                   [range [range ...]]\n",
       "    \n",
       "    Print input history (_i<n> variables), with most recent last.\n",
       "    \n",
       "    By default, input history is printed without line numbers so it can be\n",
       "    directly pasted into an editor. Use -n to show them.\n",
       "    \n",
       "    By default, all input history from the current session is displayed.\n",
       "    Ranges of history can be indicated using the syntax:\n",
       "    \n",
       "    ``4``\n",
       "        Line 4, current session\n",
       "    ``4-6``\n",
       "        Lines 4-6, current session\n",
       "    ``243/1-5``\n",
       "        Lines 1-5, session 243\n",
       "    ``~2/7``\n",
       "        Line 7, session 2 before current\n",
       "    ``~8/1-~6/5``\n",
       "        From the first line of 8 sessions ago, to the fifth line of 6\n",
       "        sessions ago.\n",
       "    \n",
       "    Multiple ranges can be entered, separated by spaces\n",
       "    \n",
       "    The same syntax is used by %macro, %save, %edit, %rerun\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    ::\n",
       "    \n",
       "      In [6]: %history -n 4-6\n",
       "      4:a = 12\n",
       "      5:print a**2\n",
       "      6:%history -n 4-6\n",
       "    \n",
       "    positional arguments:\n",
       "      range\n",
       "    \n",
       "    optional arguments:\n",
       "      -n                    print line numbers for each input. This feature is\n",
       "                            only available if numbered prompts are in use.\n",
       "      -o                    also print outputs for each input.\n",
       "      -p                    print classic '>>>' python prompts before each input.\n",
       "                            This is useful for making documentation, and in\n",
       "                            conjunction with -o, for producing doctest-ready\n",
       "                            output.\n",
       "      -t                    print the 'translated' history, as IPython understands\n",
       "                            it. IPython filters your input and converts it all\n",
       "                            into valid Python source before executing it (things\n",
       "                            like magics or aliases are turned into function calls,\n",
       "                            for example). With this option, you'll see the native\n",
       "                            history instead of the user-entered version: '%cd /'\n",
       "                            will be seen as 'get_ipython().run_line_magic(\"cd\",\n",
       "                            \"/\")' instead of '%cd /'.\n",
       "      -f FILENAME           FILENAME: instead of printing the output to the\n",
       "                            screen, redirect it to the given file. The file is\n",
       "                            always overwritten, though *when it can*, IPython asks\n",
       "                            for confirmation first. In particular, running the\n",
       "                            command 'history -f FILENAME' from the IPython\n",
       "                            Notebook interface will replace FILENAME even if it\n",
       "                            already exists *without* confirmation.\n",
       "      -g <[PATTERN [PATTERN ...]]>\n",
       "                            treat the arg as a glob pattern to search for in\n",
       "                            (full) history. This includes the saved history\n",
       "                            (almost all commands ever written). The pattern may\n",
       "                            contain '?' to match one unknown character and '*' to\n",
       "                            match any number of unknown characters. Use '%hist -g'\n",
       "                            to show full saved history (may be very long).\n",
       "      -l <[LIMIT]>          get the last n lines from all sessions. Specify n as a\n",
       "                            single arg, or the default is the last 10 lines.\n",
       "      -u                    when searching history using `-g`, show only unique\n",
       "                            history.\n",
       "%killbgscripts:\n",
       "    Kill all BG processes started by %%script and its family.\n",
       "%ldir:\n",
       "    Alias for `!ls -F -o --color %l | grep /$`\n",
       "%less:\n",
       "    Show a file through the pager.\n",
       "    \n",
       "    Files ending in .py are syntax-highlighted.\n",
       "%lf:\n",
       "    Alias for `!ls -F -o --color %l | grep ^-`\n",
       "%lk:\n",
       "    Alias for `!ls -F -o --color %l | grep ^l`\n",
       "%ll:\n",
       "    Alias for `!ls -F -o --color`\n",
       "%load:\n",
       "    Load code into the current frontend.\n",
       "    \n",
       "    Usage:\n",
       "      %load [options] source\n",
       "    \n",
       "      where source can be a filename, URL, input history range, macro, or\n",
       "      element in the user namespace\n",
       "    \n",
       "    Options:\n",
       "    \n",
       "      -r <lines>: Specify lines or ranges of lines to load from the source.\n",
       "      Ranges could be specified as x-y (x..y) or in python-style x:y \n",
       "      (x..(y-1)). Both limits x and y can be left blank (meaning the \n",
       "      beginning and end of the file, respectively).\n",
       "    \n",
       "      -s <symbols>: Specify function or classes to load from python source. \n",
       "    \n",
       "      -y : Don't ask confirmation for loading source above 200 000 characters.\n",
       "    \n",
       "      -n : Include the user's namespace when searching for source code.\n",
       "    \n",
       "    This magic command can either take a local filename, a URL, an history\n",
       "    range (see %history) or a macro as argument, it will prompt for\n",
       "    confirmation before loading source with more than 200 000 characters, unless\n",
       "    -y flag is passed or if the frontend does not support raw_input::\n",
       "    \n",
       "    %load myscript.py\n",
       "    %load 7-27\n",
       "    %load myMacro\n",
       "    %load http://www.example.com/myscript.py\n",
       "    %load -r 5-10 myscript.py\n",
       "    %load -r 10-20,30,40: foo.py\n",
       "    %load -s MyClass,wonder_function myscript.py\n",
       "    %load -n MyClass\n",
       "    %load -n my_module.wonder_function\n",
       "%load_ext:\n",
       "    Load an IPython extension by its module name.\n",
       "%loadpy:\n",
       "    Alias of `%load`\n",
       "    \n",
       "    `%loadpy` has gained some flexibility and dropped the requirement of a `.py`\n",
       "    extension. So it has been renamed simply into %load. You can look at\n",
       "    `%load`'s docstring for more info.\n",
       "%logoff:\n",
       "    Temporarily stop logging.\n",
       "    \n",
       "    You must have previously started logging.\n",
       "%logon:\n",
       "    Restart logging.\n",
       "    \n",
       "    This function is for restarting logging which you've temporarily\n",
       "    stopped with %logoff. For starting logging for the first time, you\n",
       "    must use the %logstart function, which allows you to specify an\n",
       "    optional log filename.\n",
       "%logstart:\n",
       "    Start logging anywhere in a session.\n",
       "    \n",
       "    %logstart [-o|-r|-t|-q] [log_name [log_mode]]\n",
       "    \n",
       "    If no name is given, it defaults to a file named 'ipython_log.py' in your\n",
       "    current directory, in 'rotate' mode (see below).\n",
       "    \n",
       "    '%logstart name' saves to file 'name' in 'backup' mode.  It saves your\n",
       "    history up to that point and then continues logging.\n",
       "    \n",
       "    %logstart takes a second optional parameter: logging mode. This can be one\n",
       "    of (note that the modes are given unquoted):\n",
       "    \n",
       "    append\n",
       "        Keep logging at the end of any existing file.\n",
       "    \n",
       "    backup\n",
       "        Rename any existing file to name~ and start name.\n",
       "    \n",
       "    global\n",
       "        Append to  a single logfile in your home directory.\n",
       "    \n",
       "    over\n",
       "        Overwrite any existing log.\n",
       "    \n",
       "    rotate\n",
       "        Create rotating logs: name.1~, name.2~, etc.\n",
       "    \n",
       "    Options:\n",
       "    \n",
       "      -o\n",
       "        log also IPython's output. In this mode, all commands which\n",
       "        generate an Out[NN] prompt are recorded to the logfile, right after\n",
       "        their corresponding input line. The output lines are always\n",
       "        prepended with a '#[Out]# ' marker, so that the log remains valid\n",
       "        Python code.\n",
       "    \n",
       "      Since this marker is always the same, filtering only the output from\n",
       "      a log is very easy, using for example a simple awk call::\n",
       "    \n",
       "        awk -F'#\\[Out\\]# ' '{if($2) {print $2}}' ipython_log.py\n",
       "    \n",
       "      -r\n",
       "        log 'raw' input.  Normally, IPython's logs contain the processed\n",
       "        input, so that user lines are logged in their final form, converted\n",
       "        into valid Python.  For example, %Exit is logged as\n",
       "        _ip.magic(\"Exit\").  If the -r flag is given, all input is logged\n",
       "        exactly as typed, with no transformations applied.\n",
       "    \n",
       "      -t\n",
       "        put timestamps before each input line logged (these are put in\n",
       "        comments).\n",
       "    \n",
       "      -q \n",
       "        suppress output of logstate message when logging is invoked\n",
       "%logstate:\n",
       "    Print the status of the logging system.\n",
       "%logstop:\n",
       "    Fully stop logging and close log file.\n",
       "    \n",
       "    In order to start logging again, a new %logstart call needs to be made,\n",
       "    possibly (though not necessarily) with a new filename, mode and other\n",
       "    options.\n",
       "%ls:\n",
       "    Alias for `!ls -F --color`\n",
       "%lsmagic:\n",
       "    List currently available magic functions.\n",
       "%lx:\n",
       "    Alias for `!ls -F -o --color %l | grep ^-..x`\n",
       "%macro:\n",
       "    Define a macro for future re-execution. It accepts ranges of history,\n",
       "    filenames or string objects.\n",
       "    \n",
       "    Usage:\n",
       "      %macro [options] name n1-n2 n3-n4 ... n5 .. n6 ...\n",
       "    \n",
       "    Options:\n",
       "    \n",
       "      -r: use 'raw' input.  By default, the 'processed' history is used,\n",
       "      so that magics are loaded in their transformed version to valid\n",
       "      Python.  If this option is given, the raw input as typed at the\n",
       "      command line is used instead.\n",
       "    \n",
       "      -q: quiet macro definition.  By default, a tag line is printed \n",
       "      to indicate the macro has been created, and then the contents of \n",
       "      the macro are printed.  If this option is given, then no printout\n",
       "      is produced once the macro is created.\n",
       "    \n",
       "    This will define a global variable called `name` which is a string\n",
       "    made of joining the slices and lines you specify (n1,n2,... numbers\n",
       "    above) from your input history into a single string. This variable\n",
       "    acts like an automatic function which re-executes those lines as if\n",
       "    you had typed them. You just type 'name' at the prompt and the code\n",
       "    executes.\n",
       "    \n",
       "    The syntax for indicating input ranges is described in %history.\n",
       "    \n",
       "    Note: as a 'hidden' feature, you can also use traditional python slice\n",
       "    notation, where N:M means numbers N through M-1.\n",
       "    \n",
       "    For example, if your history contains (print using %hist -n )::\n",
       "    \n",
       "      44: x=1\n",
       "      45: y=3\n",
       "      46: z=x+y\n",
       "      47: print x\n",
       "      48: a=5\n",
       "      49: print 'x',x,'y',y\n",
       "    \n",
       "    you can create a macro with lines 44 through 47 (included) and line 49\n",
       "    called my_macro with::\n",
       "    \n",
       "      In [55]: %macro my_macro 44-47 49\n",
       "    \n",
       "    Now, typing `my_macro` (without quotes) will re-execute all this code\n",
       "    in one pass.\n",
       "    \n",
       "    You don't need to give the line-numbers in order, and any given line\n",
       "    number can appear multiple times. You can assemble macros with any\n",
       "    lines from your input history in any order.\n",
       "    \n",
       "    The macro is a simple object which holds its value in an attribute,\n",
       "    but IPython's display system checks for macros and executes them as\n",
       "    code instead of printing them when you type their name.\n",
       "    \n",
       "    You can view a macro's contents by explicitly printing it with::\n",
       "    \n",
       "      print macro_name\n",
       "%magic:\n",
       "    Print information about the magic function system.\n",
       "    \n",
       "    Supported formats: -latex, -brief, -rest\n",
       "%man:\n",
       "    Find the man page for the given command and display in pager.\n",
       "%matplotlib:\n",
       "    ::\n",
       "    \n",
       "      %matplotlib [-l] [gui]\n",
       "    \n",
       "    Set up matplotlib to work interactively.\n",
       "    \n",
       "    This function lets you activate matplotlib interactive support\n",
       "    at any point during an IPython session. It does not import anything\n",
       "    into the interactive namespace.\n",
       "    \n",
       "    If you are using the inline matplotlib backend in the IPython Notebook\n",
       "    you can set which figure formats are enabled using the following::\n",
       "    \n",
       "        In [1]: from IPython.display import set_matplotlib_formats\n",
       "    \n",
       "        In [2]: set_matplotlib_formats('pdf', 'svg')\n",
       "    \n",
       "    The default for inline figures sets `bbox_inches` to 'tight'. This can\n",
       "    cause discrepancies between the displayed image and the identical\n",
       "    image created using `savefig`. This behavior can be disabled using the\n",
       "    `%config` magic::\n",
       "    \n",
       "        In [3]: %config InlineBackend.print_figure_kwargs = {'bbox_inches':None}\n",
       "    \n",
       "    In addition, see the docstring of\n",
       "    `IPython.display.set_matplotlib_formats` and\n",
       "    `IPython.display.set_matplotlib_close` for more information on\n",
       "    changing additional behaviors of the inline backend.\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    To enable the inline backend for usage with the IPython Notebook::\n",
       "    \n",
       "        In [1]: %matplotlib inline\n",
       "    \n",
       "    In this case, where the matplotlib default is TkAgg::\n",
       "    \n",
       "        In [2]: %matplotlib\n",
       "        Using matplotlib backend: TkAgg\n",
       "    \n",
       "    But you can explicitly request a different GUI backend::\n",
       "    \n",
       "        In [3]: %matplotlib qt\n",
       "    \n",
       "    You can list the available backends using the -l/--list option::\n",
       "    \n",
       "       In [4]: %matplotlib --list\n",
       "       Available matplotlib backends: ['osx', 'qt4', 'qt5', 'gtk3', 'notebook', 'wx', 'qt', 'nbagg',\n",
       "       'gtk', 'tk', 'inline']\n",
       "    \n",
       "    positional arguments:\n",
       "      gui         Name of the matplotlib backend to use ('agg', 'gtk', 'gtk3',\n",
       "                  'inline', 'ipympl', 'nbagg', 'notebook', 'osx', 'pdf', 'ps',\n",
       "                  'qt', 'qt4', 'qt5', 'svg', 'tk', 'widget', 'wx'). If given, the\n",
       "                  corresponding matplotlib backend is used, otherwise it will be\n",
       "                  matplotlib's default (which you can set in your matplotlib\n",
       "                  config file).\n",
       "    \n",
       "    optional arguments:\n",
       "      -l, --list  Show available matplotlib backends\n",
       "%mkdir:\n",
       "    Alias for `!mkdir`\n",
       "%more:\n",
       "    Show a file through the pager.\n",
       "    \n",
       "    Files ending in .py are syntax-highlighted.\n",
       "%mv:\n",
       "    Alias for `!mv`\n",
       "%notebook:\n",
       "    ::\n",
       "    \n",
       "      %notebook filename\n",
       "    \n",
       "    Export and convert IPython notebooks.\n",
       "    \n",
       "    This function can export the current IPython history to a notebook file.\n",
       "    For example, to export the history to \"foo.ipynb\" do \"%notebook foo.ipynb\".\n",
       "    \n",
       "    The -e or --export flag is deprecated in IPython 5.2, and will be\n",
       "    removed in the future.\n",
       "    \n",
       "    positional arguments:\n",
       "      filename  Notebook name or filename\n",
       "%page:\n",
       "    Pretty print the object and display it through a pager.\n",
       "    \n",
       "    %page [options] OBJECT\n",
       "    \n",
       "    If no object is given, use _ (last output).\n",
       "    \n",
       "    Options:\n",
       "    \n",
       "      -r: page str(object), don't pretty-print it.\n",
       "%pastebin:\n",
       "    Upload code to dpaste's paste bin, returning the URL.\n",
       "    \n",
       "    Usage:\n",
       "      %pastebin [-d \"Custom description\"] 1-7\n",
       "    \n",
       "    The argument can be an input history range, a filename, or the name of a\n",
       "    string or macro.\n",
       "    \n",
       "    Options:\n",
       "    \n",
       "      -d: Pass a custom description for the gist. The default will say\n",
       "          \"Pasted from IPython\".\n",
       "%pdb:\n",
       "    Control the automatic calling of the pdb interactive debugger.\n",
       "    \n",
       "    Call as '%pdb on', '%pdb 1', '%pdb off' or '%pdb 0'. If called without\n",
       "    argument it works as a toggle.\n",
       "    \n",
       "    When an exception is triggered, IPython can optionally call the\n",
       "    interactive pdb debugger after the traceback printout. %pdb toggles\n",
       "    this feature on and off.\n",
       "    \n",
       "    The initial state of this feature is set in your configuration\n",
       "    file (the option is ``InteractiveShell.pdb``).\n",
       "    \n",
       "    If you want to just activate the debugger AFTER an exception has fired,\n",
       "    without having to type '%pdb on' and rerunning your code, you can use\n",
       "    the %debug magic.\n",
       "%pdef:\n",
       "    Print the call signature for any callable object.\n",
       "    \n",
       "    If the object is a class, print the constructor information.\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    ::\n",
       "    \n",
       "      In [3]: %pdef urllib.urlopen\n",
       "      urllib.urlopen(url, data=None, proxies=None)\n",
       "%pdoc:\n",
       "    Print the docstring for an object.\n",
       "    \n",
       "    If the given object is a class, it will print both the class and the\n",
       "    constructor docstrings.\n",
       "%pfile:\n",
       "    Print (or run through pager) the file where an object is defined.\n",
       "    \n",
       "    The file opens at the line where the object definition begins. IPython\n",
       "    will honor the environment variable PAGER if set, and otherwise will\n",
       "    do its best to print the file in a convenient form.\n",
       "    \n",
       "    If the given argument is not an object currently defined, IPython will\n",
       "    try to interpret it as a filename (automatically adding a .py extension\n",
       "    if needed). You can thus use %pfile as a syntax highlighting code\n",
       "    viewer.\n",
       "%pinfo:\n",
       "    Provide detailed information about an object.\n",
       "    \n",
       "    '%pinfo object' is just a synonym for object? or ?object.\n",
       "%pinfo2:\n",
       "    Provide extra detailed information about an object.\n",
       "    \n",
       "    '%pinfo2 object' is just a synonym for object?? or ??object.\n",
       "%pip:\n",
       "    Run the pip package manager within the current kernel.\n",
       "    \n",
       "    Usage:\n",
       "      %pip install [pkgs]\n",
       "%popd:\n",
       "    Change to directory popped off the top of the stack.\n",
       "%pprint:\n",
       "    Toggle pretty printing on/off.\n",
       "%precision:\n",
       "    Set floating point precision for pretty printing.\n",
       "    \n",
       "    Can set either integer precision or a format string.\n",
       "    \n",
       "    If numpy has been imported and precision is an int,\n",
       "    numpy display precision will also be set, via ``numpy.set_printoptions``.\n",
       "    \n",
       "    If no argument is given, defaults will be restored.\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    ::\n",
       "    \n",
       "        In [1]: from math import pi\n",
       "    \n",
       "        In [2]: %precision 3\n",
       "        Out[2]: u'%.3f'\n",
       "    \n",
       "        In [3]: pi\n",
       "        Out[3]: 3.142\n",
       "    \n",
       "        In [4]: %precision %i\n",
       "        Out[4]: u'%i'\n",
       "    \n",
       "        In [5]: pi\n",
       "        Out[5]: 3\n",
       "    \n",
       "        In [6]: %precision %e\n",
       "        Out[6]: u'%e'\n",
       "    \n",
       "        In [7]: pi**10\n",
       "        Out[7]: 9.364805e+04\n",
       "    \n",
       "        In [8]: %precision\n",
       "        Out[8]: u'%r'\n",
       "    \n",
       "        In [9]: pi**10\n",
       "        Out[9]: 93648.047476082982\n",
       "%prun:\n",
       "    Run a statement through the python code profiler.\n",
       "    \n",
       "    Usage, in line mode:\n",
       "      %prun [options] statement\n",
       "    \n",
       "    Usage, in cell mode:\n",
       "      %%prun [options] [statement]\n",
       "      code...\n",
       "      code...\n",
       "    \n",
       "    In cell mode, the additional code lines are appended to the (possibly\n",
       "    empty) statement in the first line.  Cell mode allows you to easily\n",
       "    profile multiline blocks without having to put them in a separate\n",
       "    function.\n",
       "    \n",
       "    The given statement (which doesn't require quote marks) is run via the\n",
       "    python profiler in a manner similar to the profile.run() function.\n",
       "    Namespaces are internally managed to work correctly; profile.run\n",
       "    cannot be used in IPython because it makes certain assumptions about\n",
       "    namespaces which do not hold under IPython.\n",
       "    \n",
       "    Options:\n",
       "    \n",
       "    -l <limit>\n",
       "      you can place restrictions on what or how much of the\n",
       "      profile gets printed. The limit value can be:\n",
       "    \n",
       "         * A string: only information for function names containing this string\n",
       "           is printed.\n",
       "    \n",
       "         * An integer: only these many lines are printed.\n",
       "    \n",
       "         * A float (between 0 and 1): this fraction of the report is printed\n",
       "           (for example, use a limit of 0.4 to see the topmost 40% only).\n",
       "    \n",
       "      You can combine several limits with repeated use of the option. For\n",
       "      example, ``-l __init__ -l 5`` will print only the topmost 5 lines of\n",
       "      information about class constructors.\n",
       "    \n",
       "    -r\n",
       "      return the pstats.Stats object generated by the profiling. This\n",
       "      object has all the information about the profile in it, and you can\n",
       "      later use it for further analysis or in other functions.\n",
       "    \n",
       "    -s <key>\n",
       "      sort profile by given key. You can provide more than one key\n",
       "      by using the option several times: '-s key1 -s key2 -s key3...'. The\n",
       "      default sorting key is 'time'.\n",
       "    \n",
       "      The following is copied verbatim from the profile documentation\n",
       "      referenced below:\n",
       "    \n",
       "      When more than one key is provided, additional keys are used as\n",
       "      secondary criteria when the there is equality in all keys selected\n",
       "      before them.\n",
       "    \n",
       "      Abbreviations can be used for any key names, as long as the\n",
       "      abbreviation is unambiguous.  The following are the keys currently\n",
       "      defined:\n",
       "    \n",
       "      ============  =====================\n",
       "      Valid Arg     Meaning\n",
       "      ============  =====================\n",
       "      \"calls\"       call count\n",
       "      \"cumulative\"  cumulative time\n",
       "      \"file\"        file name\n",
       "      \"module\"      file name\n",
       "      \"pcalls\"      primitive call count\n",
       "      \"line\"        line number\n",
       "      \"name\"        function name\n",
       "      \"nfl\"         name/file/line\n",
       "      \"stdname\"     standard name\n",
       "      \"time\"        internal time\n",
       "      ============  =====================\n",
       "    \n",
       "      Note that all sorts on statistics are in descending order (placing\n",
       "      most time consuming items first), where as name, file, and line number\n",
       "      searches are in ascending order (i.e., alphabetical). The subtle\n",
       "      distinction between \"nfl\" and \"stdname\" is that the standard name is a\n",
       "      sort of the name as printed, which means that the embedded line\n",
       "      numbers get compared in an odd way.  For example, lines 3, 20, and 40\n",
       "      would (if the file names were the same) appear in the string order\n",
       "      \"20\" \"3\" and \"40\".  In contrast, \"nfl\" does a numeric compare of the\n",
       "      line numbers.  In fact, sort_stats(\"nfl\") is the same as\n",
       "      sort_stats(\"name\", \"file\", \"line\").\n",
       "    \n",
       "    -T <filename>\n",
       "      save profile results as shown on screen to a text\n",
       "      file. The profile is still shown on screen.\n",
       "    \n",
       "    -D <filename>\n",
       "      save (via dump_stats) profile statistics to given\n",
       "      filename. This data is in a format understood by the pstats module, and\n",
       "      is generated by a call to the dump_stats() method of profile\n",
       "      objects. The profile is still shown on screen.\n",
       "    \n",
       "    -q\n",
       "      suppress output to the pager.  Best used with -T and/or -D above.\n",
       "    \n",
       "    If you want to run complete programs under the profiler's control, use\n",
       "    ``%run -p [prof_opts] filename.py [args to program]`` where prof_opts\n",
       "    contains profiler specific options as described here.\n",
       "    \n",
       "    You can read the complete documentation for the profile module with::\n",
       "    \n",
       "      In [1]: import profile; profile.help()\n",
       "    \n",
       "    .. versionchanged:: 7.3\n",
       "        User variables are no longer expanded,\n",
       "        the magic line is always left unmodified.\n",
       "%psearch:\n",
       "    Search for object in namespaces by wildcard.\n",
       "    \n",
       "    %psearch [options] PATTERN [OBJECT TYPE]\n",
       "    \n",
       "    Note: ? can be used as a synonym for %psearch, at the beginning or at\n",
       "    the end: both a*? and ?a* are equivalent to '%psearch a*'.  Still, the\n",
       "    rest of the command line must be unchanged (options come first), so\n",
       "    for example the following forms are equivalent\n",
       "    \n",
       "    %psearch -i a* function\n",
       "    -i a* function?\n",
       "    ?-i a* function\n",
       "    \n",
       "    Arguments:\n",
       "    \n",
       "      PATTERN\n",
       "    \n",
       "      where PATTERN is a string containing * as a wildcard similar to its\n",
       "      use in a shell.  The pattern is matched in all namespaces on the\n",
       "      search path. By default objects starting with a single _ are not\n",
       "      matched, many IPython generated objects have a single\n",
       "      underscore. The default is case insensitive matching. Matching is\n",
       "      also done on the attributes of objects and not only on the objects\n",
       "      in a module.\n",
       "    \n",
       "      [OBJECT TYPE]\n",
       "    \n",
       "      Is the name of a python type from the types module. The name is\n",
       "      given in lowercase without the ending type, ex. StringType is\n",
       "      written string. By adding a type here only objects matching the\n",
       "      given type are matched. Using all here makes the pattern match all\n",
       "      types (this is the default).\n",
       "    \n",
       "    Options:\n",
       "    \n",
       "      -a: makes the pattern match even objects whose names start with a\n",
       "      single underscore.  These names are normally omitted from the\n",
       "      search.\n",
       "    \n",
       "      -i/-c: make the pattern case insensitive/sensitive.  If neither of\n",
       "      these options are given, the default is read from your configuration\n",
       "      file, with the option ``InteractiveShell.wildcards_case_sensitive``.\n",
       "      If this option is not specified in your configuration file, IPython's\n",
       "      internal default is to do a case sensitive search.\n",
       "    \n",
       "      -e/-s NAMESPACE: exclude/search a given namespace.  The pattern you\n",
       "      specify can be searched in any of the following namespaces:\n",
       "      'builtin', 'user', 'user_global','internal', 'alias', where\n",
       "      'builtin' and 'user' are the search defaults.  Note that you should\n",
       "      not use quotes when specifying namespaces.\n",
       "    \n",
       "      -l: List all available object types for object matching. This function\n",
       "      can be used without arguments.\n",
       "    \n",
       "      'Builtin' contains the python module builtin, 'user' contains all\n",
       "      user data, 'alias' only contain the shell aliases and no python\n",
       "      objects, 'internal' contains objects used by IPython.  The\n",
       "      'user_global' namespace is only used by embedded IPython instances,\n",
       "      and it contains module-level globals.  You can add namespaces to the\n",
       "      search with -s or exclude them with -e (these options can be given\n",
       "      more than once).\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    ::\n",
       "    \n",
       "      %psearch a*            -> objects beginning with an a\n",
       "      %psearch -e builtin a* -> objects NOT in the builtin space starting in a\n",
       "      %psearch a* function   -> all functions beginning with an a\n",
       "      %psearch re.e*         -> objects beginning with an e in module re\n",
       "      %psearch r*.e*         -> objects that start with e in modules starting in r\n",
       "      %psearch r*.* string   -> all strings in modules beginning with r\n",
       "    \n",
       "    Case sensitive search::\n",
       "    \n",
       "      %psearch -c a*         list all object beginning with lower case a\n",
       "    \n",
       "    Show objects beginning with a single _::\n",
       "    \n",
       "      %psearch -a _*         list objects beginning with a single underscore\n",
       "    \n",
       "    List available objects::\n",
       "    \n",
       "      %psearch -l            list all available object types\n",
       "%psource:\n",
       "    Print (or run through pager) the source code for an object.\n",
       "%pushd:\n",
       "    Place the current dir on stack and change directory.\n",
       "    \n",
       "    Usage:\n",
       "      %pushd ['dirname']\n",
       "%pwd:\n",
       "    Return the current working directory path.\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    ::\n",
       "    \n",
       "      In [9]: pwd\n",
       "      Out[9]: '/home/tsuser/sprint/ipython'\n",
       "%pycat:\n",
       "    Show a syntax-highlighted file through a pager.\n",
       "    \n",
       "    This magic is similar to the cat utility, but it will assume the file\n",
       "    to be Python source and will show it with syntax highlighting.\n",
       "    \n",
       "    This magic command can either take a local filename, an url,\n",
       "    an history range (see %history) or a macro as argument ::\n",
       "    \n",
       "    %pycat myscript.py\n",
       "    %pycat 7-27\n",
       "    %pycat myMacro\n",
       "    %pycat http://www.example.com/myscript.py\n",
       "%pylab:\n",
       "    ::\n",
       "    \n",
       "      %pylab [--no-import-all] [gui]\n",
       "    \n",
       "    Load numpy and matplotlib to work interactively.\n",
       "    \n",
       "    This function lets you activate pylab (matplotlib, numpy and\n",
       "    interactive support) at any point during an IPython session.\n",
       "    \n",
       "    %pylab makes the following imports::\n",
       "    \n",
       "        import numpy\n",
       "        import matplotlib\n",
       "        from matplotlib import pylab, mlab, pyplot\n",
       "        np = numpy\n",
       "        plt = pyplot\n",
       "    \n",
       "        from IPython.display import display\n",
       "        from IPython.core.pylabtools import figsize, getfigs\n",
       "    \n",
       "        from pylab import *\n",
       "        from numpy import *\n",
       "    \n",
       "    If you pass `--no-import-all`, the last two `*` imports will be excluded.\n",
       "    \n",
       "    See the %matplotlib magic for more details about activating matplotlib\n",
       "    without affecting the interactive namespace.\n",
       "    \n",
       "    positional arguments:\n",
       "      gui              Name of the matplotlib backend to use ('agg', 'gtk',\n",
       "                       'gtk3', 'inline', 'ipympl', 'nbagg', 'notebook', 'osx',\n",
       "                       'pdf', 'ps', 'qt', 'qt4', 'qt5', 'svg', 'tk', 'widget',\n",
       "                       'wx'). If given, the corresponding matplotlib backend is\n",
       "                       used, otherwise it will be matplotlib's default (which you\n",
       "                       can set in your matplotlib config file).\n",
       "    \n",
       "    optional arguments:\n",
       "      --no-import-all  Prevent IPython from performing ``import *`` into the\n",
       "                       interactive namespace. You can govern the default behavior\n",
       "                       of this flag with the InteractiveShellApp.pylab_import_all\n",
       "                       configurable.\n",
       "%qtconsole:\n",
       "    Open a qtconsole connected to this kernel.\n",
       "    \n",
       "    Useful for connecting a qtconsole to running notebooks, for better\n",
       "    debugging.\n",
       "%quickref:\n",
       "    Show a quick reference sheet\n",
       "%recall:\n",
       "    Repeat a command, or get command to input line for editing.\n",
       "    \n",
       "    %recall and %rep are equivalent.\n",
       "    \n",
       "    - %recall (no arguments):\n",
       "    \n",
       "    Place a string version of last computation result (stored in the\n",
       "    special '_' variable) to the next input prompt. Allows you to create\n",
       "    elaborate command lines without using copy-paste::\n",
       "    \n",
       "         In[1]: l = [\"hei\", \"vaan\"]\n",
       "         In[2]: \"\".join(l)\n",
       "        Out[2]: heivaan\n",
       "         In[3]: %recall\n",
       "         In[4]: heivaan_ <== cursor blinking\n",
       "    \n",
       "    %recall 45\n",
       "    \n",
       "    Place history line 45 on the next input prompt. Use %hist to find\n",
       "    out the number.\n",
       "    \n",
       "    %recall 1-4\n",
       "    \n",
       "    Combine the specified lines into one cell, and place it on the next\n",
       "    input prompt. See %history for the slice syntax.\n",
       "    \n",
       "    %recall foo+bar\n",
       "    \n",
       "    If foo+bar can be evaluated in the user namespace, the result is\n",
       "    placed at the next input prompt. Otherwise, the history is searched\n",
       "    for lines which contain that substring, and the most recent one is\n",
       "    placed at the next input prompt.\n",
       "%rehashx:\n",
       "    Update the alias table with all executable files in $PATH.\n",
       "    \n",
       "    rehashx explicitly checks that every entry in $PATH is a file\n",
       "    with execute access (os.X_OK).\n",
       "    \n",
       "    Under Windows, it checks executability as a match against a\n",
       "    '|'-separated string of extensions, stored in the IPython config\n",
       "    variable win_exec_ext.  This defaults to 'exe|com|bat'.\n",
       "    \n",
       "    This function also resets the root module cache of module completer,\n",
       "    used on slow filesystems.\n",
       "%reload_ext:\n",
       "    Reload an IPython extension by its module name.\n",
       "%rep:\n",
       "    Alias for `%recall`.\n",
       "%rerun:\n",
       "    Re-run previous input\n",
       "    \n",
       "    By default, you can specify ranges of input history to be repeated\n",
       "    (as with %history). With no arguments, it will repeat the last line.\n",
       "    \n",
       "    Options:\n",
       "    \n",
       "      -l <n> : Repeat the last n lines of input, not including the\n",
       "      current command.\n",
       "    \n",
       "      -g foo : Repeat the most recent line which contains foo\n",
       "%reset:\n",
       "    Resets the namespace by removing all names defined by the user, if\n",
       "    called without arguments, or by removing some types of objects, such\n",
       "    as everything currently in IPython's In[] and Out[] containers (see\n",
       "    the parameters for details).\n",
       "    \n",
       "    Parameters\n",
       "    ----------\n",
       "    -f : force reset without asking for confirmation.\n",
       "    \n",
       "    -s : 'Soft' reset: Only clears your namespace, leaving history intact.\n",
       "        References to objects may be kept. By default (without this option),\n",
       "        we do a 'hard' reset, giving you a new session and removing all\n",
       "        references to objects from the current session.\n",
       "    \n",
       "    in : reset input history\n",
       "    \n",
       "    out : reset output history\n",
       "    \n",
       "    dhist : reset directory history\n",
       "    \n",
       "    array : reset only variables that are NumPy arrays\n",
       "    \n",
       "    See Also\n",
       "    --------\n",
       "    reset_selective : invoked as ``%reset_selective``\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    ::\n",
       "    \n",
       "      In [6]: a = 1\n",
       "    \n",
       "      In [7]: a\n",
       "      Out[7]: 1\n",
       "    \n",
       "      In [8]: 'a' in get_ipython().user_ns\n",
       "      Out[8]: True\n",
       "    \n",
       "      In [9]: %reset -f\n",
       "    \n",
       "      In [1]: 'a' in get_ipython().user_ns\n",
       "      Out[1]: False\n",
       "    \n",
       "      In [2]: %reset -f in\n",
       "      Flushing input history\n",
       "    \n",
       "      In [3]: %reset -f dhist in\n",
       "      Flushing directory history\n",
       "      Flushing input history\n",
       "    \n",
       "    Notes\n",
       "    -----\n",
       "    Calling this magic from clients that do not implement standard input,\n",
       "    such as the ipython notebook interface, will reset the namespace\n",
       "    without confirmation.\n",
       "%reset_selective:\n",
       "    Resets the namespace by removing names defined by the user.\n",
       "    \n",
       "    Input/Output history are left around in case you need them.\n",
       "    \n",
       "    %reset_selective [-f] regex\n",
       "    \n",
       "    No action is taken if regex is not included\n",
       "    \n",
       "    Options\n",
       "      -f : force reset without asking for confirmation.\n",
       "    \n",
       "    See Also\n",
       "    --------\n",
       "    reset : invoked as ``%reset``\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    \n",
       "    We first fully reset the namespace so your output looks identical to\n",
       "    this example for pedagogical reasons; in practice you do not need a\n",
       "    full reset::\n",
       "    \n",
       "      In [1]: %reset -f\n",
       "    \n",
       "    Now, with a clean namespace we can make a few variables and use\n",
       "    ``%reset_selective`` to only delete names that match our regexp::\n",
       "    \n",
       "      In [2]: a=1; b=2; c=3; b1m=4; b2m=5; b3m=6; b4m=7; b2s=8\n",
       "    \n",
       "      In [3]: who_ls\n",
       "      Out[3]: ['a', 'b', 'b1m', 'b2m', 'b2s', 'b3m', 'b4m', 'c']\n",
       "    \n",
       "      In [4]: %reset_selective -f b[2-3]m\n",
       "    \n",
       "      In [5]: who_ls\n",
       "      Out[5]: ['a', 'b', 'b1m', 'b2s', 'b4m', 'c']\n",
       "    \n",
       "      In [6]: %reset_selective -f d\n",
       "    \n",
       "      In [7]: who_ls\n",
       "      Out[7]: ['a', 'b', 'b1m', 'b2s', 'b4m', 'c']\n",
       "    \n",
       "      In [8]: %reset_selective -f c\n",
       "    \n",
       "      In [9]: who_ls\n",
       "      Out[9]: ['a', 'b', 'b1m', 'b2s', 'b4m']\n",
       "    \n",
       "      In [10]: %reset_selective -f b\n",
       "    \n",
       "      In [11]: who_ls\n",
       "      Out[11]: ['a']\n",
       "    \n",
       "    Notes\n",
       "    -----\n",
       "    Calling this magic from clients that do not implement standard input,\n",
       "    such as the ipython notebook interface, will reset the namespace\n",
       "    without confirmation.\n",
       "%rm:\n",
       "    Alias for `!rm`\n",
       "%rmdir:\n",
       "    Alias for `!rmdir`\n",
       "%run:\n",
       "    Run the named file inside IPython as a program.\n",
       "    \n",
       "    Usage::\n",
       "    \n",
       "      %run [-n -i -e -G]\n",
       "           [( -t [-N<N>] | -d [-b<N>] | -p [profile options] )]\n",
       "           ( -m mod | file ) [args]\n",
       "    \n",
       "    Parameters after the filename are passed as command-line arguments to\n",
       "    the program (put in sys.argv). Then, control returns to IPython's\n",
       "    prompt.\n",
       "    \n",
       "    This is similar to running at a system prompt ``python file args``,\n",
       "    but with the advantage of giving you IPython's tracebacks, and of\n",
       "    loading all variables into your interactive namespace for further use\n",
       "    (unless -p is used, see below).\n",
       "    \n",
       "    The file is executed in a namespace initially consisting only of\n",
       "    ``__name__=='__main__'`` and sys.argv constructed as indicated. It thus\n",
       "    sees its environment as if it were being run as a stand-alone program\n",
       "    (except for sharing global objects such as previously imported\n",
       "    modules). But after execution, the IPython interactive namespace gets\n",
       "    updated with all variables defined in the program (except for __name__\n",
       "    and sys.argv). This allows for very convenient loading of code for\n",
       "    interactive work, while giving each program a 'clean sheet' to run in.\n",
       "    \n",
       "    Arguments are expanded using shell-like glob match.  Patterns\n",
       "    '*', '?', '[seq]' and '[!seq]' can be used.  Additionally,\n",
       "    tilde '~' will be expanded into user's home directory.  Unlike\n",
       "    real shells, quotation does not suppress expansions.  Use\n",
       "    *two* back slashes (e.g. ``\\\\*``) to suppress expansions.\n",
       "    To completely disable these expansions, you can use -G flag.\n",
       "    \n",
       "    On Windows systems, the use of single quotes `'` when specifying \n",
       "    a file is not supported. Use double quotes `\"`.\n",
       "    \n",
       "    Options:\n",
       "    \n",
       "    -n\n",
       "      __name__ is NOT set to '__main__', but to the running file's name\n",
       "      without extension (as python does under import).  This allows running\n",
       "      scripts and reloading the definitions in them without calling code\n",
       "      protected by an ``if __name__ == \"__main__\"`` clause.\n",
       "    \n",
       "    -i\n",
       "      run the file in IPython's namespace instead of an empty one. This\n",
       "      is useful if you are experimenting with code written in a text editor\n",
       "      which depends on variables defined interactively.\n",
       "    \n",
       "    -e\n",
       "      ignore sys.exit() calls or SystemExit exceptions in the script\n",
       "      being run.  This is particularly useful if IPython is being used to\n",
       "      run unittests, which always exit with a sys.exit() call.  In such\n",
       "      cases you are interested in the output of the test results, not in\n",
       "      seeing a traceback of the unittest module.\n",
       "    \n",
       "    -t\n",
       "      print timing information at the end of the run.  IPython will give\n",
       "      you an estimated CPU time consumption for your script, which under\n",
       "      Unix uses the resource module to avoid the wraparound problems of\n",
       "      time.clock().  Under Unix, an estimate of time spent on system tasks\n",
       "      is also given (for Windows platforms this is reported as 0.0).\n",
       "    \n",
       "    If -t is given, an additional ``-N<N>`` option can be given, where <N>\n",
       "    must be an integer indicating how many times you want the script to\n",
       "    run.  The final timing report will include total and per run results.\n",
       "    \n",
       "    For example (testing the script uniq_stable.py)::\n",
       "    \n",
       "        In [1]: run -t uniq_stable\n",
       "    \n",
       "        IPython CPU timings (estimated):\n",
       "          User  :    0.19597 s.\n",
       "          System:        0.0 s.\n",
       "    \n",
       "        In [2]: run -t -N5 uniq_stable\n",
       "    \n",
       "        IPython CPU timings (estimated):\n",
       "        Total runs performed: 5\n",
       "          Times :      Total       Per run\n",
       "          User  :   0.910862 s,  0.1821724 s.\n",
       "          System:        0.0 s,        0.0 s.\n",
       "    \n",
       "    -d\n",
       "      run your program under the control of pdb, the Python debugger.\n",
       "      This allows you to execute your program step by step, watch variables,\n",
       "      etc.  Internally, what IPython does is similar to calling::\n",
       "    \n",
       "          pdb.run('execfile(\"YOURFILENAME\")')\n",
       "    \n",
       "      with a breakpoint set on line 1 of your file.  You can change the line\n",
       "      number for this automatic breakpoint to be <N> by using the -bN option\n",
       "      (where N must be an integer). For example::\n",
       "    \n",
       "          %run -d -b40 myscript\n",
       "    \n",
       "      will set the first breakpoint at line 40 in myscript.py.  Note that\n",
       "      the first breakpoint must be set on a line which actually does\n",
       "      something (not a comment or docstring) for it to stop execution.\n",
       "    \n",
       "      Or you can specify a breakpoint in a different file::\n",
       "    \n",
       "          %run -d -b myotherfile.py:20 myscript\n",
       "    \n",
       "      When the pdb debugger starts, you will see a (Pdb) prompt.  You must\n",
       "      first enter 'c' (without quotes) to start execution up to the first\n",
       "      breakpoint.\n",
       "    \n",
       "      Entering 'help' gives information about the use of the debugger.  You\n",
       "      can easily see pdb's full documentation with \"import pdb;pdb.help()\"\n",
       "      at a prompt.\n",
       "    \n",
       "    -p\n",
       "      run program under the control of the Python profiler module (which\n",
       "      prints a detailed report of execution times, function calls, etc).\n",
       "    \n",
       "      You can pass other options after -p which affect the behavior of the\n",
       "      profiler itself. See the docs for %prun for details.\n",
       "    \n",
       "      In this mode, the program's variables do NOT propagate back to the\n",
       "      IPython interactive namespace (because they remain in the namespace\n",
       "      where the profiler executes them).\n",
       "    \n",
       "      Internally this triggers a call to %prun, see its documentation for\n",
       "      details on the options available specifically for profiling.\n",
       "    \n",
       "    There is one special usage for which the text above doesn't apply:\n",
       "    if the filename ends with .ipy[nb], the file is run as ipython script,\n",
       "    just as if the commands were written on IPython prompt.\n",
       "    \n",
       "    -m\n",
       "      specify module name to load instead of script path. Similar to\n",
       "      the -m option for the python interpreter. Use this option last if you\n",
       "      want to combine with other %run options. Unlike the python interpreter\n",
       "      only source modules are allowed no .pyc or .pyo files.\n",
       "      For example::\n",
       "    \n",
       "          %run -m example\n",
       "    \n",
       "      will run the example module.\n",
       "    \n",
       "    -G\n",
       "      disable shell-like glob expansion of arguments.\n",
       "%save:\n",
       "    Save a set of lines or a macro to a given filename.\n",
       "    \n",
       "    Usage:\n",
       "      %save [options] filename n1-n2 n3-n4 ... n5 .. n6 ...\n",
       "    \n",
       "    Options:\n",
       "    \n",
       "      -r: use 'raw' input.  By default, the 'processed' history is used,\n",
       "      so that magics are loaded in their transformed version to valid\n",
       "      Python.  If this option is given, the raw input as typed as the\n",
       "      command line is used instead.\n",
       "    \n",
       "      -f: force overwrite.  If file exists, %save will prompt for overwrite\n",
       "      unless -f is given.\n",
       "    \n",
       "      -a: append to the file instead of overwriting it.\n",
       "    \n",
       "    This function uses the same syntax as %history for input ranges,\n",
       "    then saves the lines to the filename you specify.\n",
       "    \n",
       "    It adds a '.py' extension to the file if you don't do so yourself, and\n",
       "    it asks for confirmation before overwriting existing files.\n",
       "    \n",
       "    If `-r` option is used, the default extension is `.ipy`.\n",
       "%sc:\n",
       "    Shell capture - run shell command and capture output (DEPRECATED use !).\n",
       "    \n",
       "    DEPRECATED. Suboptimal, retained for backwards compatibility.\n",
       "    \n",
       "    You should use the form 'var = !command' instead. Example:\n",
       "    \n",
       "     \"%sc -l myfiles = ls ~\" should now be written as\n",
       "    \n",
       "     \"myfiles = !ls ~\"\n",
       "    \n",
       "    myfiles.s, myfiles.l and myfiles.n still apply as documented\n",
       "    below.\n",
       "    \n",
       "    --\n",
       "    %sc [options] varname=command\n",
       "    \n",
       "    IPython will run the given command using commands.getoutput(), and\n",
       "    will then update the user's interactive namespace with a variable\n",
       "    called varname, containing the value of the call.  Your command can\n",
       "    contain shell wildcards, pipes, etc.\n",
       "    \n",
       "    The '=' sign in the syntax is mandatory, and the variable name you\n",
       "    supply must follow Python's standard conventions for valid names.\n",
       "    \n",
       "    (A special format without variable name exists for internal use)\n",
       "    \n",
       "    Options:\n",
       "    \n",
       "      -l: list output.  Split the output on newlines into a list before\n",
       "      assigning it to the given variable.  By default the output is stored\n",
       "      as a single string.\n",
       "    \n",
       "      -v: verbose.  Print the contents of the variable.\n",
       "    \n",
       "    In most cases you should not need to split as a list, because the\n",
       "    returned value is a special type of string which can automatically\n",
       "    provide its contents either as a list (split on newlines) or as a\n",
       "    space-separated string.  These are convenient, respectively, either\n",
       "    for sequential processing or to be passed to a shell command.\n",
       "    \n",
       "    For example::\n",
       "    \n",
       "        # Capture into variable a\n",
       "        In [1]: sc a=ls *py\n",
       "    \n",
       "        # a is a string with embedded newlines\n",
       "        In [2]: a\n",
       "        Out[2]: 'setup.py\\nwin32_manual_post_install.py'\n",
       "    \n",
       "        # which can be seen as a list:\n",
       "        In [3]: a.l\n",
       "        Out[3]: ['setup.py', 'win32_manual_post_install.py']\n",
       "    \n",
       "        # or as a whitespace-separated string:\n",
       "        In [4]: a.s\n",
       "        Out[4]: 'setup.py win32_manual_post_install.py'\n",
       "    \n",
       "        # a.s is useful to pass as a single command line:\n",
       "        In [5]: !wc -l $a.s\n",
       "          146 setup.py\n",
       "          130 win32_manual_post_install.py\n",
       "          276 total\n",
       "    \n",
       "        # while the list form is useful to loop over:\n",
       "        In [6]: for f in a.l:\n",
       "          ...:      !wc -l $f\n",
       "          ...:\n",
       "        146 setup.py\n",
       "        130 win32_manual_post_install.py\n",
       "    \n",
       "    Similarly, the lists returned by the -l option are also special, in\n",
       "    the sense that you can equally invoke the .s attribute on them to\n",
       "    automatically get a whitespace-separated string from their contents::\n",
       "    \n",
       "        In [7]: sc -l b=ls *py\n",
       "    \n",
       "        In [8]: b\n",
       "        Out[8]: ['setup.py', 'win32_manual_post_install.py']\n",
       "    \n",
       "        In [9]: b.s\n",
       "        Out[9]: 'setup.py win32_manual_post_install.py'\n",
       "    \n",
       "    In summary, both the lists and strings used for output capture have\n",
       "    the following special attributes::\n",
       "    \n",
       "        .l (or .list) : value as list.\n",
       "        .n (or .nlstr): value as newline-separated string.\n",
       "        .s (or .spstr): value as space-separated string.\n",
       "%set_env:\n",
       "    Set environment variables.  Assumptions are that either \"val\" is a\n",
       "    name in the user namespace, or val is something that evaluates to a\n",
       "    string.\n",
       "    \n",
       "    Usage:\n",
       "      %set_env var val: set value for var\n",
       "      %set_env var=val: set value for var\n",
       "      %set_env var=$val: set value for var, using python expansion if possible\n",
       "%store:\n",
       "    Lightweight persistence for python variables.\n",
       "    \n",
       "    Example::\n",
       "    \n",
       "      In [1]: l = ['hello',10,'world']\n",
       "      In [2]: %store l\n",
       "      In [3]: exit\n",
       "    \n",
       "      (IPython session is closed and started again...)\n",
       "    \n",
       "      ville@badger:~$ ipython\n",
       "      In [1]: l\n",
       "      NameError: name 'l' is not defined\n",
       "      In [2]: %store -r\n",
       "      In [3]: l\n",
       "      Out[3]: ['hello', 10, 'world']\n",
       "    \n",
       "    Usage:\n",
       "    \n",
       "    * ``%store``          - Show list of all variables and their current\n",
       "                            values\n",
       "    * ``%store spam bar`` - Store the *current* value of the variables spam\n",
       "                            and bar to disk\n",
       "    * ``%store -d spam``  - Remove the variable and its value from storage\n",
       "    * ``%store -z``       - Remove all variables from storage\n",
       "    * ``%store -r``       - Refresh all variables, aliases and directory history\n",
       "                            from store (overwrite current vals)\n",
       "    * ``%store -r spam bar`` - Refresh specified variables and aliases from store\n",
       "                               (delete current val)\n",
       "    * ``%store foo >a.txt``  - Store value of foo to new file a.txt\n",
       "    * ``%store foo >>a.txt`` - Append value of foo to file a.txt\n",
       "    \n",
       "    It should be noted that if you change the value of a variable, you\n",
       "    need to %store it again if you want to persist the new value.\n",
       "    \n",
       "    Note also that the variables will need to be pickleable; most basic\n",
       "    python types can be safely %store'd.\n",
       "    \n",
       "    Also aliases can be %store'd across sessions.\n",
       "    To remove an alias from the storage, use the %unalias magic.\n",
       "%sx:\n",
       "    Shell execute - run shell command and capture output (!! is short-hand).\n",
       "    \n",
       "    %sx command\n",
       "    \n",
       "    IPython will run the given command using commands.getoutput(), and\n",
       "    return the result formatted as a list (split on '\\n').  Since the\n",
       "    output is _returned_, it will be stored in ipython's regular output\n",
       "    cache Out[N] and in the '_N' automatic variables.\n",
       "    \n",
       "    Notes:\n",
       "    \n",
       "    1) If an input line begins with '!!', then %sx is automatically\n",
       "    invoked.  That is, while::\n",
       "    \n",
       "      !ls\n",
       "    \n",
       "    causes ipython to simply issue system('ls'), typing::\n",
       "    \n",
       "      !!ls\n",
       "    \n",
       "    is a shorthand equivalent to::\n",
       "    \n",
       "      %sx ls\n",
       "    \n",
       "    2) %sx differs from %sc in that %sx automatically splits into a list,\n",
       "    like '%sc -l'.  The reason for this is to make it as easy as possible\n",
       "    to process line-oriented shell output via further python commands.\n",
       "    %sc is meant to provide much finer control, but requires more\n",
       "    typing.\n",
       "    \n",
       "    3) Just like %sc -l, this is a list with special attributes:\n",
       "    ::\n",
       "    \n",
       "      .l (or .list) : value as list.\n",
       "      .n (or .nlstr): value as newline-separated string.\n",
       "      .s (or .spstr): value as whitespace-separated string.\n",
       "    \n",
       "    This is very useful when trying to use such lists as arguments to\n",
       "    system commands.\n",
       "%system:\n",
       "    Shell execute - run shell command and capture output (!! is short-hand).\n",
       "    \n",
       "    %sx command\n",
       "    \n",
       "    IPython will run the given command using commands.getoutput(), and\n",
       "    return the result formatted as a list (split on '\\n').  Since the\n",
       "    output is _returned_, it will be stored in ipython's regular output\n",
       "    cache Out[N] and in the '_N' automatic variables.\n",
       "    \n",
       "    Notes:\n",
       "    \n",
       "    1) If an input line begins with '!!', then %sx is automatically\n",
       "    invoked.  That is, while::\n",
       "    \n",
       "      !ls\n",
       "    \n",
       "    causes ipython to simply issue system('ls'), typing::\n",
       "    \n",
       "      !!ls\n",
       "    \n",
       "    is a shorthand equivalent to::\n",
       "    \n",
       "      %sx ls\n",
       "    \n",
       "    2) %sx differs from %sc in that %sx automatically splits into a list,\n",
       "    like '%sc -l'.  The reason for this is to make it as easy as possible\n",
       "    to process line-oriented shell output via further python commands.\n",
       "    %sc is meant to provide much finer control, but requires more\n",
       "    typing.\n",
       "    \n",
       "    3) Just like %sc -l, this is a list with special attributes:\n",
       "    ::\n",
       "    \n",
       "      .l (or .list) : value as list.\n",
       "      .n (or .nlstr): value as newline-separated string.\n",
       "      .s (or .spstr): value as whitespace-separated string.\n",
       "    \n",
       "    This is very useful when trying to use such lists as arguments to\n",
       "    system commands.\n",
       "%tb:\n",
       "    Print the last traceback.\n",
       "    \n",
       "    Optionally, specify an exception reporting mode, tuning the\n",
       "    verbosity of the traceback. By default the currently-active exception\n",
       "    mode is used. See %xmode for changing exception reporting modes.\n",
       "    \n",
       "    Valid modes: Plain, Context, Verbose, and Minimal.\n",
       "%time:\n",
       "    Time execution of a Python statement or expression.\n",
       "    \n",
       "    The CPU and wall clock times are printed, and the value of the\n",
       "    expression (if any) is returned.  Note that under Win32, system time\n",
       "    is always reported as 0, since it can not be measured.\n",
       "    \n",
       "    This function can be used both as a line and cell magic:\n",
       "    \n",
       "    - In line mode you can time a single-line statement (though multiple\n",
       "      ones can be chained with using semicolons).\n",
       "    \n",
       "    - In cell mode, you can time the cell body (a directly\n",
       "      following statement raises an error).\n",
       "    \n",
       "    This function provides very basic timing functionality.  Use the timeit\n",
       "    magic for more control over the measurement.\n",
       "    \n",
       "    .. versionchanged:: 7.3\n",
       "        User variables are no longer expanded,\n",
       "        the magic line is always left unmodified.\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    ::\n",
       "    \n",
       "      In [1]: %time 2**128\n",
       "      CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s\n",
       "      Wall time: 0.00\n",
       "      Out[1]: 340282366920938463463374607431768211456L\n",
       "    \n",
       "      In [2]: n = 1000000\n",
       "    \n",
       "      In [3]: %time sum(range(n))\n",
       "      CPU times: user 1.20 s, sys: 0.05 s, total: 1.25 s\n",
       "      Wall time: 1.37\n",
       "      Out[3]: 499999500000L\n",
       "    \n",
       "      In [4]: %time print 'hello world'\n",
       "      hello world\n",
       "      CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s\n",
       "      Wall time: 0.00\n",
       "    \n",
       "      Note that the time needed by Python to compile the given expression\n",
       "      will be reported if it is more than 0.1s.  In this example, the\n",
       "      actual exponentiation is done by Python at compilation time, so while\n",
       "      the expression can take a noticeable amount of time to compute, that\n",
       "      time is purely due to the compilation:\n",
       "    \n",
       "      In [5]: %time 3**9999;\n",
       "      CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s\n",
       "      Wall time: 0.00 s\n",
       "    \n",
       "      In [6]: %time 3**999999;\n",
       "      CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s\n",
       "      Wall time: 0.00 s\n",
       "      Compiler : 0.78 s\n",
       "%timeit:\n",
       "    Time execution of a Python statement or expression\n",
       "    \n",
       "    Usage, in line mode:\n",
       "      %timeit [-n<N> -r<R> [-t|-c] -q -p<P> -o] statement\n",
       "    or in cell mode:\n",
       "      %%timeit [-n<N> -r<R> [-t|-c] -q -p<P> -o] setup_code\n",
       "      code\n",
       "      code...\n",
       "    \n",
       "    Time execution of a Python statement or expression using the timeit\n",
       "    module.  This function can be used both as a line and cell magic:\n",
       "    \n",
       "    - In line mode you can time a single-line statement (though multiple\n",
       "      ones can be chained with using semicolons).\n",
       "    \n",
       "    - In cell mode, the statement in the first line is used as setup code\n",
       "      (executed but not timed) and the body of the cell is timed.  The cell\n",
       "      body has access to any variables created in the setup code.\n",
       "    \n",
       "    Options:\n",
       "    -n<N>: execute the given statement <N> times in a loop. If <N> is not\n",
       "    provided, <N> is determined so as to get sufficient accuracy.\n",
       "    \n",
       "    -r<R>: number of repeats <R>, each consisting of <N> loops, and take the\n",
       "    best result.\n",
       "    Default: 7\n",
       "    \n",
       "    -t: use time.time to measure the time, which is the default on Unix.\n",
       "    This function measures wall time.\n",
       "    \n",
       "    -c: use time.clock to measure the time, which is the default on\n",
       "    Windows and measures wall time. On Unix, resource.getrusage is used\n",
       "    instead and returns the CPU user time.\n",
       "    \n",
       "    -p<P>: use a precision of <P> digits to display the timing result.\n",
       "    Default: 3\n",
       "    \n",
       "    -q: Quiet, do not print result.\n",
       "    \n",
       "    -o: return a TimeitResult that can be stored in a variable to inspect\n",
       "        the result in more details.\n",
       "    \n",
       "    .. versionchanged:: 7.3\n",
       "        User variables are no longer expanded,\n",
       "        the magic line is always left unmodified.\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    ::\n",
       "    \n",
       "      In [1]: %timeit pass\n",
       "      8.26 ns ± 0.12 ns per loop (mean ± std. dev. of 7 runs, 100000000 loops each)\n",
       "    \n",
       "      In [2]: u = None\n",
       "    \n",
       "      In [3]: %timeit u is None\n",
       "      29.9 ns ± 0.643 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)\n",
       "    \n",
       "      In [4]: %timeit -r 4 u == None\n",
       "    \n",
       "      In [5]: import time\n",
       "    \n",
       "      In [6]: %timeit -n1 time.sleep(2)\n",
       "    \n",
       "    \n",
       "    The times reported by %timeit will be slightly higher than those\n",
       "    reported by the timeit.py script when variables are accessed. This is\n",
       "    due to the fact that %timeit executes the statement in the namespace\n",
       "    of the shell, compared with timeit.py, which uses a single setup\n",
       "    statement to import function or create variables. Generally, the bias\n",
       "    does not matter as long as results from timeit.py are not mixed with\n",
       "    those from %timeit.\n",
       "%unalias:\n",
       "    Remove an alias\n",
       "%unload_ext:\n",
       "    Unload an IPython extension by its module name.\n",
       "    \n",
       "    Not all extensions can be unloaded, only those which define an\n",
       "    ``unload_ipython_extension`` function.\n",
       "%who:\n",
       "    Print all interactive variables, with some minimal formatting.\n",
       "    \n",
       "    If any arguments are given, only variables whose type matches one of\n",
       "    these are printed.  For example::\n",
       "    \n",
       "      %who function str\n",
       "    \n",
       "    will only list functions and strings, excluding all other types of\n",
       "    variables.  To find the proper type names, simply use type(var) at a\n",
       "    command line to see how python prints type names.  For example:\n",
       "    \n",
       "    ::\n",
       "    \n",
       "      In [1]: type('hello')\n",
       "      Out[1]: <type 'str'>\n",
       "    \n",
       "    indicates that the type name for strings is 'str'.\n",
       "    \n",
       "    ``%who`` always excludes executed names loaded through your configuration\n",
       "    file and things which are internal to IPython.\n",
       "    \n",
       "    This is deliberate, as typically you may load many modules and the\n",
       "    purpose of %who is to show you only what you've manually defined.\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    \n",
       "    Define two variables and list them with who::\n",
       "    \n",
       "      In [1]: alpha = 123\n",
       "    \n",
       "      In [2]: beta = 'test'\n",
       "    \n",
       "      In [3]: %who\n",
       "      alpha   beta\n",
       "    \n",
       "      In [4]: %who int\n",
       "      alpha\n",
       "    \n",
       "      In [5]: %who str\n",
       "      beta\n",
       "%who_ls:\n",
       "    Return a sorted list of all interactive variables.\n",
       "    \n",
       "    If arguments are given, only variables of types matching these\n",
       "    arguments are returned.\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    \n",
       "    Define two variables and list them with who_ls::\n",
       "    \n",
       "      In [1]: alpha = 123\n",
       "    \n",
       "      In [2]: beta = 'test'\n",
       "    \n",
       "      In [3]: %who_ls\n",
       "      Out[3]: ['alpha', 'beta']\n",
       "    \n",
       "      In [4]: %who_ls int\n",
       "      Out[4]: ['alpha']\n",
       "    \n",
       "      In [5]: %who_ls str\n",
       "      Out[5]: ['beta']\n",
       "%whos:\n",
       "    Like %who, but gives some extra information about each variable.\n",
       "    \n",
       "    The same type filtering of %who can be applied here.\n",
       "    \n",
       "    For all variables, the type is printed. Additionally it prints:\n",
       "    \n",
       "      - For {},[],(): their length.\n",
       "    \n",
       "      - For numpy arrays, a summary with shape, number of\n",
       "        elements, typecode and size in memory.\n",
       "    \n",
       "      - Everything else: a string representation, snipping their middle if\n",
       "        too long.\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    \n",
       "    Define two variables and list them with whos::\n",
       "    \n",
       "      In [1]: alpha = 123\n",
       "    \n",
       "      In [2]: beta = 'test'\n",
       "    \n",
       "      In [3]: %whos\n",
       "      Variable   Type        Data/Info\n",
       "      --------------------------------\n",
       "      alpha      int         123\n",
       "      beta       str         test\n",
       "%xdel:\n",
       "    Delete a variable, trying to clear it from anywhere that\n",
       "    IPython's machinery has references to it. By default, this uses\n",
       "    the identity of the named object in the user namespace to remove\n",
       "    references held under other names. The object is also removed\n",
       "    from the output history.\n",
       "    \n",
       "    Options\n",
       "      -n : Delete the specified name from all namespaces, without\n",
       "      checking their identity.\n",
       "%xmode:\n",
       "    Switch modes for the exception handlers.\n",
       "    \n",
       "    Valid modes: Plain, Context, Verbose, and Minimal.\n",
       "    \n",
       "    If called without arguments, acts as a toggle.\n",
       "%%!:\n",
       "    Shell execute - run shell command and capture output (!! is short-hand).\n",
       "    \n",
       "    %sx command\n",
       "    \n",
       "    IPython will run the given command using commands.getoutput(), and\n",
       "    return the result formatted as a list (split on '\\n').  Since the\n",
       "    output is _returned_, it will be stored in ipython's regular output\n",
       "    cache Out[N] and in the '_N' automatic variables.\n",
       "    \n",
       "    Notes:\n",
       "    \n",
       "    1) If an input line begins with '!!', then %sx is automatically\n",
       "    invoked.  That is, while::\n",
       "    \n",
       "      !ls\n",
       "    \n",
       "    causes ipython to simply issue system('ls'), typing::\n",
       "    \n",
       "      !!ls\n",
       "    \n",
       "    is a shorthand equivalent to::\n",
       "    \n",
       "      %sx ls\n",
       "    \n",
       "    2) %sx differs from %sc in that %sx automatically splits into a list,\n",
       "    like '%sc -l'.  The reason for this is to make it as easy as possible\n",
       "    to process line-oriented shell output via further python commands.\n",
       "    %sc is meant to provide much finer control, but requires more\n",
       "    typing.\n",
       "    \n",
       "    3) Just like %sc -l, this is a list with special attributes:\n",
       "    ::\n",
       "    \n",
       "      .l (or .list) : value as list.\n",
       "      .n (or .nlstr): value as newline-separated string.\n",
       "      .s (or .spstr): value as whitespace-separated string.\n",
       "    \n",
       "    This is very useful when trying to use such lists as arguments to\n",
       "    system commands.\n",
       "%%HTML:\n",
       "    Alias for `%%html`.\n",
       "%%SVG:\n",
       "    Alias for `%%svg`.\n",
       "%%bash:\n",
       "    %%bash script magic\n",
       "    \n",
       "    Run cells with bash in a subprocess.\n",
       "    \n",
       "    This is a shortcut for `%%script bash`\n",
       "%%capture:\n",
       "    ::\n",
       "    \n",
       "      %capture [--no-stderr] [--no-stdout] [--no-display] [output]\n",
       "    \n",
       "    run the cell, capturing stdout, stderr, and IPython's rich display() calls.\n",
       "    \n",
       "    positional arguments:\n",
       "      output        The name of the variable in which to store output. This is a\n",
       "                    utils.io.CapturedIO object with stdout/err attributes for the\n",
       "                    text of the captured output. CapturedOutput also has a show()\n",
       "                    method for displaying the output, and __call__ as well, so you\n",
       "                    can use that to quickly display the output. If unspecified,\n",
       "                    captured output is discarded.\n",
       "    \n",
       "    optional arguments:\n",
       "      --no-stderr   Don't capture stderr.\n",
       "      --no-stdout   Don't capture stdout.\n",
       "      --no-display  Don't capture IPython's rich display.\n",
       "%%debug:\n",
       "    ::\n",
       "    \n",
       "      %debug [--breakpoint FILE:LINE] [statement [statement ...]]\n",
       "    \n",
       "    Activate the interactive debugger.\n",
       "    \n",
       "    This magic command support two ways of activating debugger.\n",
       "    One is to activate debugger before executing code.  This way, you\n",
       "    can set a break point, to step through the code from the point.\n",
       "    You can use this mode by giving statements to execute and optionally\n",
       "    a breakpoint.\n",
       "    \n",
       "    The other one is to activate debugger in post-mortem mode.  You can\n",
       "    activate this mode simply running %debug without any argument.\n",
       "    If an exception has just occurred, this lets you inspect its stack\n",
       "    frames interactively.  Note that this will always work only on the last\n",
       "    traceback that occurred, so you must call this quickly after an\n",
       "    exception that you wish to inspect has fired, because if another one\n",
       "    occurs, it clobbers the previous one.\n",
       "    \n",
       "    If you want IPython to automatically do this on every exception, see\n",
       "    the %pdb magic for more details.\n",
       "    \n",
       "    .. versionchanged:: 7.3\n",
       "        When running code, user variables are no longer expanded,\n",
       "        the magic line is always left unmodified.\n",
       "    \n",
       "    positional arguments:\n",
       "      statement             Code to run in debugger. You can omit this in cell\n",
       "                            magic mode.\n",
       "    \n",
       "    optional arguments:\n",
       "      --breakpoint <FILE:LINE>, -b <FILE:LINE>\n",
       "                            Set break point at LINE in FILE.\n",
       "%%file:\n",
       "    Alias for `%%writefile`.\n",
       "%%html:\n",
       "    ::\n",
       "    \n",
       "      %html [--isolated]\n",
       "    \n",
       "    Render the cell as a block of HTML\n",
       "    \n",
       "    optional arguments:\n",
       "      --isolated  Annotate the cell as 'isolated'. Isolated cells are rendered\n",
       "                  inside their own <iframe> tag\n",
       "%%javascript:\n",
       "    Run the cell block of Javascript code\n",
       "%%js:\n",
       "    Run the cell block of Javascript code\n",
       "    \n",
       "    Alias of `%%javascript`\n",
       "%%latex:\n",
       "    Render the cell as a block of latex\n",
       "    \n",
       "    The subset of latex which is support depends on the implementation in\n",
       "    the client.  In the Jupyter Notebook, this magic only renders the subset\n",
       "    of latex defined by MathJax\n",
       "    [here](https://docs.mathjax.org/en/v2.5-latest/tex.html).\n",
       "%%markdown:\n",
       "    Render the cell as Markdown text block\n",
       "%%perl:\n",
       "    %%perl script magic\n",
       "    \n",
       "    Run cells with perl in a subprocess.\n",
       "    \n",
       "    This is a shortcut for `%%script perl`\n",
       "%%prun:\n",
       "    Run a statement through the python code profiler.\n",
       "    \n",
       "    Usage, in line mode:\n",
       "      %prun [options] statement\n",
       "    \n",
       "    Usage, in cell mode:\n",
       "      %%prun [options] [statement]\n",
       "      code...\n",
       "      code...\n",
       "    \n",
       "    In cell mode, the additional code lines are appended to the (possibly\n",
       "    empty) statement in the first line.  Cell mode allows you to easily\n",
       "    profile multiline blocks without having to put them in a separate\n",
       "    function.\n",
       "    \n",
       "    The given statement (which doesn't require quote marks) is run via the\n",
       "    python profiler in a manner similar to the profile.run() function.\n",
       "    Namespaces are internally managed to work correctly; profile.run\n",
       "    cannot be used in IPython because it makes certain assumptions about\n",
       "    namespaces which do not hold under IPython.\n",
       "    \n",
       "    Options:\n",
       "    \n",
       "    -l <limit>\n",
       "      you can place restrictions on what or how much of the\n",
       "      profile gets printed. The limit value can be:\n",
       "    \n",
       "         * A string: only information for function names containing this string\n",
       "           is printed.\n",
       "    \n",
       "         * An integer: only these many lines are printed.\n",
       "    \n",
       "         * A float (between 0 and 1): this fraction of the report is printed\n",
       "           (for example, use a limit of 0.4 to see the topmost 40% only).\n",
       "    \n",
       "      You can combine several limits with repeated use of the option. For\n",
       "      example, ``-l __init__ -l 5`` will print only the topmost 5 lines of\n",
       "      information about class constructors.\n",
       "    \n",
       "    -r\n",
       "      return the pstats.Stats object generated by the profiling. This\n",
       "      object has all the information about the profile in it, and you can\n",
       "      later use it for further analysis or in other functions.\n",
       "    \n",
       "    -s <key>\n",
       "      sort profile by given key. You can provide more than one key\n",
       "      by using the option several times: '-s key1 -s key2 -s key3...'. The\n",
       "      default sorting key is 'time'.\n",
       "    \n",
       "      The following is copied verbatim from the profile documentation\n",
       "      referenced below:\n",
       "    \n",
       "      When more than one key is provided, additional keys are used as\n",
       "      secondary criteria when the there is equality in all keys selected\n",
       "      before them.\n",
       "    \n",
       "      Abbreviations can be used for any key names, as long as the\n",
       "      abbreviation is unambiguous.  The following are the keys currently\n",
       "      defined:\n",
       "    \n",
       "      ============  =====================\n",
       "      Valid Arg     Meaning\n",
       "      ============  =====================\n",
       "      \"calls\"       call count\n",
       "      \"cumulative\"  cumulative time\n",
       "      \"file\"        file name\n",
       "      \"module\"      file name\n",
       "      \"pcalls\"      primitive call count\n",
       "      \"line\"        line number\n",
       "      \"name\"        function name\n",
       "      \"nfl\"         name/file/line\n",
       "      \"stdname\"     standard name\n",
       "      \"time\"        internal time\n",
       "      ============  =====================\n",
       "    \n",
       "      Note that all sorts on statistics are in descending order (placing\n",
       "      most time consuming items first), where as name, file, and line number\n",
       "      searches are in ascending order (i.e., alphabetical). The subtle\n",
       "      distinction between \"nfl\" and \"stdname\" is that the standard name is a\n",
       "      sort of the name as printed, which means that the embedded line\n",
       "      numbers get compared in an odd way.  For example, lines 3, 20, and 40\n",
       "      would (if the file names were the same) appear in the string order\n",
       "      \"20\" \"3\" and \"40\".  In contrast, \"nfl\" does a numeric compare of the\n",
       "      line numbers.  In fact, sort_stats(\"nfl\") is the same as\n",
       "      sort_stats(\"name\", \"file\", \"line\").\n",
       "    \n",
       "    -T <filename>\n",
       "      save profile results as shown on screen to a text\n",
       "      file. The profile is still shown on screen.\n",
       "    \n",
       "    -D <filename>\n",
       "      save (via dump_stats) profile statistics to given\n",
       "      filename. This data is in a format understood by the pstats module, and\n",
       "      is generated by a call to the dump_stats() method of profile\n",
       "      objects. The profile is still shown on screen.\n",
       "    \n",
       "    -q\n",
       "      suppress output to the pager.  Best used with -T and/or -D above.\n",
       "    \n",
       "    If you want to run complete programs under the profiler's control, use\n",
       "    ``%run -p [prof_opts] filename.py [args to program]`` where prof_opts\n",
       "    contains profiler specific options as described here.\n",
       "    \n",
       "    You can read the complete documentation for the profile module with::\n",
       "    \n",
       "      In [1]: import profile; profile.help()\n",
       "    \n",
       "    .. versionchanged:: 7.3\n",
       "        User variables are no longer expanded,\n",
       "        the magic line is always left unmodified.\n",
       "%%pypy:\n",
       "    %%pypy script magic\n",
       "    \n",
       "    Run cells with pypy in a subprocess.\n",
       "    \n",
       "    This is a shortcut for `%%script pypy`\n",
       "%%python:\n",
       "    %%python script magic\n",
       "    \n",
       "    Run cells with python in a subprocess.\n",
       "    \n",
       "    This is a shortcut for `%%script python`\n",
       "%%python2:\n",
       "    %%python2 script magic\n",
       "    \n",
       "    Run cells with python2 in a subprocess.\n",
       "    \n",
       "    This is a shortcut for `%%script python2`\n",
       "%%python3:\n",
       "    %%python3 script magic\n",
       "    \n",
       "    Run cells with python3 in a subprocess.\n",
       "    \n",
       "    This is a shortcut for `%%script python3`\n",
       "%%ruby:\n",
       "    %%ruby script magic\n",
       "    \n",
       "    Run cells with ruby in a subprocess.\n",
       "    \n",
       "    This is a shortcut for `%%script ruby`\n",
       "%%script:\n",
       "    ::\n",
       "    \n",
       "      %shebang [--no-raise-error] [--proc PROC] [--bg] [--err ERR] [--out OUT]\n",
       "    \n",
       "    Run a cell via a shell command\n",
       "    \n",
       "    The `%%script` line is like the #! line of script,\n",
       "    specifying a program (bash, perl, ruby, etc.) with which to run.\n",
       "    \n",
       "    The rest of the cell is run by that program.\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    ::\n",
       "    \n",
       "        In [1]: %%script bash\n",
       "           ...: for i in 1 2 3; do\n",
       "           ...:   echo $i\n",
       "           ...: done\n",
       "        1\n",
       "        2\n",
       "        3\n",
       "    \n",
       "    optional arguments:\n",
       "      --no-raise-error  Whether you should raise an error message in addition to a\n",
       "                        stream on stderr if you get a nonzero exit code.\n",
       "      --proc PROC       The variable in which to store Popen instance. This is\n",
       "                        used only when --bg option is given.\n",
       "      --bg              Whether to run the script in the background. If given, the\n",
       "                        only way to see the output of the command is with\n",
       "                        --out/err.\n",
       "      --err ERR         The variable in which to store stderr from the script. If\n",
       "                        the script is backgrounded, this will be the stderr\n",
       "                        *pipe*, instead of the stderr text itself and will not be\n",
       "                        autoclosed.\n",
       "      --out OUT         The variable in which to store stdout from the script. If\n",
       "                        the script is backgrounded, this will be the stdout\n",
       "                        *pipe*, instead of the stderr text itself and will not be\n",
       "                        auto closed.\n",
       "%%sh:\n",
       "    %%sh script magic\n",
       "    \n",
       "    Run cells with sh in a subprocess.\n",
       "    \n",
       "    This is a shortcut for `%%script sh`\n",
       "%%svg:\n",
       "    Render the cell as an SVG literal\n",
       "%%sx:\n",
       "    Shell execute - run shell command and capture output (!! is short-hand).\n",
       "    \n",
       "    %sx command\n",
       "    \n",
       "    IPython will run the given command using commands.getoutput(), and\n",
       "    return the result formatted as a list (split on '\\n').  Since the\n",
       "    output is _returned_, it will be stored in ipython's regular output\n",
       "    cache Out[N] and in the '_N' automatic variables.\n",
       "    \n",
       "    Notes:\n",
       "    \n",
       "    1) If an input line begins with '!!', then %sx is automatically\n",
       "    invoked.  That is, while::\n",
       "    \n",
       "      !ls\n",
       "    \n",
       "    causes ipython to simply issue system('ls'), typing::\n",
       "    \n",
       "      !!ls\n",
       "    \n",
       "    is a shorthand equivalent to::\n",
       "    \n",
       "      %sx ls\n",
       "    \n",
       "    2) %sx differs from %sc in that %sx automatically splits into a list,\n",
       "    like '%sc -l'.  The reason for this is to make it as easy as possible\n",
       "    to process line-oriented shell output via further python commands.\n",
       "    %sc is meant to provide much finer control, but requires more\n",
       "    typing.\n",
       "    \n",
       "    3) Just like %sc -l, this is a list with special attributes:\n",
       "    ::\n",
       "    \n",
       "      .l (or .list) : value as list.\n",
       "      .n (or .nlstr): value as newline-separated string.\n",
       "      .s (or .spstr): value as whitespace-separated string.\n",
       "    \n",
       "    This is very useful when trying to use such lists as arguments to\n",
       "    system commands.\n",
       "%%system:\n",
       "    Shell execute - run shell command and capture output (!! is short-hand).\n",
       "    \n",
       "    %sx command\n",
       "    \n",
       "    IPython will run the given command using commands.getoutput(), and\n",
       "    return the result formatted as a list (split on '\\n').  Since the\n",
       "    output is _returned_, it will be stored in ipython's regular output\n",
       "    cache Out[N] and in the '_N' automatic variables.\n",
       "    \n",
       "    Notes:\n",
       "    \n",
       "    1) If an input line begins with '!!', then %sx is automatically\n",
       "    invoked.  That is, while::\n",
       "    \n",
       "      !ls\n",
       "    \n",
       "    causes ipython to simply issue system('ls'), typing::\n",
       "    \n",
       "      !!ls\n",
       "    \n",
       "    is a shorthand equivalent to::\n",
       "    \n",
       "      %sx ls\n",
       "    \n",
       "    2) %sx differs from %sc in that %sx automatically splits into a list,\n",
       "    like '%sc -l'.  The reason for this is to make it as easy as possible\n",
       "    to process line-oriented shell output via further python commands.\n",
       "    %sc is meant to provide much finer control, but requires more\n",
       "    typing.\n",
       "    \n",
       "    3) Just like %sc -l, this is a list with special attributes:\n",
       "    ::\n",
       "    \n",
       "      .l (or .list) : value as list.\n",
       "      .n (or .nlstr): value as newline-separated string.\n",
       "      .s (or .spstr): value as whitespace-separated string.\n",
       "    \n",
       "    This is very useful when trying to use such lists as arguments to\n",
       "    system commands.\n",
       "%%time:\n",
       "    Time execution of a Python statement or expression.\n",
       "    \n",
       "    The CPU and wall clock times are printed, and the value of the\n",
       "    expression (if any) is returned.  Note that under Win32, system time\n",
       "    is always reported as 0, since it can not be measured.\n",
       "    \n",
       "    This function can be used both as a line and cell magic:\n",
       "    \n",
       "    - In line mode you can time a single-line statement (though multiple\n",
       "      ones can be chained with using semicolons).\n",
       "    \n",
       "    - In cell mode, you can time the cell body (a directly\n",
       "      following statement raises an error).\n",
       "    \n",
       "    This function provides very basic timing functionality.  Use the timeit\n",
       "    magic for more control over the measurement.\n",
       "    \n",
       "    .. versionchanged:: 7.3\n",
       "        User variables are no longer expanded,\n",
       "        the magic line is always left unmodified.\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    ::\n",
       "    \n",
       "      In [1]: %time 2**128\n",
       "      CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s\n",
       "      Wall time: 0.00\n",
       "      Out[1]: 340282366920938463463374607431768211456L\n",
       "    \n",
       "      In [2]: n = 1000000\n",
       "    \n",
       "      In [3]: %time sum(range(n))\n",
       "      CPU times: user 1.20 s, sys: 0.05 s, total: 1.25 s\n",
       "      Wall time: 1.37\n",
       "      Out[3]: 499999500000L\n",
       "    \n",
       "      In [4]: %time print 'hello world'\n",
       "      hello world\n",
       "      CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s\n",
       "      Wall time: 0.00\n",
       "    \n",
       "      Note that the time needed by Python to compile the given expression\n",
       "      will be reported if it is more than 0.1s.  In this example, the\n",
       "      actual exponentiation is done by Python at compilation time, so while\n",
       "      the expression can take a noticeable amount of time to compute, that\n",
       "      time is purely due to the compilation:\n",
       "    \n",
       "      In [5]: %time 3**9999;\n",
       "      CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s\n",
       "      Wall time: 0.00 s\n",
       "    \n",
       "      In [6]: %time 3**999999;\n",
       "      CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s\n",
       "      Wall time: 0.00 s\n",
       "      Compiler : 0.78 s\n",
       "%%timeit:\n",
       "    Time execution of a Python statement or expression\n",
       "    \n",
       "    Usage, in line mode:\n",
       "      %timeit [-n<N> -r<R> [-t|-c] -q -p<P> -o] statement\n",
       "    or in cell mode:\n",
       "      %%timeit [-n<N> -r<R> [-t|-c] -q -p<P> -o] setup_code\n",
       "      code\n",
       "      code...\n",
       "    \n",
       "    Time execution of a Python statement or expression using the timeit\n",
       "    module.  This function can be used both as a line and cell magic:\n",
       "    \n",
       "    - In line mode you can time a single-line statement (though multiple\n",
       "      ones can be chained with using semicolons).\n",
       "    \n",
       "    - In cell mode, the statement in the first line is used as setup code\n",
       "      (executed but not timed) and the body of the cell is timed.  The cell\n",
       "      body has access to any variables created in the setup code.\n",
       "    \n",
       "    Options:\n",
       "    -n<N>: execute the given statement <N> times in a loop. If <N> is not\n",
       "    provided, <N> is determined so as to get sufficient accuracy.\n",
       "    \n",
       "    -r<R>: number of repeats <R>, each consisting of <N> loops, and take the\n",
       "    best result.\n",
       "    Default: 7\n",
       "    \n",
       "    -t: use time.time to measure the time, which is the default on Unix.\n",
       "    This function measures wall time.\n",
       "    \n",
       "    -c: use time.clock to measure the time, which is the default on\n",
       "    Windows and measures wall time. On Unix, resource.getrusage is used\n",
       "    instead and returns the CPU user time.\n",
       "    \n",
       "    -p<P>: use a precision of <P> digits to display the timing result.\n",
       "    Default: 3\n",
       "    \n",
       "    -q: Quiet, do not print result.\n",
       "    \n",
       "    -o: return a TimeitResult that can be stored in a variable to inspect\n",
       "        the result in more details.\n",
       "    \n",
       "    .. versionchanged:: 7.3\n",
       "        User variables are no longer expanded,\n",
       "        the magic line is always left unmodified.\n",
       "    \n",
       "    Examples\n",
       "    --------\n",
       "    ::\n",
       "    \n",
       "      In [1]: %timeit pass\n",
       "      8.26 ns ± 0.12 ns per loop (mean ± std. dev. of 7 runs, 100000000 loops each)\n",
       "    \n",
       "      In [2]: u = None\n",
       "    \n",
       "      In [3]: %timeit u is None\n",
       "      29.9 ns ± 0.643 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)\n",
       "    \n",
       "      In [4]: %timeit -r 4 u == None\n",
       "    \n",
       "      In [5]: import time\n",
       "    \n",
       "      In [6]: %timeit -n1 time.sleep(2)\n",
       "    \n",
       "    \n",
       "    The times reported by %timeit will be slightly higher than those\n",
       "    reported by the timeit.py script when variables are accessed. This is\n",
       "    due to the fact that %timeit executes the statement in the namespace\n",
       "    of the shell, compared with timeit.py, which uses a single setup\n",
       "    statement to import function or create variables. Generally, the bias\n",
       "    does not matter as long as results from timeit.py are not mixed with\n",
       "    those from %timeit.\n",
       "%%writefile:\n",
       "    ::\n",
       "    \n",
       "      %writefile [-a] filename\n",
       "    \n",
       "    Write the contents of the cell to a file.\n",
       "    \n",
       "    The file will be overwritten unless the -a (--append) flag is specified.\n",
       "    \n",
       "    positional arguments:\n",
       "      filename      file to write\n",
       "    \n",
       "    optional arguments:\n",
       "      -a, --append  Append contents of the cell to an existing file. The file will\n",
       "                    be created if it does not exist.\n",
       "\n",
       "Summary of magic functions (from %lsmagic):\n",
       "Available line magics:\n",
       "%alias  %alias_magic  %autoawait  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %colors  %conda  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %pip  %popd  %pprint  %precision  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode\n",
       "\n",
       "Available cell magics:\n",
       "%%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%markdown  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile\n",
       "\n",
       "Automagic is ON, % prefix IS NOT needed for line magics."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Getting some general help\n",
    "%magic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python native data types\n",
    "\n",
    "Python has many native datatypes. Here are the important ones:\n",
    "\n",
    " - **Booleans** are either `True` or `False`.\n",
    " - **Numbers** can be integers (1 and 2), floats (1.1 and 1.2), fractions (1/2 and 2/3), or even complex numbers.\n",
    " - **Strings** are sequences of Unicode characters, e.g. an html document.\n",
    " - **Lists** are ordered sequences of values.\n",
    " - **Tuples** are ordered, immutable sequences of values.\n",
    " - **Sets** are unordered bags of values.\n",
    " - **Dictionaries** are unordered bags of key-value pairs.\n",
    " \n",
    "See [here](http://www.diveintopython3.net/native-datatypes.html) for a complete overview.\n",
    "\n",
    "### More resources\n",
    "\n",
    " 1. Brief Python introduction [here](https://learnxinyminutes.com/docs/python3/).\n",
    " 2. Full Python tutorial [here](https://docs.python.org/3/tutorial/).\n",
    " 3. A Whirlwind Tour of Python [here](https://github.com/jakevdp/WhirlwindTourOfPython).\n",
    " 4. Python Data Science Handbook [here](https://github.com/jakevdp/PythonDataScienceHandbook)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Torch!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate a tensor of size 2x3x4\n",
    "t = torch.Tensor(2, 3, 4)\n",
    "type(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the size of the tensor\n",
    "t.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# t.size() is a classic tuple =>\n",
    "print('t size:', ' \\u00D7 '.join(map(str, t.size())))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prints dimensional space and sub-dimensions\n",
    "print(f'point in a {t.numel()} dimensional space')\n",
    "print(f'organised in {t.dim()} sub-dimensions')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Mind the underscore!\n",
    "# Any operation that mutates a tensor in-place is post-fixed with an _.\n",
    "# For example: x.copy_(y), x.t_(), x.random_(n) will change x.\n",
    "t.random_(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This resizes the tensor permanently \n",
    "r = torch.Tensor(t)\n",
    "r.resize_(3, 8)\n",
    "r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# As you can see zero_ would replace r with 0's which was originally filled with integers\n",
    "r.zero_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This *is* important, sigh...\n",
    "s = r.clone()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In-place fill of 1's\n",
    "s.fill_(1)\n",
    "s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Because we cloned r, even though we did an in-place operation, this doesn't affect r\n",
    "r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vectors (1D Tensors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creates a 1D tensor of integers 1 to 4\n",
    "v = torch.Tensor([1, 2, 3, 4])\n",
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print number of dimensions (1D) and size of tensor\n",
    "print(f'dim: {v.dim()}, size: {v.size()[0]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w = torch.Tensor([1, 0, 2, 0])\n",
    "w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Element-wise multiplication\n",
    "v * w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scalar product: 1*1 + 2*0 + 3*2 + 4*0\n",
    "v @ w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In-place replacement of random number from 0 to 10\n",
    "x = torch.Tensor(5).random_(10)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'first: {x[0]}, last: {x[-1]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract sub-Tensor [from:to)\n",
    "x[1:2 + 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tensor with integers ranging from 1 to 5, excluding 5\n",
    "v = torch.arange(1, 4 + 1)\n",
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Square all elements in the tensor\n",
    "print(v.pow(2), v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrices (2D Tensors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a 2x4 tensor\n",
    "m = torch.Tensor([[2, 5, 3, 7],\n",
    "                  [4, 2, 1, 9]])\n",
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m.dim()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(m.size(0), m.size(1), m.size(), sep=' -- ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Returns the total number of elements, hence num-el (number of elements)\n",
    "m.numel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Indexing row 0, column 2 (0-indexed)\n",
    "m[0][2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Indexing row 0, column 2 (0-indexed)\n",
    "m[0, 2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Indexing column 1, all rows (returns size 2)\n",
    "m[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Indexing column 1, all rows (returns size 2x2)\n",
    "m[:, [1]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Indexes row 0, all columns (returns 1x4)\n",
    "m[[0], :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Indexes row 0, all columns (returns size 4)\n",
    "m[0, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create tensor of numbers from 1 to 5 (excluding 5)\n",
    "v = torch.arange(1., 4 + 1)\n",
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scalar product\n",
    "m @ v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculated by 1*2 + 2*5 + 3*3 + 4*7\n",
    "m[[0], :] @ v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculated by \n",
    "m[[1], :] @ v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add a random tensor of size 2x4 to m\n",
    "m + torch.rand(2, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Subtract a random tensor of size 2x4 to m\n",
    "m - torch.rand(2, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Multiply a random tensor of size 2x4 to m\n",
    "m * torch.rand(2, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Divide m by a random tensor of size 2x4\n",
    "m / torch.rand(2, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transpose tensor m, which is essentially 2x4 to 4x2\n",
    "m.t()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Same as\n",
    "m.transpose(0, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Constructors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create tensor from 3 to 8, with each having a space of 1\n",
    "torch.arange(3., 8 + 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create tensor from 5.7 to -2.1 with each having a space of -3\n",
    "torch.arange(5.7, -2.1, -3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# returns a 1D tensor of steps equally spaced points between start=3, end=8 and steps=20\n",
    "torch.linspace(3, 8, 20).view(1, -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tensor filled with 0's\n",
    "torch.zeros(3, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tensor filled with 1's\n",
    "torch.ones(3, 2, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tensor with the diagonal filled with 1\n",
    "torch.eye(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set default plots\n",
    "from res.plot_lib import set_default\n",
    "from matplotlib import pyplot as plt\n",
    "set_default()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Numpy bridge!\n",
    "plt.hist(torch.randn(1000).numpy(), 100);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hist(torch.randn(10**6).numpy(), 100);  # how much does this chart weight?\n",
    "# use rasterized=True for SVG/EPS/PDF!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hist(torch.rand(10**6).numpy(), 100);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Casting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helper to get what kind of tensor types\n",
    "torch.*Tensor?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is basically a 64 bit float tensor\n",
    "m_double = m.double()\n",
    "m_double"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This creates a tensor of type int8\n",
    "m_byte = m.byte()\n",
    "m_byte"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Move your tensor to GPU device 0 if there is one (first GPU in the system)\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "m.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Converts tensor to numpy array\n",
    "m_np = m.numpy()\n",
    "m_np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In-place fill of column 0 and row 0 with value -1\n",
    "m_np[0, 0] = -1\n",
    "m_np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tensor of integers ranging from 0 to 4\n",
    "import numpy as np\n",
    "n_np = np.arange(5)\n",
    "n = torch.from_numpy(n_np)\n",
    "print(n_np, n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In-place multiplication of all elements by 2 for tensor n\n",
    "# Because n is essentiall n_np, not a clone, this affects n_np\n",
    "n.mul_(2)\n",
    "n_np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More fun"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creates two tensor of size 1x4\n",
    "a = torch.Tensor([[1, 2, 3, 4]])\n",
    "b = torch.Tensor([[5, 6, 7, 8]])\n",
    "print(a.size(), b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Concatenate on axis 0, so you get 2x4\n",
    "torch.cat((a, b), 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Concatenate on axis 1, so you get 1x8\n",
    "torch.cat((a, b), 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Much more\n",
    "\n",
    "There's definitely much more, but this was the basics about `Tensor`s fun.\n",
    "\n",
    "*Torch* full API should be read at least once.\n",
    "Hence, go [here](https://pytorch.org/docs/stable/index.html).\n",
    "You'll find 100+ `Tensor` operations, including transposing, indexing, slicing, mathematical operations, linear algebra, random numbers, etc are described."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}