6_LunarLander_PolicyBased.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    },
    "colab": {
      "name": "6_LunarLander_PolicyBased.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/pabair/rl-course-ss21/blob/main/6_LunarLander_PolicyBased.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iCLrRFHSKl_5"
      },
      "source": [
        "# Lunar Lander with Cross-Entropy Method\n",
        "\n",
        "In this notebook we look at the lunar lander environment and solve it with the cross-entropy method."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "96dExX1TKm2m",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "59a0cc23-613d-4378-8de6-2b4d280e9fa9"
      },
      "source": [
        "!pip3 install box2d-py"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: box2d-py in /usr/local/lib/python3.6/dist-packages (2.3.8)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CZXskDwXKl_-"
      },
      "source": [
        "import gym\n",
        "import numpy as np\n",
        "\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "import torch.nn.functional as F\n",
        "from torch.distributions import Categorical\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "from collections import deque\n",
        "\n",
        "torch.manual_seed(1)\n",
        "np.random.seed(1)"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xhZ0fzBkKmAA"
      },
      "source": [
        "# Neural Network\n",
        "\n",
        "We define a simple neural network that generates the action scores based on a given state."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vWQr7TZgKmAB"
      },
      "source": [
        "class Net(nn.Module):\n",
        "    def __init__(self, obs_size, hidden_size, n_actions):\n",
        "        super(Net, self).__init__()\n",
        "        self.fc1 = nn.Linear(obs_size, hidden_size)\n",
        "        self.fc2 = nn.Linear(hidden_size, n_actions)\n",
        "        \n",
        "    def forward(self, x):\n",
        "        x = F.relu(self.fc1(x))\n",
        "        return self.fc2(x)"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6zFMlVViKmAE"
      },
      "source": [
        "# Generate Episodes\n",
        "\n",
        "We generate a batch of episodes and remember the traversed states, actions and rewards. To select the next action we use the output of the network. For this we first pass the scores through a softmax to get probabilites. In the second step we sampel from this distribution to get the next action to execute."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AIiayltZKmAF"
      },
      "source": [
        "def generate_batch(env, batch_size, t_max=5000):\n",
        "    \n",
        "    activation = nn.Softmax(dim=1)\n",
        "    batch_actions,batch_states, batch_rewards = [],[],[]\n",
        "    \n",
        "    for b in range(batch_size):\n",
        "        states,actions = [],[]\n",
        "        total_reward = 0\n",
        "        s = env.reset()\n",
        "        for t in range(t_max):\n",
        "            \n",
        "            s_v = torch.FloatTensor([s])\n",
        "            act_probs_v = activation(net(s_v))\n",
        "            act_probs = act_probs_v.data.numpy()[0]\n",
        "            a = np.random.choice(len(act_probs), p=act_probs)\n",
        "\n",
        "            new_s, r, done, info = env.step(a)\n",
        "\n",
        "            #record sessions like you did before\n",
        "            states.append(s)\n",
        "            actions.append(a)\n",
        "            total_reward += r\n",
        "\n",
        "            s = new_s\n",
        "            if done:\n",
        "                batch_actions.append(actions)\n",
        "                batch_states.append(states)\n",
        "                batch_rewards.append(total_reward)\n",
        "                break\n",
        "                \n",
        "    return batch_states, batch_actions, batch_rewards"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cvq2ZIvlKmAJ"
      },
      "source": [
        "# Training\n",
        "\n",
        "In the training step, we first use the neural network to generate a batch of episodes and then use the state-action pairs to improve the neural network."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pFUzEnaDKmAJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a5344f76-e542-4566-808e-8864fcdd4f09"
      },
      "source": [
        "batch_size = 100\n",
        "session_size = 100\n",
        "percentile = 80\n",
        "hidden_size = 200\n",
        "completion_score = 100\n",
        "learning_rate = 0.01\n",
        "\n",
        "env = gym.make(\"LunarLander-v2\")\n",
        "n_states = env.observation_space.shape[0]\n",
        "n_actions = env.action_space.n\n",
        "\n",
        "#neural network\n",
        "net = Net(n_states, hidden_size, n_actions)\n",
        "#loss function\n",
        "objective = nn.CrossEntropyLoss()\n",
        "#optimisation function\n",
        "optimizer = optim.Adam(params=net.parameters(), lr=learning_rate)\n",
        "\n",
        "for i in range(session_size):\n",
        "    #generate new sessions\n",
        "    states, actions, rewards = generate_batch(env, batch_size, t_max=500)\n",
        "\n",
        "    # TODO-1: here we need to filter out episodes that are not good\n",
        "\n",
        "\n",
        "    # train on the states using actions as targets\n",
        "    for s_i in range(len(states)):\n",
        "      optimizer.zero_grad()\n",
        "      tensor_states = torch.FloatTensor(states[s_i])\n",
        "      tensor_actions = torch.LongTensor(actions[s_i])\n",
        "      action_scores_v = net(tensor_states)\n",
        "      loss_v = objective(action_scores_v, tensor_actions)\n",
        "      loss_v.backward()\n",
        "      optimizer.step()\n",
        "\n",
        "    #show results\n",
        "    mean_reward = np.mean(rewards)\n",
        "    print(\"%d: loss=%.3f, reward_mean=%.1f\" % (\n",
        "            i, loss_v.item(), mean_reward))\n",
        "    \n",
        "    #check if \n",
        "    if np.mean(rewards)> completion_score:\n",
        "        print(\"Environment has been successfullly completed!\")"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0: loss=1.386, reward_mean=-177.3\n",
            "1: loss=1.381, reward_mean=-175.4\n",
            "2: loss=1.326, reward_mean=-209.5\n",
            "3: loss=1.375, reward_mean=-173.4\n",
            "4: loss=1.394, reward_mean=-181.5\n",
            "5: loss=1.395, reward_mean=-193.7\n",
            "6: loss=1.372, reward_mean=-144.9\n",
            "7: loss=1.375, reward_mean=-168.2\n",
            "8: loss=1.351, reward_mean=-168.4\n",
            "9: loss=1.236, reward_mean=-178.6\n",
            "10: loss=1.353, reward_mean=-185.9\n",
            "11: loss=1.341, reward_mean=-192.2\n",
            "12: loss=1.399, reward_mean=-190.1\n",
            "13: loss=1.367, reward_mean=-169.7\n",
            "14: loss=1.357, reward_mean=-198.1\n",
            "15: loss=1.354, reward_mean=-172.0\n",
            "16: loss=0.800, reward_mean=-195.9\n",
            "17: loss=1.392, reward_mean=-172.7\n",
            "18: loss=1.345, reward_mean=-169.3\n",
            "19: loss=1.389, reward_mean=-178.3\n",
            "20: loss=1.358, reward_mean=-195.7\n",
            "21: loss=1.365, reward_mean=-170.6\n",
            "22: loss=1.330, reward_mean=-192.1\n",
            "23: loss=1.358, reward_mean=-183.1\n",
            "24: loss=1.379, reward_mean=-177.9\n",
            "25: loss=1.378, reward_mean=-183.5\n",
            "26: loss=1.387, reward_mean=-189.3\n",
            "27: loss=1.357, reward_mean=-190.0\n",
            "28: loss=1.393, reward_mean=-196.6\n",
            "29: loss=1.365, reward_mean=-186.3\n",
            "30: loss=1.373, reward_mean=-178.2\n",
            "31: loss=1.339, reward_mean=-187.0\n",
            "32: loss=1.339, reward_mean=-175.4\n",
            "33: loss=1.363, reward_mean=-174.2\n",
            "34: loss=1.377, reward_mean=-164.5\n",
            "35: loss=1.337, reward_mean=-188.6\n",
            "36: loss=1.306, reward_mean=-191.9\n",
            "37: loss=1.273, reward_mean=-182.5\n",
            "38: loss=1.339, reward_mean=-171.8\n",
            "39: loss=1.292, reward_mean=-196.1\n",
            "40: loss=1.308, reward_mean=-197.0\n",
            "41: loss=1.342, reward_mean=-201.8\n",
            "42: loss=1.285, reward_mean=-187.7\n",
            "43: loss=1.284, reward_mean=-207.1\n",
            "44: loss=1.252, reward_mean=-208.9\n",
            "45: loss=1.257, reward_mean=-200.1\n",
            "46: loss=1.322, reward_mean=-214.1\n",
            "47: loss=1.272, reward_mean=-216.5\n",
            "48: loss=1.157, reward_mean=-228.2\n",
            "49: loss=1.246, reward_mean=-247.2\n",
            "50: loss=1.238, reward_mean=-273.8\n",
            "51: loss=1.230, reward_mean=-255.2\n",
            "52: loss=1.275, reward_mean=-281.1\n",
            "53: loss=1.363, reward_mean=-251.4\n",
            "54: loss=1.238, reward_mean=-245.1\n",
            "55: loss=1.370, reward_mean=-237.9\n",
            "56: loss=1.135, reward_mean=-256.8\n",
            "57: loss=1.327, reward_mean=-263.0\n",
            "58: loss=1.279, reward_mean=-234.4\n",
            "59: loss=1.261, reward_mean=-258.0\n",
            "60: loss=1.169, reward_mean=-274.4\n",
            "61: loss=1.293, reward_mean=-250.3\n",
            "62: loss=1.256, reward_mean=-245.6\n",
            "63: loss=1.272, reward_mean=-211.2\n",
            "64: loss=1.325, reward_mean=-256.7\n",
            "65: loss=1.274, reward_mean=-250.1\n",
            "66: loss=1.261, reward_mean=-269.6\n",
            "67: loss=1.198, reward_mean=-240.6\n",
            "68: loss=1.268, reward_mean=-241.2\n",
            "69: loss=1.352, reward_mean=-255.9\n",
            "70: loss=0.929, reward_mean=-256.3\n",
            "71: loss=1.209, reward_mean=-261.1\n",
            "72: loss=1.283, reward_mean=-234.1\n",
            "73: loss=1.337, reward_mean=-235.8\n",
            "74: loss=1.254, reward_mean=-266.5\n",
            "75: loss=1.270, reward_mean=-249.6\n",
            "76: loss=1.182, reward_mean=-252.1\n",
            "77: loss=1.155, reward_mean=-273.4\n",
            "78: loss=1.054, reward_mean=-283.0\n",
            "79: loss=1.164, reward_mean=-288.6\n",
            "80: loss=1.276, reward_mean=-309.7\n",
            "81: loss=1.223, reward_mean=-301.2\n",
            "82: loss=1.207, reward_mean=-290.5\n",
            "83: loss=1.232, reward_mean=-294.4\n",
            "84: loss=1.229, reward_mean=-285.2\n",
            "85: loss=1.076, reward_mean=-285.6\n",
            "86: loss=1.164, reward_mean=-288.4\n",
            "87: loss=1.121, reward_mean=-309.5\n",
            "88: loss=1.099, reward_mean=-317.8\n",
            "89: loss=1.104, reward_mean=-321.9\n",
            "90: loss=1.148, reward_mean=-298.0\n",
            "91: loss=1.087, reward_mean=-338.9\n",
            "92: loss=1.230, reward_mean=-337.5\n",
            "93: loss=1.000, reward_mean=-339.7\n",
            "94: loss=0.959, reward_mean=-362.7\n",
            "95: loss=1.088, reward_mean=-363.0\n",
            "96: loss=1.060, reward_mean=-362.9\n",
            "97: loss=1.251, reward_mean=-353.1\n",
            "98: loss=1.180, reward_mean=-373.5\n",
            "99: loss=1.102, reward_mean=-371.3\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dA06LBJ5JgZH"
      },
      "source": [
        "# Evaluation\n",
        "\n",
        "The following code enables Colab to record sessions (not needed when using executing code locally).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yUlnIIwC9AyV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1ec2a831-8c21-424c-a4fd-287207974da0"
      },
      "source": [
        "!apt-get install -y xvfb x11-utils\n",
        "\n",
        "!pip install pyvirtualdisplay==0.2.* \\\n",
        "             PyOpenGL==3.1.* \\\n",
        "             PyOpenGL-accelerate==3.1.*\n",
        "\n",
        "!pip install gym[box2d]==0.17.*\n",
        "\n",
        "import pyvirtualdisplay\n",
        "\n",
        "_display = pyvirtualdisplay.Display(visible=False, size=(1400, 900))\n",
        "_ = _display.start()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "The following additional packages will be installed:\n",
            "  libxxf86dga1\n",
            "Suggested packages:\n",
            "  mesa-utils\n",
            "The following NEW packages will be installed:\n",
            "  libxxf86dga1 x11-utils xvfb\n",
            "0 upgraded, 3 newly installed, 0 to remove and 13 not upgraded.\n",
            "Need to get 993 kB of archives.\n",
            "After this operation, 2,981 kB of additional disk space will be used.\n",
            "Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxxf86dga1 amd64 2:1.1.4-1 [13.7 kB]\n",
            "Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11-utils amd64 7.7+3build1 [196 kB]\n",
            "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 xvfb amd64 2:1.19.6-1ubuntu4.8 [784 kB]\n",
            "Fetched 993 kB in 1s (850 kB/s)\n",
            "Selecting previously unselected package libxxf86dga1:amd64.\n",
            "(Reading database ... 146374 files and directories currently installed.)\n",
            "Preparing to unpack .../libxxf86dga1_2%3a1.1.4-1_amd64.deb ...\n",
            "Unpacking libxxf86dga1:amd64 (2:1.1.4-1) ...\n",
            "Selecting previously unselected package x11-utils.\n",
            "Preparing to unpack .../x11-utils_7.7+3build1_amd64.deb ...\n",
            "Unpacking x11-utils (7.7+3build1) ...\n",
            "Selecting previously unselected package xvfb.\n",
            "Preparing to unpack .../xvfb_2%3a1.19.6-1ubuntu4.8_amd64.deb ...\n",
            "Unpacking xvfb (2:1.19.6-1ubuntu4.8) ...\n",
            "Setting up xvfb (2:1.19.6-1ubuntu4.8) ...\n",
            "Setting up libxxf86dga1:amd64 (2:1.1.4-1) ...\n",
            "Setting up x11-utils (7.7+3build1) ...\n",
            "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...\n",
            "Processing triggers for libc-bin (2.27-3ubuntu1.3) ...\n",
            "/sbin/ldconfig.real: /usr/local/lib/python3.6/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link\n",
            "\n",
            "Collecting pyvirtualdisplay==0.2.*\n",
            "  Downloading https://files.pythonhosted.org/packages/69/ec/8221a07850d69fa3c57c02e526edd23d18c7c05d58ed103e3b19172757c1/PyVirtualDisplay-0.2.5-py2.py3-none-any.whl\n",
            "Requirement already satisfied: PyOpenGL==3.1.* in /usr/local/lib/python3.6/dist-packages (3.1.5)\n",
            "Collecting PyOpenGL-accelerate==3.1.*\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a2/3c/f42a62b7784c04b20f8b88d6c8ad04f4f20b0767b721102418aad94d8389/PyOpenGL-accelerate-3.1.5.tar.gz (538kB)\n",
            "\u001b[K     |████████████████████████████████| 542kB 6.4MB/s \n",
            "\u001b[?25hCollecting EasyProcess\n",
            "  Downloading https://files.pythonhosted.org/packages/48/3c/75573613641c90c6d094059ac28adb748560d99bd27ee6f80cce398f404e/EasyProcess-0.3-py2.py3-none-any.whl\n",
            "Building wheels for collected packages: PyOpenGL-accelerate\n",
            "  Building wheel for PyOpenGL-accelerate (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for PyOpenGL-accelerate: filename=PyOpenGL_accelerate-3.1.5-cp36-cp36m-linux_x86_64.whl size=1593666 sha256=61c0bf3be2e263880209d3e75c1a13587e886a16cf8f57d9b62c71cd146219f9\n",
            "  Stored in directory: /root/.cache/pip/wheels/bd/21/77/99670ceca25fddb3c2b60a7ae44644b8253d1006e8ec417bcc\n",
            "Successfully built PyOpenGL-accelerate\n",
            "Installing collected packages: EasyProcess, pyvirtualdisplay, PyOpenGL-accelerate\n",
            "Successfully installed EasyProcess-0.3 PyOpenGL-accelerate-3.1.5 pyvirtualdisplay-0.2.5\n",
            "Requirement already satisfied: gym[box2d]==0.17.* in /usr/local/lib/python3.6/dist-packages (0.17.3)\n",
            "Requirement already satisfied: cloudpickle<1.7.0,>=1.2.0 in /usr/local/lib/python3.6/dist-packages (from gym[box2d]==0.17.*) (1.3.0)\n",
            "Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.6/dist-packages (from gym[box2d]==0.17.*) (1.5.0)\n",
            "Requirement already satisfied: numpy>=1.10.4 in /usr/local/lib/python3.6/dist-packages (from gym[box2d]==0.17.*) (1.19.5)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from gym[box2d]==0.17.*) (1.4.1)\n",
            "Requirement already satisfied: box2d-py~=2.3.5; extra == \"box2d\" in /usr/local/lib/python3.6/dist-packages (from gym[box2d]==0.17.*) (2.3.8)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym[box2d]==0.17.*) (0.16.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bg_Cz4SwKJJm"
      },
      "source": [
        "Use the trained model to play and record one episode. The recorded video will be stored into the video-subfolder on disk.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_2KaeLMD9Dx7"
      },
      "source": [
        "# TODO-2: Play episode with agent and record it!"
      ],
      "execution_count": 7,
      "outputs": []
    }
  ]
}