lecture_chap5_homework_pub.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第5回講義 宿題"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 課題. Tensorflowを用いて, MNISTを多層パーセプトロン(MLP)で学習せよ"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 注意"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- homework関数を完成させて提出してください\n",
    "    - 訓練データはtrain_X, train_y, テストデータはtest_Xで与えられます\n",
    "    - train_Xとtrain_yをtrain_X, train_yとvalid_X, valid_yに分けるなどしてモデルを学習させてください\n",
    "    - test_Xに対して予想ラベルpred_yを作り, homework関数の戻り値としてください\n",
    "- pred_yのtest_yに対する精度(F値)で評価します\n",
    "- 全体の実行時間がiLect上で60分を超えないようにしてください\n",
    "- homework関数の外には何も書かないでください (必要なものは全てhomework関数に入れてください)\n",
    "- 解答提出時には Answer Cell の内容のみを提出してください"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- CNNは使わないでください"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**`tf` の以下のモジュールはこの回では使用できないように制限されています. 注意してください.**\n",
    "```python\n",
    "tf.app\n",
    "tf.compat\n",
    "tf.contrib\n",
    "tf.erros\n",
    "tf.gfile\n",
    "tf.graph_util\n",
    "tf.image\n",
    "tf.layers\n",
    "tf.logging\n",
    "tf.losses\n",
    "tf.metrics\n",
    "tf.python_io\n",
    "tf.resource_loader\n",
    "tf.saved_model\n",
    "tf.sdca\n",
    "tf.sets\n",
    "tf.summary\n",
    "tf.sysconfig\n",
    "tf.test\n",
    "tf.train\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "次のセルのhomework関数を完成させて提出してください"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Answer Cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def homework(train_X, train_y, test_X):\n",
    "    # WRITE ME!\n",
    "    import time\n",
    "    import matplotlib.pyplot as plt\n",
    "    %matplotlib inline\n",
    "    \n",
    "    #definition\n",
    "    \n",
    "    eps=0.1\n",
    "    #epoch_num=60\n",
    "    rng = np.random.RandomState(1234)\n",
    "    random_state = 42\n",
    "    \n",
    "    x = tf.placeholder(tf.float32, [None, 784])\n",
    "    t = tf.placeholder(tf.float32, [None, 10])\n",
    "    \n",
    "    w1 = tf.Variable(np.random.uniform(low=-0.08, high=0.08, size=(784, 100)).astype('float32'), name='w1')\n",
    "    b1 = tf.Variable(tf.zeros(100), name='b1')\n",
    "    \n",
    "    w2 = tf.Variable(np.random.uniform(low=-0.08, high=0.08, size=(100, 50)).astype('float32'), name='w2')\n",
    "    b2 = tf.Variable(tf.zeros(50), name='b2')\n",
    "    \n",
    "    w3 = tf.Variable(np.random.uniform(low=-0.08, high=0.08, size=(50, 10)).astype('float32'), name='w3')\n",
    "    b3 = tf.Variable(tf.zeros(10), name='b3')\n",
    "    \n",
    "    params = [w1, b1, w2, b2, w3, b3]\n",
    "    \n",
    "    #forward\n",
    "    \n",
    "    u1 = tf.matmul(x,w1)+b1\n",
    "    z1 = tf.nn.relu(u1)\n",
    "    \n",
    "    u2 = tf.matmul(z1,w2)+b2\n",
    "    z2 = tf.nn.relu(u2)\n",
    "    \n",
    "    u3 = tf.matmul(z2,w3)+b3\n",
    "    z3 = tf.nn.softmax(u3)\n",
    "    \n",
    "    y=z3\n",
    "    \n",
    "    # Step3. 誤差関数の定義\n",
    "    #cost = -tf.reduce_mean(tf.reduce_sum(t*tf.log(y)))\n",
    "    # tf.log(0)によるnanを防ぐ\n",
    "    cost = -tf.reduce_mean(tf.reduce_sum(t*tf.log(tf.clip_by_value(y, 1e-10, 1.0)),axis=1))\n",
    "    #cost = -np.log(y[0,np.nonzero(t)])\n",
    "    \n",
    "    #delta\n",
    "    gw1, gb1, gw2, gb2, gw3, gb3 = tf.gradients(cost, params)\n",
    "    updates = [\n",
    "        w1.assign_add(-eps*gw1), # 勾配降下法\n",
    "        b1.assign_add(-eps*gb1),\n",
    "        w2.assign_add(-eps*gw2), # 勾配降下法\n",
    "        b2.assign_add(-eps*gb2),\n",
    "        w3.assign_add(-eps*gw3), # 勾配降下法\n",
    "        b3.assign_add(-eps*gb3),\n",
    "    ]\n",
    "\n",
    "    train = tf.group(*updates)\n",
    "    \n",
    "    valid = tf.argmax(y, 1)\n",
    "    \n",
    "    start_time=time.time()\n",
    "    \n",
    "    #Online learning\n",
    "    \n",
    "    #cost_plt =[]\n",
    "    #sess = tf.Session()\n",
    "    #sess.run(tf.global_variables_initializer()) # 重みの初期化\n",
    "    #for epoch in range(epoch_num):\n",
    "    #    cost_arry = []\n",
    "    #    for x_in, y_in in zip(train_X, train_y):\n",
    "    #        x_in = x_in.reshape(1,len(x_in))\n",
    "    #        y_tmp = np.zeros(10).reshape(1, 10)\n",
    "    #        y_tmp[0,y_in] = 1.0\n",
    "    #        y_in=y_tmp\n",
    "    #        _cost, _ = sess.run([cost, train], feed_dict={x: x_in, t: y_in})\n",
    "    #        cost_arry.append(_cost)\n",
    "    #        \n",
    "    #    cost_plt.append(np.sum(cost_arry))\n",
    "    #    tttttt = time.time() - start\n",
    "    #    print (\"epoch\",epoch,\"time\",tttttt,\"cost\",cost_plt[len(cost_plt)-1])\n",
    "        #print(cost_plt)\n",
    "    #plt.plot(cost_plt)\n",
    "    \n",
    "    #batch learning\n",
    "    n_epochs = 300\n",
    "    batch_size = 100\n",
    "    n_batches = train_X.shape[0] // batch_size\n",
    "    \n",
    "    nb_classes = 10                                #train_y one-hot\n",
    "    targets = train_y.reshape(-1)\n",
    "    one_hot_targets = np.eye(nb_classes)[targets]\n",
    "    train_y = one_hot_targets\n",
    "    \n",
    "    # Step5. 学習\n",
    "    cost_plt =[]\n",
    "    with tf.Session() as sess:\n",
    "        print(\"start time\",start_time)\n",
    "        sess.run(tf.global_variables_initializer())\n",
    "        for epoch in range(n_epochs):\n",
    "            train_X, train_y = shuffle(train_X, train_y, random_state=random_state)\n",
    "            cost_arry = []\n",
    "            for i in range(n_batches):\n",
    "                start = i * batch_size\n",
    "                end = start + batch_size\n",
    "                _cost, _ = sess.run([cost,train], feed_dict={x: train_X[start:end], t: train_y[start:end]})\n",
    "                cost_arry.append(_cost)\n",
    "                \n",
    "            cost_plt.append(np.sum(cost_arry))\n",
    "            tttttt = time.time()-start_time\n",
    "            #predict\n",
    "            print (\"epoch\",epoch,\"time\",tttttt,\"cost\",cost_plt[len(cost_plt)-1])\n",
    "        pred_y = sess.run(valid, feed_dict={x: test_X})\n",
    "        print(\"time\",time.time()-start_time)\n",
    "        plt.plot(cost_plt)\n",
    "    \n",
    "    # Step6. 予測\n",
    "    #pred_y = np.argmax(sess.run(y, feed_dict={x: test_X}),1)\n",
    "\n",
    "    #sess.close()\n",
    "    \n",
    "    \n",
    "    \n",
    "    return pred_y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 以下のvalidate_homework関数を用いてエラーが起きないか動作確認をして下さい。\n",
    "- 提出に際して、以下のscore_homework関数で60分で実行が終わることを確認して下さい。\n",
    "- 評価は以下のscore_homework関数で行われますが、random_stateの値は変更されます。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Checker Cell (for student)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "app",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-9-40a7a46cbe88>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m del [\n\u001b[0;32m---> 10\u001b[0;31m         \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mapp\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     11\u001b[0m         \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompat\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m         \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontrib\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: app"
     ]
    }
   ],
   "source": [
    "from sklearn.utils import shuffle\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.datasets import fetch_mldata\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "del [\n",
    "        tf.app,\n",
    "        tf.compat,\n",
    "        tf.contrib,\n",
    "        tf.errors,\n",
    "        tf.gfile,\n",
    "        tf.graph_util,\n",
    "        tf.image,\n",
    "        tf.layers,\n",
    "        tf.logging,\n",
    "        tf.losses,\n",
    "        tf.metrics,\n",
    "        tf.python_io,\n",
    "        tf.resource_loader,\n",
    "        tf.saved_model,\n",
    "        tf.sdca,\n",
    "        tf.sets,\n",
    "        tf.summary,\n",
    "        tf.sysconfig,\n",
    "        tf.test,\n",
    "        tf.train\n",
    "    ]\n",
    "\n",
    "def load_mnist():\n",
    "    mnist = fetch_mldata('MNIST original')\n",
    "    mnist_X, mnist_y = shuffle(mnist.data.astype('float32'),\n",
    "                               mnist.target.astype('int32'), random_state=42)\n",
    "\n",
    "    mnist_X = mnist_X / 255.0\n",
    "\n",
    "    return train_test_split(mnist_X, mnist_y,\n",
    "                test_size=0.2,\n",
    "                random_state=42)\n",
    "\n",
    "def validate_homework():\n",
    "    train_X, test_X, train_y, test_y = load_mnist()\n",
    "\n",
    "    # validate for small dataset\n",
    "    train_X_mini = train_X[:1000]\n",
    "    train_y_mini = train_y[:1000]\n",
    "    test_X_mini = test_X[:1000]\n",
    "    test_y_mini = test_y[:1000]\n",
    "\n",
    "    pred_y = homework(train_X_mini, train_y_mini, test_X_mini)\n",
    "    print(f1_score(test_y_mini, pred_y, average='macro'))\n",
    "\n",
    "def score_homework():\n",
    "    train_X, test_X, train_y, test_y = load_mnist()\n",
    "    pred_y = homework(train_X, train_y, test_X)\n",
    "    print(f1_score(test_y, pred_y, average='macro'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start time 1496142103.7335393\n",
      "epoch 0 time 0.03874778747558594 cost 23.0029\n",
      "epoch 1 time 0.05200910568237305 cost 22.8406\n",
      "epoch 2 time 0.06470537185668945 cost 22.6444\n",
      "epoch 3 time 0.07740521430969238 cost 22.3509\n",
      "epoch 4 time 0.08967733383178711 cost 21.8872\n",
      "epoch 5 time 0.10206842422485352 cost 21.1054\n",
      "epoch 6 time 0.11456131935119629 cost 19.7764\n",
      "epoch 7 time 0.12697958946228027 cost 17.7838\n",
      "epoch 8 time 0.1396484375 cost 15.3881\n",
      "epoch 9 time 0.15201520919799805 cost 13.091\n",
      "epoch 10 time 0.16434764862060547 cost 11.1661\n",
      "epoch 11 time 0.17666172981262207 cost 9.64908\n",
      "epoch 12 time 0.18934202194213867 cost 8.58046\n",
      "epoch 13 time 0.20312857627868652 cost 7.66647\n",
      "epoch 14 time 0.21887898445129395 cost 7.07267\n",
      "epoch 15 time 0.23125433921813965 cost 6.37648\n",
      "epoch 16 time 0.24346113204956055 cost 5.74379\n",
      "epoch 17 time 0.25840234756469727 cost 5.35746\n",
      "epoch 18 time 0.27471184730529785 cost 5.23242\n",
      "epoch 19 time 0.2874634265899658 cost 4.50678\n",
      "epoch 20 time 0.300342321395874 cost 4.13645\n",
      "epoch 21 time 0.3127782344818115 cost 4.05886\n",
      "epoch 22 time 0.32508015632629395 cost 3.92072\n",
      "epoch 23 time 0.3376889228820801 cost 3.46436\n",
      "epoch 24 time 0.3503413200378418 cost 3.4111\n",
      "epoch 25 time 0.36279869079589844 cost 3.10896\n",
      "epoch 26 time 0.37520360946655273 cost 2.97501\n",
      "epoch 27 time 0.3873169422149658 cost 2.83156\n",
      "epoch 28 time 0.40000224113464355 cost 2.60904\n",
      "epoch 29 time 0.4147953987121582 cost 2.47496\n",
      "epoch 30 time 0.4279978275299072 cost 2.53646\n",
      "epoch 31 time 0.4404630661010742 cost 2.28084\n",
      "epoch 32 time 0.4528317451477051 cost 2.27513\n",
      "epoch 33 time 0.4653949737548828 cost 2.02534\n",
      "epoch 34 time 0.477933406829834 cost 1.93091\n",
      "epoch 35 time 0.49062180519104004 cost 1.93616\n",
      "epoch 36 time 0.5043566226959229 cost 1.7776\n",
      "epoch 37 time 0.516442060470581 cost 1.68\n",
      "epoch 38 time 0.5287587642669678 cost 1.60077\n",
      "epoch 39 time 0.5411851406097412 cost 1.46583\n",
      "epoch 40 time 0.5535614490509033 cost 1.51555\n",
      "epoch 41 time 0.5658724308013916 cost 1.37193\n",
      "epoch 42 time 0.5784397125244141 cost 1.36155\n",
      "epoch 43 time 0.591381311416626 cost 1.21262\n",
      "epoch 44 time 0.603956937789917 cost 1.13446\n",
      "epoch 45 time 0.6165308952331543 cost 1.07394\n",
      "epoch 46 time 0.6309781074523926 cost 1.03705\n",
      "epoch 47 time 0.6440722942352295 cost 0.979025\n",
      "epoch 48 time 0.6566727161407471 cost 0.998826\n",
      "epoch 49 time 0.6688416004180908 cost 0.899388\n",
      "epoch 50 time 0.6812219619750977 cost 0.877172\n",
      "epoch 51 time 0.6936745643615723 cost 0.810995\n",
      "epoch 52 time 0.7062640190124512 cost 0.805211\n",
      "epoch 53 time 0.7193398475646973 cost 0.706204\n",
      "epoch 54 time 0.7316715717315674 cost 0.70676\n",
      "epoch 55 time 0.7443368434906006 cost 0.625249\n",
      "epoch 56 time 0.7565875053405762 cost 0.611486\n",
      "epoch 57 time 0.7690327167510986 cost 0.595652\n",
      "epoch 58 time 0.7813506126403809 cost 0.562261\n",
      "epoch 59 time 0.7941160202026367 cost 0.551864\n",
      "epoch 60 time 0.8064839839935303 cost 0.517954\n",
      "epoch 61 time 0.8191876411437988 cost 0.519323\n",
      "epoch 62 time 0.8361115455627441 cost 0.467545\n",
      "epoch 63 time 0.8491048812866211 cost 0.472184\n",
      "epoch 64 time 0.8618209362030029 cost 0.436139\n",
      "epoch 65 time 0.8747644424438477 cost 0.422113\n",
      "epoch 66 time 0.8870744705200195 cost 0.415087\n",
      "epoch 67 time 0.8995633125305176 cost 0.387001\n",
      "epoch 68 time 0.9124209880828857 cost 0.369121\n",
      "epoch 69 time 0.9249849319458008 cost 0.353896\n",
      "epoch 70 time 0.9375898838043213 cost 0.339871\n",
      "epoch 71 time 0.9500749111175537 cost 0.331809\n",
      "epoch 72 time 0.9622957706451416 cost 0.312257\n",
      "epoch 73 time 0.9746699333190918 cost 0.302302\n",
      "epoch 74 time 0.9867372512817383 cost 0.294425\n",
      "epoch 75 time 0.9993851184844971 cost 0.278751\n",
      "epoch 76 time 1.0120189189910889 cost 0.278288\n",
      "epoch 77 time 1.0257937908172607 cost 0.269858\n",
      "epoch 78 time 1.0413892269134521 cost 0.271383\n",
      "epoch 79 time 1.0543508529663086 cost 0.244066\n",
      "epoch 80 time 1.0671207904815674 cost 0.24077\n",
      "epoch 81 time 1.0792551040649414 cost 0.230331\n",
      "epoch 82 time 1.092029333114624 cost 0.220388\n",
      "epoch 83 time 1.1046783924102783 cost 0.215615\n",
      "epoch 84 time 1.1173794269561768 cost 0.210913\n",
      "epoch 85 time 1.130131721496582 cost 0.199975\n",
      "epoch 86 time 1.142725944519043 cost 0.199091\n",
      "epoch 87 time 1.1552233695983887 cost 0.188611\n",
      "epoch 88 time 1.1678595542907715 cost 0.188734\n",
      "epoch 89 time 1.180023193359375 cost 0.180384\n",
      "epoch 90 time 1.1924667358398438 cost 0.177075\n",
      "epoch 91 time 1.2052600383758545 cost 0.171809\n",
      "epoch 92 time 1.217782735824585 cost 0.167396\n",
      "epoch 93 time 1.2310519218444824 cost 0.16404\n",
      "epoch 94 time 1.2455356121063232 cost 0.160107\n",
      "epoch 95 time 1.2587783336639404 cost 0.156321\n",
      "epoch 96 time 1.2715444564819336 cost 0.151318\n",
      "epoch 97 time 1.284149408340454 cost 0.148861\n",
      "epoch 98 time 1.2971601486206055 cost 0.14753\n",
      "epoch 99 time 1.3097383975982666 cost 0.14035\n",
      "epoch 100 time 1.3222558498382568 cost 0.138635\n",
      "epoch 101 time 1.3350892066955566 cost 0.137092\n",
      "epoch 102 time 1.3477578163146973 cost 0.131445\n",
      "epoch 103 time 1.3605084419250488 cost 0.129353\n",
      "epoch 104 time 1.3729181289672852 cost 0.127135\n",
      "epoch 105 time 1.3853094577789307 cost 0.125264\n",
      "epoch 106 time 1.397613286972046 cost 0.121835\n",
      "epoch 107 time 1.4100008010864258 cost 0.118816\n",
      "epoch 108 time 1.422715425491333 cost 0.114755\n",
      "epoch 109 time 1.435462474822998 cost 0.114327\n",
      "epoch 110 time 1.4508240222930908 cost 0.111824\n",
      "epoch 111 time 1.4646406173706055 cost 0.110107\n",
      "epoch 112 time 1.4819655418395996 cost 0.106825\n",
      "epoch 113 time 1.494985818862915 cost 0.105435\n",
      "epoch 114 time 1.5079333782196045 cost 0.103864\n",
      "epoch 115 time 1.5203356742858887 cost 0.101292\n",
      "epoch 116 time 1.5327818393707275 cost 0.100529\n",
      "epoch 117 time 1.5452969074249268 cost 0.0984472\n",
      "epoch 118 time 1.557570219039917 cost 0.0954328\n",
      "epoch 119 time 1.5704360008239746 cost 0.093978\n",
      "epoch 120 time 1.5829789638519287 cost 0.0923369\n",
      "epoch 121 time 1.5954957008361816 cost 0.0928059\n",
      "epoch 122 time 1.6080780029296875 cost 0.0895773\n",
      "epoch 123 time 1.6204440593719482 cost 0.0874945\n",
      "epoch 124 time 1.633016586303711 cost 0.0869513\n",
      "epoch 125 time 1.645535945892334 cost 0.0852817\n",
      "epoch 126 time 1.6613996028900146 cost 0.0844636\n",
      "epoch 127 time 1.6741068363189697 cost 0.082611\n",
      "epoch 128 time 1.686783790588379 cost 0.0816646\n",
      "epoch 129 time 1.6994953155517578 cost 0.0800092\n",
      "epoch 130 time 1.7121474742889404 cost 0.0791832\n",
      "epoch 131 time 1.7245371341705322 cost 0.0774849\n",
      "epoch 132 time 1.7370262145996094 cost 0.0765817\n",
      "epoch 133 time 1.7490532398223877 cost 0.0748401\n",
      "epoch 134 time 1.761582612991333 cost 0.0743915\n",
      "epoch 135 time 1.774742841720581 cost 0.0729337\n",
      "epoch 136 time 1.7894721031188965 cost 0.0720979\n",
      "epoch 137 time 1.8045878410339355 cost 0.0706364\n",
      "epoch 138 time 1.820472240447998 cost 0.0704239\n",
      "epoch 139 time 1.8362114429473877 cost 0.0687656\n",
      "epoch 140 time 1.8540174961090088 cost 0.0681858\n",
      "epoch 141 time 1.8673138618469238 cost 0.0673606\n",
      "epoch 142 time 1.8845770359039307 cost 0.0664356\n",
      "epoch 143 time 1.9010510444641113 cost 0.0660822\n",
      "epoch 144 time 1.916410207748413 cost 0.0644798\n",
      "epoch 145 time 1.929990530014038 cost 0.0639888\n",
      "epoch 146 time 1.9458205699920654 cost 0.0628442\n",
      "epoch 147 time 1.9596517086029053 cost 0.0622455\n",
      "epoch 148 time 1.9749042987823486 cost 0.0616345\n",
      "epoch 149 time 1.9881000518798828 cost 0.0601772\n",
      "epoch 150 time 2.001129627227783 cost 0.0596268\n",
      "epoch 151 time 2.0140116214752197 cost 0.0588642\n",
      "epoch 152 time 2.0271034240722656 cost 0.0585574\n",
      "epoch 153 time 2.042771816253662 cost 0.0576294\n",
      "epoch 154 time 2.0604476928710938 cost 0.0571855\n",
      "epoch 155 time 2.0773491859436035 cost 0.0560786\n",
      "epoch 156 time 2.0916333198547363 cost 0.0555652\n",
      "epoch 157 time 2.1060967445373535 cost 0.0548045\n",
      "epoch 158 time 2.1210172176361084 cost 0.0540455\n",
      "epoch 159 time 2.1357181072235107 cost 0.0537165\n",
      "epoch 160 time 2.1497936248779297 cost 0.0527054\n",
      "epoch 161 time 2.164665699005127 cost 0.0525417\n",
      "epoch 162 time 2.1793549060821533 cost 0.051845\n",
      "epoch 163 time 2.194354772567749 cost 0.0508748\n",
      "epoch 164 time 2.2100894451141357 cost 0.0503857\n",
      "epoch 165 time 2.225278854370117 cost 0.0498425\n",
      "epoch 166 time 2.238436222076416 cost 0.0493447\n",
      "epoch 167 time 2.2513582706451416 cost 0.0487386\n",
      "epoch 168 time 2.2658016681671143 cost 0.0483098\n",
      "epoch 169 time 2.2816665172576904 cost 0.0476742\n",
      "epoch 170 time 2.294955253601074 cost 0.0475353\n",
      "epoch 171 time 2.3075690269470215 cost 0.0469454\n",
      "epoch 172 time 2.3200767040252686 cost 0.0464123\n",
      "epoch 173 time 2.3327255249023438 cost 0.045897\n",
      "epoch 174 time 2.3448948860168457 cost 0.0452149\n",
      "epoch 175 time 2.3574533462524414 cost 0.0450625\n",
      "epoch 176 time 2.369731903076172 cost 0.0445365\n",
      "epoch 177 time 2.3821325302124023 cost 0.0440591\n",
      "epoch 178 time 2.3950753211975098 cost 0.0438117\n",
      "epoch 179 time 2.407304286956787 cost 0.0430077\n",
      "epoch 180 time 2.419710159301758 cost 0.0428246\n",
      "epoch 181 time 2.432295083999634 cost 0.0421958\n",
      "epoch 182 time 2.445164203643799 cost 0.0420619\n",
      "epoch 183 time 2.458157777786255 cost 0.0415203\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 184 time 2.4717814922332764 cost 0.0412309\n",
      "epoch 185 time 2.485333204269409 cost 0.0409037\n",
      "epoch 186 time 2.513293981552124 cost 0.040266\n",
      "epoch 187 time 2.5296499729156494 cost 0.0398368\n",
      "epoch 188 time 2.547183036804199 cost 0.0394436\n",
      "epoch 189 time 2.5644145011901855 cost 0.0392672\n",
      "epoch 190 time 2.5799028873443604 cost 0.0390097\n",
      "epoch 191 time 2.599092721939087 cost 0.038589\n",
      "epoch 192 time 2.616260528564453 cost 0.0381521\n",
      "epoch 193 time 2.637242078781128 cost 0.0379192\n",
      "epoch 194 time 2.655317783355713 cost 0.0375002\n",
      "epoch 195 time 2.671840190887451 cost 0.0373607\n",
      "epoch 196 time 2.6863479614257812 cost 0.0369575\n",
      "epoch 197 time 2.704854965209961 cost 0.036428\n",
      "epoch 198 time 2.7183308601379395 cost 0.0362131\n",
      "epoch 199 time 2.7310822010040283 cost 0.0359436\n",
      "epoch 200 time 2.743396759033203 cost 0.0355256\n",
      "epoch 201 time 2.755817174911499 cost 0.0353903\n",
      "epoch 202 time 2.7680702209472656 cost 0.0350098\n",
      "epoch 203 time 2.780555248260498 cost 0.0347421\n",
      "epoch 204 time 2.793365240097046 cost 0.0346022\n",
      "epoch 205 time 2.8060410022735596 cost 0.0340259\n",
      "epoch 206 time 2.8185412883758545 cost 0.0337799\n",
      "epoch 207 time 2.8312301635742188 cost 0.0336284\n",
      "epoch 208 time 2.8441927433013916 cost 0.0332815\n",
      "epoch 209 time 2.857292413711548 cost 0.0329405\n",
      "epoch 210 time 2.8701529502868652 cost 0.0327336\n",
      "epoch 211 time 2.8857219219207764 cost 0.0324171\n",
      "epoch 212 time 2.899183988571167 cost 0.0323343\n",
      "epoch 213 time 2.91166353225708 cost 0.0319173\n",
      "epoch 214 time 2.9242961406707764 cost 0.0317779\n",
      "epoch 215 time 2.9366464614868164 cost 0.0314683\n",
      "epoch 216 time 2.949105739593506 cost 0.0311228\n",
      "epoch 217 time 2.9617819786071777 cost 0.0310945\n",
      "epoch 218 time 2.974321126937866 cost 0.030669\n",
      "epoch 219 time 2.987161159515381 cost 0.0306272\n",
      "epoch 220 time 2.9996984004974365 cost 0.0302732\n",
      "epoch 221 time 3.0123345851898193 cost 0.0299713\n",
      "epoch 222 time 3.0247135162353516 cost 0.0297751\n",
      "epoch 223 time 3.0374417304992676 cost 0.0296477\n",
      "epoch 224 time 3.050023317337036 cost 0.0293779\n",
      "epoch 225 time 3.0623021125793457 cost 0.0293465\n",
      "epoch 226 time 3.0745596885681152 cost 0.0288979\n",
      "epoch 227 time 3.088491201400757 cost 0.0287003\n",
      "epoch 228 time 3.102450370788574 cost 0.0285712\n",
      "epoch 229 time 3.1153564453125 cost 0.0282773\n",
      "epoch 230 time 3.1283376216888428 cost 0.0281874\n",
      "epoch 231 time 3.14115571975708 cost 0.027954\n",
      "epoch 232 time 3.1538445949554443 cost 0.0277835\n",
      "epoch 233 time 3.1664044857025146 cost 0.0274588\n",
      "epoch 234 time 3.1787538528442383 cost 0.0272727\n",
      "epoch 235 time 3.1908035278320312 cost 0.0271055\n",
      "epoch 236 time 3.2032065391540527 cost 0.0271387\n",
      "epoch 237 time 3.2155158519744873 cost 0.026778\n",
      "epoch 238 time 3.228395462036133 cost 0.0265922\n",
      "epoch 239 time 3.2410037517547607 cost 0.0263889\n",
      "epoch 240 time 3.2538349628448486 cost 0.0262707\n",
      "epoch 241 time 3.2664825916290283 cost 0.0260106\n",
      "epoch 242 time 3.279172420501709 cost 0.0258591\n",
      "epoch 243 time 3.293667793273926 cost 0.0256987\n",
      "epoch 244 time 3.3068511486053467 cost 0.0255756\n",
      "epoch 245 time 3.319593906402588 cost 0.0253626\n",
      "epoch 246 time 3.332106828689575 cost 0.0251652\n",
      "epoch 247 time 3.3446404933929443 cost 0.0250043\n",
      "epoch 248 time 3.3570122718811035 cost 0.0248786\n",
      "epoch 249 time 3.3697383403778076 cost 0.0246862\n",
      "epoch 250 time 3.3821353912353516 cost 0.0245034\n",
      "epoch 251 time 3.3942272663116455 cost 0.0244766\n",
      "epoch 252 time 3.406367540359497 cost 0.0242476\n",
      "epoch 253 time 3.4185988903045654 cost 0.0241055\n",
      "epoch 254 time 3.430891275405884 cost 0.0239864\n",
      "epoch 255 time 3.4434938430786133 cost 0.0238653\n",
      "epoch 256 time 3.455983877182007 cost 0.0236525\n",
      "epoch 257 time 3.4680984020233154 cost 0.0234361\n",
      "epoch 258 time 3.4801995754241943 cost 0.0234058\n",
      "epoch 259 time 3.4927451610565186 cost 0.0231567\n",
      "epoch 260 time 3.512613534927368 cost 0.0230558\n",
      "epoch 261 time 3.5253758430480957 cost 0.0228873\n",
      "epoch 262 time 3.538015365600586 cost 0.0228544\n",
      "epoch 263 time 3.550549268722534 cost 0.0226468\n",
      "epoch 264 time 3.563032865524292 cost 0.0224639\n",
      "epoch 265 time 3.5754196643829346 cost 0.022337\n",
      "epoch 266 time 3.5879149436950684 cost 0.0222365\n",
      "epoch 267 time 3.6003551483154297 cost 0.022082\n",
      "epoch 268 time 3.6125800609588623 cost 0.0220006\n",
      "epoch 269 time 3.625337600708008 cost 0.0218742\n",
      "epoch 270 time 3.638080358505249 cost 0.0217128\n",
      "epoch 271 time 3.6504452228546143 cost 0.0216153\n",
      "epoch 272 time 3.6629419326782227 cost 0.0215218\n",
      "epoch 273 time 3.6753921508789062 cost 0.0212792\n",
      "epoch 274 time 3.687910556793213 cost 0.021212\n",
      "epoch 275 time 3.700356960296631 cost 0.0210554\n",
      "epoch 276 time 3.713284730911255 cost 0.0210046\n",
      "epoch 277 time 3.7274680137634277 cost 0.0208202\n",
      "epoch 278 time 3.7404139041900635 cost 0.0206676\n",
      "epoch 279 time 3.7528469562530518 cost 0.0206094\n",
      "epoch 280 time 3.7652530670166016 cost 0.0204909\n",
      "epoch 281 time 3.778116464614868 cost 0.0203564\n",
      "epoch 282 time 3.790526866912842 cost 0.0202543\n",
      "epoch 283 time 3.80292010307312 cost 0.0200766\n",
      "epoch 284 time 3.815516710281372 cost 0.0201234\n",
      "epoch 285 time 3.828005790710449 cost 0.0199112\n",
      "epoch 286 time 3.8407435417175293 cost 0.0198879\n",
      "epoch 287 time 3.8536276817321777 cost 0.019677\n",
      "epoch 288 time 3.867097854614258 cost 0.0196663\n",
      "epoch 289 time 3.879223108291626 cost 0.0194947\n",
      "epoch 290 time 3.8922462463378906 cost 0.0193755\n",
      "epoch 291 time 3.904552936553955 cost 0.0192968\n",
      "epoch 292 time 3.918449878692627 cost 0.0191588\n",
      "epoch 293 time 3.9329147338867188 cost 0.0190969\n",
      "epoch 294 time 3.9489152431488037 cost 0.0189905\n",
      "epoch 295 time 3.9617958068847656 cost 0.018912\n",
      "epoch 296 time 3.974590301513672 cost 0.0188322\n",
      "epoch 297 time 3.9874374866485596 cost 0.0187222\n",
      "epoch 298 time 3.999967098236084 cost 0.0186052\n",
      "epoch 299 time 4.012726068496704 cost 0.0184725\n",
      "time 4.018543004989624\n",
      "0.856722237166\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGAZJREFUeJzt3X1wHHd9x/H393Q6PT9asi3HduyEPDkhOEFNXMhQQgiE\nkKlDCzTQAbeFptMCw0OBpgMtdDpQoBM6MGWSCSQkMGl4pgk0PIQQSIPBjpzajo1tbOfJViRbfpJk\nW5Il3a9/7Eo6C50l607a299+XjOa2/vd7u13tfbnfvrt3q455xARkfhLRV2AiIgUhwJdRMQTCnQR\nEU8o0EVEPKFAFxHxhAJdRMQTCnQREU8o0EVEPKFAFxHxRHo+V9bS0uJWrFgxn6sUEYm9TZs2HXLO\ntU4337wG+ooVK+jo6JjPVYqIxJ6ZPT+T+TTkIiLiCQW6iIgnFOgiIp5QoIuIeEKBLiLiCQW6iIgn\nFOgiIp6IRaBvev4od/xib9RliIiUtFgE+g+2vMhnf7yTx3YejLoUEZGSFYtAv+0NF3Px4jo+/O0t\n9J4cjrocEZGSFItArywv4/a3voyjJ0/x7z/dGXU5IiIlKRaBDnDpkgb+/OpzeWDjPg70DUZdjohI\nyYlNoAO865qVjGYd33pyX9SliIiUnFgF+oqWGq55SQvfeHIfzrmoyxERKSmxCnSAmy5vo/PYAHt7\njkddiohISYldoL/i/BYA1u89HHElIiKlJXaBvqy5inMaq/i1Al1E5DSxC3QzY815C/jNM4c1ji4i\nkiN2gQ5w+dIGjp4cpqd/KOpSRERKRiwD/bzWGgD29pyIuBIRkdIRy0Bf2RIE+jOHdKaLiMiYWAb6\nkoYqKstTPKseuojIuFgGeiplrFhQwzOHFOgiImNiGegA57fW8oy+XCQiMi62gb6ipZp9RwcYGc1G\nXYqISEmIbaAvqq9kNOs4cvJU1KWIiJSE2AZ6S20FAIf6FegiIuBDoB/Xl4tERCDWgZ4BFOgiImPi\nG+h1QQ/98HENuYiIwAwC3cyWmdljZvZbM9tuZu8P25vN7BEz2x0+Ns19uRPqKtJk0in10EVEQjPp\noY8Af++cWwWsAd5jZquA24BHnXMXAI+Gz+eNmdFSk6FHgS4iAswg0J1zXc65p8LpfmAHcA6wFrgv\nnO0+4Oa5KjKflroKDmnIRUQEOMsxdDNbAVwBbAAWOee6wpe6gUVFrWwGWmorOKweuogIcBaBbma1\nwHeBDzjn+nJfc8GdJqa824SZ3WpmHWbW0dPTU1Cxk7XUZjSGLiISmlGgm1k5QZjf75z7Xth8wMza\nwtfbgINTLeucu8s51+6ca29tbS1GzeOCHvopslnduUhEZCZnuRhwN7DDOff5nJceAtaF0+uAB4tf\n3pk1VWcYyTpOnBqZ71WLiJScmfTQXwm8A3iNmW0Of24EPgNcb2a7gdeGz+dVXWUagL5BBbqISHq6\nGZxzTwCW5+XrilvO2amrLAegf3AYqIqyFBGRyMX2m6IA9VXB51G/eugiIvEO9NN76CIiyRbzQA/H\n0AfUQxcR8SLQ1UMXEYl5oNeHQy46y0VEJOaBXpFOUV5mOigqIkLMA93MqK8s15CLiAgxD3QIxtE1\n5CIi4kWgq4cuIgJeBHpaY+giIngT6Oqhi4jEPtCDg6LqoYuIxD7Q6yrL6RtQD11ExINAT3Pi1Cij\nusmFiCScF4EOcHxIwy4ikmyxD/SaiiDQTyjQRSThFOgiIp6IfaDXVpQBGnIREYl9oNdkxnrooxFX\nIiISrfgHeoUOioqIgAeBXqsxdBERwINAHz8oekqBLiLJFvtAr9WQi4gI4EGgV5anSBmc1EFREUm4\n2Ae6mVFTkVYPXUQSL/aBDsGwiw6KikjSeRHo1ZkyHRQVkcTzItBrK9Ic1xi6iCScF4FeoyEXEREF\nuoiIL7wI9Fqd5SIi4keg11SUqYcuIonnSaCndbVFEUk8LwK9NpPm1GiWUyPZqEsREYnMtIFuZveY\n2UEz25bT9kkz6zSzzeHPjXNb5pmNXaDrpM5FF5EEm0kP/V7ghina/8M5tzr8ebi4ZZ0dXaBLRGQG\nge6cexw4Mg+1zNrEfUU1ji4iyVXIGPp7zWxrOCTTVLSKZqFG9xUVEZl1oN8BnA+sBrqA2/PNaGa3\nmlmHmXX09PTMcnVnprsWiYjMMtCdcwecc6POuSzwZeCqM8x7l3Ou3TnX3traOts6z6hGgS4iMrtA\nN7O2nKdvArblm3c+6KCoiAikp5vBzB4AXg20mNl+4BPAq81sNeCA54C/mcMap1WdCcbQ1UMXkSSb\nNtCdc2+bovnuOahl1iZuFK2zXEQkubz4pmhFOkU6ZRpyEZFE8yLQx+4rqiEXEUkyLwIdxu4rqiEX\nEUkubwJdl9AVkaTzKNDTulG0iCSaN4GuuxaJSNJ5E+g1GR0UFZFk8yfQdVBURBLOm0CvrSjTkIuI\nJJo3gT52HrpzLupSREQi4VWgj2QdQ7qvqIgklDeBrmuii0jSeRPo9VVBoPcPKtBFJJm8CfS6inIA\n+gaHI65ERCQa3gR6fVUY6APqoYtIMnkU6MGQi3roIpJU3gR6XWXQQ+9XoItIQnkT6PWVYQ9dQy4i\nklDeBHpNJk3KNOQiIsnlTaCnUkZtRZq+AQW6iCSTN4EOwZkuOg9dRJLKr0CvLNeQi4gkll+BXpXW\nQVERSSyvAr1OPXQRSTCvAr2+slwHRUUksfwK9Kq0DoqKSGJ5Feh1leX0D40wmtVNLkQkebwK9LFv\ni+rr/yKSRF4FemN1BoBejaOLSAJ5FejNNcEFuo6cOBVxJSIi88+rQG8Ke+hHTyrQRSR5vAr05pog\n0I+c0JCLiCSPV4HeFAb6MfXQRSSBvAr0uoo06ZRpDF1EEmnaQDeze8zsoJlty2lrNrNHzGx3+Ng0\nt2XOjJnRWJ3RGLqIJNJMeuj3AjdMarsNeNQ5dwHwaPi8JDTXlKuHLiKJNG2gO+ceB45Mal4L3BdO\n3wfcXOS6Zq2pOsNRHRQVkQSa7Rj6IudcVzjdDSwqUj0Fa67JcERDLiKSQAUfFHXOOSDvxVPM7FYz\n6zCzjp6enkJXN62mmgxHNeQiIgk020A/YGZtAOHjwXwzOufucs61O+faW1tbZ7m6mWsOD4pmdYEu\nEUmY2Qb6Q8C6cHod8GBxyilcU02GrEM3uhCRxJnJaYsPAL8GLjKz/Wb2LuAzwPVmtht4bfi8JLTU\nBl8uOtg/FHElIiLzKz3dDM65t+V56boi11IUbQ1VAHT1DnLhorqIqxERmT9efVMUoK2hEoDu3oGI\nKxERmV/eBfqi+krMgh66iEiSeBfomXSKltoKuo4p0EUkWbwLdAiGXbr6FOgikizeBrrG0EUkaTwN\n9CoNuYhI4nga6JX0D43Qry8XiUiC+BnojRPnoouIJIWXgb60KQj0fUdORlyJiMj88TLQlzVVAwp0\nEUkWLwO9pTZDVXkZ+47qTBcRSQ4vA93MWNZcxQvqoYtIgngZ6BAMu2jIRUSSxN9Ab65m/9EBghsq\niYj4z9tAX9pUxfGhEY6d1LnoIpIM3gb6ea01AOw+eDziSkRE5oe3gX7ZkgYAnu7sjbgSEZH54W2g\nL6yvZGFdBdsU6CKSEN4GOsDlSxvUQxeRxPA60C87p4G9Pcc5MTQSdSkiInPO70Bf0oBzsLO7L+pS\nRETmnNeBfnFbHQA7u/sjrkREZO55HejnNFZRV5FmZ5cCXUT853WgmxkXLa5jl3roIpIAXgc6BMMu\nO7r7dAkAEfGe/4G+uJ7+wRFe1N2LRMRzCQj04MDoLp3pIiKe8z7QLwwDfYcOjIqI57wP9PrKcs5p\nrNKpiyLiPe8DHeCStjoNuYiI9xIR6BcvrmdvzwmGRkajLkVEZM4kI9Db6hjNOp2PLiJeS0SgX71y\nAWbw2M6eqEsREZkziQj01roKrljWyM92HIi6FBGROVNQoJvZc2b2tJltNrOOYhU1F65ftZinO3vZ\nc1DDLiLip2L00K91zq12zrUX4b3mzNrVS2iqLmfdPU/SqxtHi4iHEjHkArCksYovvu0KOo8N8Otn\nDkddjohI0RUa6A74qZltMrNbi1HQXHr5uU0AGnYRES+lC1z+Gudcp5ktBB4xs53OucdzZwiD/laA\n5cuXF7i6wlRn0pzTWMXug8cjrUNEZC4U1EN3znWGjweB7wNXTTHPXc65dudce2trayGrK4oLFtWy\n+4ACXUT8M+tAN7MaM6sbmwZeB2wrVmFz5YKFteztOc5oVtdHFxG/FNJDXwQ8YWZbgI3A/zjnflyc\nsubOSxbWMjSSZf/Rk1GXIiJSVLMeQ3fOPQO8rIi1zItL2uoBeOqFo5y7oCbiakREiicxpy2OuWxJ\nAwvrKvjJNn1rVET8krhAT6WM11+6mF/+roeBU7r6ooj4I3GBDvDGy9sYGB5l3Vc30tM/FHU5IiJF\nkchAX3PeAj735svZuv8Y7/5aB4PD6qmLSPwlMtAB3tq+jNvfspot+47xw61dUZcjIlKwxAY6wBsu\nW8yCmgzr9xyKuhQRkYIlOtBTKWPN+QtYv/cwzumLRiISb4kOdIBXnt9Cd98gm/cdi7oUEZGCJD7Q\nr724ldqKNH96x3p+paEXEYmxxAd6W0MVP//wH9FSW8HdTzwbdTkiIrOW+EAHWFhXyVval/KLXQfp\n6h2IuhwRkVlRoIf+rH05KTM++p2tDI9moy5HROSsKdBDyxdU8+k/eSn/u/sQn3hou856EZHYKfSO\nRV55a/synj10gjt+sZcrlzfx5pcvjbokEZEZUw99ko+87iJetqyR23+6S5cEEJFYUaBPkkoZ/3DD\nRXT1DvJvD+/gm0++oAt4iUgsaMhlCq84v4W/eMUK7l3/HABrVx/mC7dcEW1RIiLTUKDn8U83rWJl\nSw0bnz3CD7d2sXb1El594UJSKYu6NBGRKWnIJY+ylLHuFSv4+E2XUJYy/ureDj718I6oyxIRyUuB\nPo22hioe/dAf8caXtnH/huc5fFzj6SJSmhToM7CsuZoPXn8hQyNZPvitLfzstwc4PjQSdVkiIqdR\noM/QSxbW8qmbX8r6PYd499c6eOfdG/SNUhEpKQr0s/D2q5fz+Eev5V9vvoynXjjGR7+zlaERnasu\nIqVBZ7mcpSWNVbxjzbkcO3GK2x/5HTu7+/niLau5YFFd1KWJSMKphz5L77vuAr7yznZ6+gd5+1c2\n8LHvP82DmzujLktEEkyBXoDXrlrEf/31GgaHR7l/wwt86Ftb+NJje9h35GTUpYlIAtl8XlWwvb3d\ndXR0zNv65ktP/xDDo1lu/XoH2zr7WFRfwQ/edw0L6yqjLk1EPGBmm5xz7dPNpx56EbTWVbCksYqH\n3nMN//2eV9I7MMxb7/w163VLOxGZRwr0IkqljNXLGrn3L69iJOt4+1c28Mf/+QT3/upZtnX26iJf\nIjKnNOQyRwaHR3lg4wt876lOnu7sBaCuIs3Hb7qEay9ayMJ6DceIyMzMdMhFgT4PNj1/lP1HT3Ln\nL59hR1cfmXSK161axFUrm3nHmnMx0wW/RCQ/BXoJGhnNsrO7n3vXP8ev9hyiq3eQS9rqWb2sgY+8\n/mKaazJRlygiJUiBXuKcc3z1V8/xk+3dPPXCUUayjoV1FbxkYS3t5zazoqWay5c20lSdUdCLJJwC\nPUZ2dffzo21d7D86wLbOXnZ294+/ljL4gxXNNFaXc/XKBSxvrmZxQyVLGqtoqi7XcI1IAsw00Av6\n6r+Z3QB8ASgDvuKc+0wh75dUFy2u46LFE5cOODWSZffBfnZ197PrQD+/2XuY7S8O8JPtB05bLpNO\n0dZQGf5UnTa9OJxuqConXaaTmUSSYNaBbmZlwJeA64H9wJNm9pBz7rfFKi6pMukUly5p4NIlDeNt\nzjl6+od4sXeQ7t4BXjw2SHffIC8eG6C7d5CNzx7hQN8gI9nf/4urJlNGQ1U59VXl1FeGj1Xp8emG\nqnLqK9Pjr1dlyigvMzJlKcrLUmTS4WNZivJ00F6WMv11IFJiCumhXwXscc49A2Bm3wDWAgr0OWBm\nLKyvDE53XNY45TyjWcfh4xOh3907yLGBYfoGRugbHKZvYJjegWE6jw2woyt43j/L67qbMRHyZXZ6\n6I9/CNj49Fh7edieyfmgCJYL5k2ljJQZKWP8Q6PMyGk3ylKE7UYqRU57sFzucwvfZ6xt4n2D9zCC\n+c3AGGufaEuF80w1fyr8QEulJuaZmH/sPSeWG/u9jT8L1xVO5p0v93NzcvvE/JOW1YdtIhUS6OcA\n+3Ke7weuLqwcKURZavrQn2w06zg+GAR+70AQ8kMjWYZGsgyPTvycGnWcGmsLH4dGswyPuJx5shPz\njAbtQyNZjg+NhMtNtE28txtfTubGGYM/98NlivYpl530ITTVfOPrnlQHU74y6UMrb/tMl8n/YTb5\nw3Gq9873vpPfO9/m5Fvm0296KVetbM5bWzHM+eVzzexW4FaA5cuXz/Xq5CyVpYyG6nIaqstZFmEd\nzjlGso6sczgXfNBknSObhaxzjLpJz7PhfOPtjmzucm7iuQvnz7qcZcN1OgeO4H3H2oJRq+DRhcvk\nmz8bNOTMM8X84YkHY69PTLuc7Q/ed2J66vbc39fp7zU27U5bHufyvtdU7ZzW/vvryFtLznuNt3P6\n9k20n+708zLyLDNpoZm895mWyTN5+j45Q50zXSa3oaaibPKrRVdIoHfCaRmwNGw7jXPuLuAuCM5y\nKWB94jEzo7xMwwQihSjk9IcngQvMbKWZZYBbgIeKU5aIiJytWffQnXMjZvZe4CcEpy3e45zbXrTK\nRETkrBQ0hu6cexh4uEi1iIhIAfSNExERTyjQRUQ8oUAXEfGEAl1ExBMKdBERT8zr5XPNrAd4fpaL\ntwC+3HVZ21KatC2lSdsC5zrnWqebaV4DvRBm1jGT6wHHgbalNGlbSpO2ZeY05CIi4gkFuoiIJ+IU\n6HdFXUARaVtKk7alNGlbZig2Y+giInJmceqhi4jIGcQi0M3sBjPbZWZ7zOy2qOs5W2b2nJk9bWab\nzawjbGs2s0fMbHf42BR1nVMxs3vM7KCZbctpm7J2C3wx3E9bzezK6Co/XZ7t+KSZdYb7ZbOZ3Zjz\n2j+G27HLzF4fTdVTM7NlZvaYmf3WzLab2fvD9jjul3zbErt9Y2aVZrbRzLaE2/IvYftKM9sQ1vzN\n8HLjmFlF+HxP+PqKgosI7kpSuj8El+bdC5wHZIAtwKqo6zrLbXgOaJnU9jngtnD6NuCzUdeZp/ZX\nAVcC26arHbgR+BHBXbjWABuirn+a7fgk8OEp5l0V/jurAFaG//7Kot6GnPragCvD6Trgd2HNcdwv\n+bYldvsm/P3WhtPlwIbw9/0t4Jaw/U7gb8PpvwPuDKdvAb5ZaA1x6KGP34zaOXcKGLsZddytBe4L\np+8Dbo6wlrycc48DRyY156t9LfA1F/gN0GhmbfNT6Znl2Y581gLfcM4NOeeeBfYQ/DssCc65Lufc\nU+F0P7CD4B6/cdwv+bYln5LdN+Hv93j4tDz8ccBrgO+E7ZP3y9j++g5wnRV4d+84BPpUN6M+0w4v\nRQ74qZltCu+xCrDIOdcVTncDi6IpbVby1R7HffXecBjinpxhr9hsR/hn+hUEvcFY75dJ2wIx3Ddm\nVmZmm4GDwCMEf0Ecc86NhLPk1ju+LeHrvcCCQtYfh0D3wTXOuSuBNwDvMbNX5b7ogr+5Ynm6UZxr\nB+4AzgdWA13A7dGWc3bMrBb4LvAB51xf7mtx2y9TbEss941zbtQ5t5rgHstXARfP5/rjEOgzuhl1\nKXPOdYaPB4HvE+zoA2N/9oaPB6Or8Kzlqz1W+8o5dyD8D5gFvszEn+4lvx1mVk4QgPc7574XNsdy\nv0y1LXHeNwDOuWPAY8AfEgxxjd0dLrfe8W0JX28ADhey3jgEeqxvRm1mNWZWNzYNvA7YRrAN68LZ\n1gEPRlPhrOSr/SHgneFZFWuA3pwhgJIzaRz5TQT7BYLtuCU8C2ElcAGwcb7ryyccZ70b2OGc+3zO\nS7HbL/m2JY77xsxazawxnK4Cric4JvAY8OZwtsn7ZWx/vRn4efiX1exFfWR4hkePbyQ4+r0X+FjU\n9Zxl7ecRHJXfAmwfq59grOxRYDfwM6A56lrz1P8AwZ+8wwTjf+/KVzvBUf4vhfvpaaA96vqn2Y6v\nh3VuDf9zteXM/7FwO3YBb4i6/knbcg3BcMpWYHP4c2NM90u+bYndvgEuB/4vrHkb8M9h+3kEHzp7\ngG8DFWF7Zfh8T/j6eYXWoG+Kioh4Ig5DLiIiMgMKdBERTyjQRUQ8oUAXEfGEAl1ExBMKdBERTyjQ\nRUQ8oUAXEfHE/wOo22jGBnHCeAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f3a70360fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "validate_homework()\n",
    "#score_homework()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start time 1496142108.1020088\n",
      "epoch 0 time 0.7207388877868652 cost 413.101\n",
      "epoch 1 time 1.479499101638794 cost 144.251\n",
      "epoch 2 time 2.165708541870117 cost 104.12\n",
      "epoch 3 time 2.8440725803375244 cost 82.4562\n",
      "epoch 4 time 3.5276787281036377 cost 67.9612\n",
      "epoch 5 time 4.20912504196167 cost 56.6846\n",
      "epoch 6 time 5.003258228302002 cost 49.0914\n",
      "epoch 7 time 5.805946588516235 cost 43.1323\n",
      "epoch 8 time 6.536020040512085 cost 38.1202\n",
      "epoch 9 time 7.240007638931274 cost 33.7654\n",
      "epoch 10 time 7.928476810455322 cost 29.7394\n",
      "epoch 11 time 8.618265390396118 cost 26.5829\n",
      "epoch 12 time 9.308708190917969 cost 23.7253\n",
      "epoch 13 time 9.996573209762573 cost 21.6041\n",
      "epoch 14 time 10.688621520996094 cost 19.4012\n",
      "epoch 15 time 11.409627676010132 cost 17.1219\n",
      "epoch 16 time 12.118242979049683 cost 15.2361\n",
      "epoch 17 time 12.819770097732544 cost 13.8781\n",
      "epoch 18 time 13.505347728729248 cost 12.1757\n",
      "epoch 19 time 14.218791007995605 cost 10.5838\n",
      "epoch 20 time 14.903971672058105 cost 9.76465\n",
      "epoch 21 time 15.585092544555664 cost 9.19028\n",
      "epoch 22 time 16.26268696784973 cost 7.79638\n",
      "epoch 23 time 16.94363045692444 cost 6.85206\n",
      "epoch 24 time 17.619305849075317 cost 6.12399\n",
      "epoch 25 time 18.300414562225342 cost 5.49337\n",
      "epoch 26 time 18.98270893096924 cost 4.92482\n",
      "epoch 27 time 19.662325382232666 cost 4.23721\n",
      "epoch 28 time 20.336927890777588 cost 3.65909\n",
      "epoch 29 time 21.014528274536133 cost 3.56188\n",
      "epoch 30 time 21.69064497947693 cost 3.08151\n",
      "epoch 31 time 22.369674682617188 cost 2.68961\n",
      "epoch 32 time 23.046993732452393 cost 2.4466\n",
      "epoch 33 time 23.725372791290283 cost 2.07209\n",
      "epoch 34 time 24.399319887161255 cost 1.85867\n",
      "epoch 35 time 25.07584857940674 cost 1.70996\n",
      "epoch 36 time 25.758323907852173 cost 1.59335\n",
      "epoch 37 time 26.43937635421753 cost 1.43322\n",
      "epoch 38 time 27.121687650680542 cost 1.33125\n",
      "epoch 39 time 27.798295736312866 cost 1.24812\n",
      "epoch 40 time 28.497491598129272 cost 1.16638\n",
      "epoch 41 time 29.179673194885254 cost 1.0908\n",
      "epoch 42 time 29.857133388519287 cost 1.01401\n",
      "epoch 43 time 30.538844347000122 cost 0.970624\n",
      "epoch 44 time 31.22041893005371 cost 0.910392\n",
      "epoch 45 time 31.90003800392151 cost 0.86069\n",
      "epoch 46 time 32.57936191558838 cost 0.836511\n",
      "epoch 47 time 33.261260747909546 cost 0.782387\n",
      "epoch 48 time 33.94424271583557 cost 0.755054\n",
      "epoch 49 time 34.62477254867554 cost 0.71549\n",
      "epoch 50 time 35.30284833908081 cost 0.688426\n",
      "epoch 51 time 35.978044271469116 cost 0.65852\n",
      "epoch 52 time 36.65898084640503 cost 0.629772\n",
      "epoch 53 time 37.33378791809082 cost 0.60752\n",
      "epoch 54 time 38.011908531188965 cost 0.589113\n",
      "epoch 55 time 38.68384313583374 cost 0.56565\n",
      "epoch 56 time 39.36768698692322 cost 0.543472\n",
      "epoch 57 time 40.041282653808594 cost 0.527223\n",
      "epoch 58 time 40.71592569351196 cost 0.506158\n",
      "epoch 59 time 41.39215540885925 cost 0.494614\n",
      "epoch 60 time 42.071008920669556 cost 0.47969\n",
      "epoch 61 time 42.74505686759949 cost 0.464809\n",
      "epoch 62 time 43.41887903213501 cost 0.447418\n",
      "epoch 63 time 44.09699368476868 cost 0.434472\n",
      "epoch 64 time 44.7724552154541 cost 0.42358\n",
      "epoch 65 time 45.44840431213379 cost 0.410563\n",
      "epoch 66 time 46.124786138534546 cost 0.400012\n",
      "epoch 67 time 46.80566740036011 cost 0.389656\n",
      "epoch 68 time 47.48638725280762 cost 0.377353\n",
      "epoch 69 time 48.1612765789032 cost 0.369787\n",
      "epoch 70 time 48.842918157577515 cost 0.359108\n",
      "epoch 71 time 49.519346952438354 cost 0.352584\n",
      "epoch 72 time 50.19697618484497 cost 0.342143\n",
      "epoch 73 time 50.87242341041565 cost 0.334033\n",
      "epoch 74 time 51.550556659698486 cost 0.327725\n",
      "epoch 75 time 52.22495436668396 cost 0.320453\n",
      "epoch 76 time 52.90189456939697 cost 0.311703\n",
      "epoch 77 time 53.576759815216064 cost 0.306631\n",
      "epoch 78 time 54.261125326156616 cost 0.299531\n",
      "epoch 79 time 54.93938636779785 cost 0.291979\n",
      "epoch 80 time 55.620814085006714 cost 0.286976\n",
      "epoch 81 time 56.300376176834106 cost 0.281309\n",
      "epoch 82 time 56.97684669494629 cost 0.274285\n",
      "epoch 83 time 57.65705156326294 cost 0.270175\n",
      "epoch 84 time 58.337238073349 cost 0.265022\n",
      "epoch 85 time 59.01378321647644 cost 0.259824\n",
      "epoch 86 time 59.69552803039551 cost 0.254793\n",
      "epoch 87 time 60.37419605255127 cost 0.250097\n",
      "epoch 88 time 61.05813217163086 cost 0.244378\n",
      "epoch 89 time 61.743627071380615 cost 0.240348\n",
      "epoch 90 time 62.42293334007263 cost 0.236653\n",
      "epoch 91 time 63.10418105125427 cost 0.23238\n",
      "epoch 92 time 63.7853741645813 cost 0.228087\n",
      "epoch 93 time 64.45958232879639 cost 0.224166\n",
      "epoch 94 time 65.13483786582947 cost 0.220712\n",
      "epoch 95 time 65.81496953964233 cost 0.217433\n",
      "epoch 96 time 66.49327659606934 cost 0.214133\n",
      "epoch 97 time 67.17392015457153 cost 0.208788\n",
      "epoch 98 time 67.85678458213806 cost 0.206857\n",
      "epoch 99 time 68.53443956375122 cost 0.202726\n",
      "epoch 100 time 69.21171879768372 cost 0.20037\n",
      "epoch 101 time 69.88574695587158 cost 0.19767\n",
      "epoch 102 time 70.5636260509491 cost 0.194154\n",
      "epoch 103 time 71.2414026260376 cost 0.191358\n",
      "epoch 104 time 71.91983532905579 cost 0.188208\n",
      "epoch 105 time 72.59354591369629 cost 0.185518\n",
      "epoch 106 time 73.27172589302063 cost 0.183651\n",
      "epoch 107 time 73.95155358314514 cost 0.180313\n",
      "epoch 108 time 74.62864065170288 cost 0.178056\n",
      "epoch 109 time 75.30710434913635 cost 0.17557\n",
      "epoch 110 time 75.98680686950684 cost 0.172616\n",
      "epoch 111 time 76.67048048973083 cost 0.170603\n",
      "epoch 112 time 77.34705519676208 cost 0.168096\n",
      "epoch 113 time 78.02415823936462 cost 0.165849\n",
      "epoch 114 time 78.70628237724304 cost 0.163686\n",
      "epoch 115 time 79.3869481086731 cost 0.161774\n",
      "epoch 116 time 80.05973529815674 cost 0.160008\n",
      "epoch 117 time 80.73504900932312 cost 0.157735\n",
      "epoch 118 time 81.40997314453125 cost 0.155565\n",
      "epoch 119 time 82.08614134788513 cost 0.153998\n",
      "epoch 120 time 82.77365636825562 cost 0.151471\n",
      "epoch 121 time 83.44770407676697 cost 0.149778\n",
      "epoch 122 time 84.12150979042053 cost 0.14797\n",
      "epoch 123 time 84.80061411857605 cost 0.146331\n",
      "epoch 124 time 85.48407888412476 cost 0.144773\n",
      "epoch 125 time 86.15843200683594 cost 0.142788\n",
      "epoch 126 time 86.84153079986572 cost 0.141284\n",
      "epoch 127 time 87.52205991744995 cost 0.139394\n",
      "epoch 128 time 88.19651222229004 cost 0.138094\n",
      "epoch 129 time 88.87815737724304 cost 0.136112\n",
      "epoch 130 time 89.55277013778687 cost 0.135145\n",
      "epoch 131 time 90.23075437545776 cost 0.133324\n",
      "epoch 132 time 90.90794086456299 cost 0.131664\n",
      "epoch 133 time 91.5857629776001 cost 0.13051\n",
      "epoch 134 time 92.26869869232178 cost 0.128813\n",
      "epoch 135 time 92.95333242416382 cost 0.127516\n",
      "epoch 136 time 93.63281393051147 cost 0.126155\n",
      "epoch 137 time 94.3065025806427 cost 0.124923\n",
      "epoch 138 time 94.98462438583374 cost 0.123674\n",
      "epoch 139 time 95.66635274887085 cost 0.122087\n",
      "epoch 140 time 96.34024953842163 cost 0.121125\n",
      "epoch 141 time 97.02124857902527 cost 0.119614\n",
      "epoch 142 time 97.70658707618713 cost 0.118384\n",
      "epoch 143 time 98.38688850402832 cost 0.116874\n",
      "epoch 144 time 99.06523633003235 cost 0.116201\n",
      "epoch 145 time 99.74531984329224 cost 0.114979\n",
      "epoch 146 time 100.42024755477905 cost 0.113883\n",
      "epoch 147 time 101.10214328765869 cost 0.112647\n",
      "epoch 148 time 101.78306245803833 cost 0.11163\n",
      "epoch 149 time 102.45920062065125 cost 0.110481\n",
      "epoch 150 time 103.13665318489075 cost 0.109724\n",
      "epoch 151 time 103.8144884109497 cost 0.10831\n",
      "epoch 152 time 104.49896383285522 cost 0.107333\n",
      "epoch 153 time 105.17926526069641 cost 0.106499\n",
      "epoch 154 time 105.85888028144836 cost 0.105154\n",
      "epoch 155 time 106.54358100891113 cost 0.10449\n",
      "epoch 156 time 107.2266161441803 cost 0.103355\n",
      "epoch 157 time 107.90041041374207 cost 0.102659\n",
      "epoch 158 time 108.57726407051086 cost 0.101743\n",
      "epoch 159 time 109.25820779800415 cost 0.100965\n",
      "epoch 160 time 109.93572473526001 cost 0.099934\n",
      "epoch 161 time 110.61668705940247 cost 0.0988292\n",
      "epoch 162 time 111.29280710220337 cost 0.0982765\n",
      "epoch 163 time 111.96748208999634 cost 0.0974402\n",
      "epoch 164 time 112.64670705795288 cost 0.0966484\n",
      "epoch 165 time 113.3249990940094 cost 0.0955845\n",
      "epoch 166 time 114.00494766235352 cost 0.0949026\n",
      "epoch 167 time 114.68052101135254 cost 0.0939287\n",
      "epoch 168 time 115.35760402679443 cost 0.0933735\n",
      "epoch 169 time 116.0355532169342 cost 0.0925085\n",
      "epoch 170 time 116.71027612686157 cost 0.0917172\n",
      "epoch 171 time 117.39099955558777 cost 0.0909742\n",
      "epoch 172 time 118.06707286834717 cost 0.0902499\n",
      "epoch 173 time 118.7494432926178 cost 0.0895159\n",
      "epoch 174 time 119.43017411231995 cost 0.0886928\n",
      "epoch 175 time 120.10789823532104 cost 0.0880247\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 176 time 120.78199458122253 cost 0.0873951\n",
      "epoch 177 time 121.46211194992065 cost 0.0865744\n",
      "epoch 178 time 122.13696479797363 cost 0.0860436\n",
      "epoch 179 time 122.81354665756226 cost 0.085417\n",
      "epoch 180 time 123.49779343605042 cost 0.0845466\n",
      "epoch 181 time 124.17520928382874 cost 0.0839036\n",
      "epoch 182 time 124.85418224334717 cost 0.0833841\n",
      "epoch 183 time 125.53646111488342 cost 0.0826825\n",
      "epoch 184 time 126.21570229530334 cost 0.0821642\n",
      "epoch 185 time 126.89334416389465 cost 0.0813151\n",
      "epoch 186 time 127.57325458526611 cost 0.0808353\n",
      "epoch 187 time 128.24826169013977 cost 0.0803085\n",
      "epoch 188 time 128.92513751983643 cost 0.0796371\n",
      "epoch 189 time 129.60371685028076 cost 0.079176\n",
      "epoch 190 time 130.27658987045288 cost 0.0785262\n",
      "epoch 191 time 130.95496439933777 cost 0.07804\n",
      "epoch 192 time 131.62975430488586 cost 0.0774454\n",
      "epoch 193 time 132.30295276641846 cost 0.0767866\n",
      "epoch 194 time 132.97950506210327 cost 0.0761455\n",
      "epoch 195 time 133.65897965431213 cost 0.0758514\n",
      "epoch 196 time 134.33875513076782 cost 0.0753447\n",
      "epoch 197 time 135.01778316497803 cost 0.0747291\n",
      "epoch 198 time 135.69337558746338 cost 0.0741263\n",
      "epoch 199 time 136.37030339241028 cost 0.0736472\n",
      "epoch 200 time 137.04665446281433 cost 0.0732183\n",
      "epoch 201 time 137.7248876094818 cost 0.072669\n",
      "epoch 202 time 138.404700756073 cost 0.072172\n",
      "epoch 203 time 139.08435535430908 cost 0.0715977\n",
      "epoch 204 time 139.76511025428772 cost 0.0712739\n",
      "epoch 205 time 140.4408745765686 cost 0.0706891\n",
      "epoch 206 time 141.119526386261 cost 0.0702381\n",
      "epoch 207 time 141.80144619941711 cost 0.0698292\n",
      "epoch 208 time 142.47910332679749 cost 0.069365\n",
      "epoch 209 time 143.16374826431274 cost 0.068859\n",
      "epoch 210 time 143.83889603614807 cost 0.068428\n",
      "epoch 211 time 144.51417088508606 cost 0.0681006\n",
      "epoch 212 time 145.18877959251404 cost 0.067607\n",
      "epoch 213 time 145.8664195537567 cost 0.0671462\n",
      "epoch 214 time 146.5464026927948 cost 0.0667407\n",
      "epoch 215 time 147.22734928131104 cost 0.0663043\n",
      "epoch 216 time 147.9072871208191 cost 0.0658939\n",
      "epoch 217 time 148.5872015953064 cost 0.0653416\n",
      "epoch 218 time 149.27718329429626 cost 0.0649611\n",
      "epoch 219 time 149.98444032669067 cost 0.0646968\n",
      "epoch 220 time 150.66879320144653 cost 0.0642728\n",
      "epoch 221 time 151.41013765335083 cost 0.0638082\n",
      "epoch 222 time 152.15497398376465 cost 0.0632958\n",
      "epoch 223 time 152.87975573539734 cost 0.0630202\n",
      "epoch 224 time 153.5840404033661 cost 0.0624972\n",
      "epoch 225 time 154.29142808914185 cost 0.0623062\n",
      "epoch 226 time 155.00610446929932 cost 0.0619088\n",
      "epoch 227 time 155.72652292251587 cost 0.0615579\n",
      "epoch 228 time 156.43746709823608 cost 0.0612003\n",
      "epoch 229 time 157.143000125885 cost 0.0608556\n",
      "epoch 230 time 157.84243726730347 cost 0.0604641\n",
      "epoch 231 time 158.54408431053162 cost 0.0601031\n",
      "epoch 232 time 159.2410409450531 cost 0.0597793\n",
      "epoch 233 time 159.9505877494812 cost 0.0594737\n",
      "epoch 234 time 160.64701128005981 cost 0.0590202\n",
      "epoch 235 time 161.35717678070068 cost 0.0588161\n",
      "epoch 236 time 162.06593346595764 cost 0.0583637\n",
      "epoch 237 time 162.78749179840088 cost 0.0580166\n",
      "epoch 238 time 163.5455837249756 cost 0.0577539\n",
      "epoch 239 time 164.2607297897339 cost 0.0572873\n",
      "epoch 240 time 164.9633457660675 cost 0.0571128\n",
      "epoch 241 time 165.66700768470764 cost 0.0567534\n",
      "epoch 242 time 166.37861824035645 cost 0.0564729\n",
      "epoch 243 time 167.07825541496277 cost 0.0561455\n",
      "epoch 244 time 167.7891504764557 cost 0.0558708\n",
      "epoch 245 time 168.4924976825714 cost 0.0555628\n",
      "epoch 246 time 169.20313596725464 cost 0.0551384\n",
      "epoch 247 time 169.92691087722778 cost 0.0548655\n",
      "epoch 248 time 170.65394377708435 cost 0.0546517\n",
      "epoch 249 time 171.3743691444397 cost 0.0543457\n",
      "epoch 250 time 172.11171650886536 cost 0.0540602\n",
      "epoch 251 time 172.82691049575806 cost 0.0537347\n",
      "epoch 252 time 173.5410532951355 cost 0.0535422\n",
      "epoch 253 time 174.267893075943 cost 0.0532326\n",
      "epoch 254 time 175.0228841304779 cost 0.0529175\n",
      "epoch 255 time 175.7272446155548 cost 0.0526539\n",
      "epoch 256 time 176.43394947052002 cost 0.0523911\n",
      "epoch 257 time 177.1363501548767 cost 0.0520967\n",
      "epoch 258 time 177.84670853614807 cost 0.0518459\n",
      "epoch 259 time 178.55844950675964 cost 0.0515752\n",
      "epoch 260 time 179.2610411643982 cost 0.0513151\n",
      "epoch 261 time 179.96760630607605 cost 0.0510576\n",
      "epoch 262 time 180.67283129692078 cost 0.0508071\n",
      "epoch 263 time 181.3743634223938 cost 0.05057\n",
      "epoch 264 time 182.07517170906067 cost 0.0502225\n",
      "epoch 265 time 182.77405858039856 cost 0.0500738\n",
      "epoch 266 time 183.47418475151062 cost 0.0497811\n",
      "epoch 267 time 184.17258310317993 cost 0.0495093\n",
      "epoch 268 time 184.87323307991028 cost 0.0493101\n",
      "epoch 269 time 185.58073806762695 cost 0.0490678\n",
      "epoch 270 time 186.28322744369507 cost 0.0488286\n",
      "epoch 271 time 186.98494577407837 cost 0.0485397\n",
      "epoch 272 time 187.6852321624756 cost 0.0482954\n",
      "epoch 273 time 188.3849503993988 cost 0.0480922\n",
      "epoch 274 time 189.08204579353333 cost 0.0478455\n",
      "epoch 275 time 189.79022693634033 cost 0.047595\n",
      "epoch 276 time 190.4894778728485 cost 0.0473557\n",
      "epoch 277 time 191.18864035606384 cost 0.0471216\n",
      "epoch 278 time 191.88536953926086 cost 0.0469213\n",
      "epoch 279 time 192.5807855129242 cost 0.0466786\n",
      "epoch 280 time 193.2781503200531 cost 0.0465384\n",
      "epoch 281 time 193.979416847229 cost 0.0462396\n",
      "epoch 282 time 194.6758975982666 cost 0.0460691\n",
      "epoch 283 time 195.37924098968506 cost 0.0458322\n",
      "epoch 284 time 196.08511543273926 cost 0.0456208\n",
      "epoch 285 time 196.7881259918213 cost 0.0454443\n",
      "epoch 286 time 197.49156785011292 cost 0.0451742\n",
      "epoch 287 time 198.18952059745789 cost 0.0449968\n",
      "epoch 288 time 198.88682079315186 cost 0.0447987\n",
      "epoch 289 time 199.59346723556519 cost 0.0445575\n",
      "epoch 290 time 200.290465593338 cost 0.0443971\n",
      "epoch 291 time 200.99119424819946 cost 0.0441451\n",
      "epoch 292 time 201.68710827827454 cost 0.0440178\n",
      "epoch 293 time 202.38242077827454 cost 0.0437779\n",
      "epoch 294 time 203.0808608531952 cost 0.043584\n",
      "epoch 295 time 203.78632426261902 cost 0.0433821\n",
      "epoch 296 time 204.48734283447266 cost 0.0432201\n",
      "epoch 297 time 205.18880414962769 cost 0.0430101\n",
      "epoch 298 time 205.89088225364685 cost 0.0428214\n",
      "epoch 299 time 206.59098076820374 cost 0.0426362\n",
      "time 206.63255667686462\n",
      "0.979459983995\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFZRJREFUeJzt3X+QXeV93/H39+6uduWVkCy0UmQJIykodRjbMUQheHDc\nxDKtoWlFpjhDmsSMw1TT1ElI4k5Lkpk26eQPOzOFmE5rDw2eyp5MDCFpYRxnyi+5bkuMs9hCIFTB\nGoORIrSLACFQ9XOf/nGeFevl/pJ2L+eeo/drZuee+5yz936Pzuqzzz7nuedESglJUn01yi5AktRb\nBr0k1ZxBL0k1Z9BLUs0Z9JJUcwa9JNWcQS9JNWfQS1LNGfSSVHODZRcAsHLlyrR+/fqyy5CkSnns\nscdeSimNddquL4J+/fr1jI+Pl12GJFVKRDzfzXYO3UhSzRn0klRzBr0k1ZxBL0k1Z9BLUs0Z9JJU\ncwa9JNVcpYN+/LmXufX+vZw4NV12KZLUtyod9N/+/ivc/vAEJ08b9JLUSqWDvhEBwLQ3OJekliod\n9HEm6EsuRJL6WNdBHxEDEfGdiPhqfr4hIh6NiImIuCsiFuX24fx8Iq9f35vSoVHkPMkevSS1dDY9\n+puBPbOefxa4LaV0CfAKcFNuvwl4JbfflrfriYY9eknqqKugj4h1wD8C/iQ/D+AjwD15k+3AdXl5\na35OXr8lZsZYFthMj94xeklqrdse/R8D/xqYmd5yIfBqSulUfr4PWJuX1wIvAOT1h/P2Cy48GStJ\nHXUM+oj4WWAypfTYQr5xRGyLiPGIGJ+amjqn15gZujHnJam1bnr0VwH/JCKeA75CMWTzOWB5RMzc\nuGQdsD8v7wcuAsjrlwGH5r5oSumOlNLmlNLmsbGON0hpXrxDN5LUUcegTyn9TkppXUppPXAD8HBK\n6ReBHcD1ebMbgXvz8n35OXn9w6lH02I8GStJnc1nHv2/AX47IiYoxuDvzO13Ahfm9t8Gbplfia3N\nnOKdNuklqaWzumdsSunrwNfz8rPAFU22OQZ8fAFq68gxeknqrNKfjG3k6h2jl6TWqh30Tq+UpI4q\nHfRe60aSOqt00HutG0nqrOJBb49ekjqpeNAXj47RS1JrlQ56r3UjSZ1VOuidRy9JnVU86ItHe/SS\n1FrFg96TsZLUSaWDPuzRS1JHlQ76N8foDXpJaqUWQe/QjSS1VvGgLx69TLEktVbpoPdaN5LUWaWD\n3mvdSFJn1Q76hj16Seqk2kHv9EpJ6qjSQe+1biSps0oHvde6kaTOKh70xaM9eklqreJB78lYSeqk\n0kHvtW4kqbNKB73XupGkzmoR9A7dSFJrFQ/64tGhG0lqrdJBPzOP/rRdeklqqdJB/+a1bsqtQ5L6\nWcWD3k/GSlInlQ76AS9qJkkdVTronUcvSZ1VOuidRy9JndUi6B26kaTWKh70xaNDN5LUWqWD3nvG\nSlJnlQ567xkrSZ1VPOhzj94uvSS1VI+gN+clqaVKB33k6j0ZK0mtdQz6iBiJiG9FxOMRsTsi/iC3\nb4iIRyNiIiLuiohFuX04P5/I69f3rHjvGStJHXXToz8OfCSl9GPAB4CPRcSVwGeB21JKlwCvADfl\n7W8CXsntt+XtesLplZLUWcegT4XX89Oh/JWAjwD35PbtwHV5eWt+Tl6/JWbmQS4wx+glqbOuxugj\nYiAidgKTwAPAd4FXU0qn8ib7gLV5eS3wAkBefxi4sMlrbouI8YgYn5qaOqfivdaNJHXWVdCnlE6n\nlD4ArAOuAN4z3zdOKd2RUtqcUto8NjZ2Tq/htW4kqbOzmnWTUnoV2AF8EFgeEYN51Tpgf17eD1wE\nkNcvAw4tSLVzOHQjSZ11M+tmLCKW5+XFwNXAHorAvz5vdiNwb16+Lz8nr3849ajL7clYSepssPMm\nrAG2R8QAxS+Gu1NKX42Ip4CvRMQfAt8B7szb3wl8OSImgJeBG3pQN+C1biSpGx2DPqW0C7isSfuz\nFOP1c9uPAR9fkOq60AjH6CWpnUp/MhaKcXqHbiSptZoEfdlVSFL/qnzQR3gyVpLaqXzQNyK81o0k\ntVGDoPd69JLUTg2C3jF6SWqn8kHvGL0ktVf5oG80wnn0ktRG9YPeoRtJaqsGQe/QjSS1U/mgD3v0\nktRW5YPea91IUns1CHqvdSNJ7dQk6MuuQpL6V+WD3nn0ktRe5YPea91IUns1CHp79JLUTg2C3jF6\nSWqn8kHvGL0ktVf5oG9EeJliSWqjHkFvj16SWqp80BdDN2VXIUn9q/JBP+BliiWprcoHvbNuJKm9\nGgS9s24kqZ3KB72XKZak9iof9F6mWJLaq0HQO71SktqpR9BPl12FJPWvyge9l0CQpPYqH/RepliS\n2qt+0Dfs0UtSO9UPek/GSlJblQ9659FLUnuVD3rn0UtSezUIenv0ktRODYLek7GS1E7lg94xeklq\nr/JB7xi9JLXXMegj4qKI2BERT0XE7oi4ObeviIgHIuKZ/PjO3B4RcXtETETEroi4vKc74PRKSWqr\nmx79KeDTKaVLgSuBT0XEpcAtwEMppU3AQ/k5wDXApvy1Dfj8glc9iydjJam9jkGfUjqQUvp2Xj4C\n7AHWAluB7Xmz7cB1eXkr8KVU+CawPCLWLHjlmde6kaT2zmqMPiLWA5cBjwKrU0oH8qoXgdV5eS3w\nwqxv25fb5r7WtogYj4jxqampsyz7TV7rRpLa6zroI2IJ8BfAb6aUXpu9LhVnQ88qblNKd6SUNqeU\nNo+NjZ3Nt/4Ap1dKUntdBX1EDFGE/J+mlP4yNx+cGZLJj5O5fT9w0axvX5fbesKTsZLUXjezbgK4\nE9iTUrp11qr7gBvz8o3AvbPaP5Fn31wJHJ41xLPgwhuPSFJbg11scxXwy8ATEbEzt/0u8Bng7oi4\nCXge+Pm87mvAtcAEcBT45IJWPIfz6CWpvY5Bn1L630C0WL2lyfYJ+NQ86+qa0yslqb3qfzLWG49I\nUluVD3qvdSNJ7VU+6B2jl6T2ahD0Tq+UpHZqEvRlVyFJ/avyQe+1biSpvcoHvde6kaT2ahD09ugl\nqZ0aBL0nYyWpncoHvfPoJam9yge98+glqb0aBH1w2i69JLVUg6DHoRtJaqPyQV9cLt/hG0lqpfJB\n38hBb69ekpqrfNAP5D1wiqUkNVf5oI8zPXqDXpKaqXzQN86M0ZdciCT1qRoEffFoj16SmqtB0Hsy\nVpLaqXzQ55z3Q1OS1ELlg354aACA46dOl1yJJPWnygf90uFBAF4/dqrkSiSpP1U+6Edz0L9x3B69\nJDVT+aBfkoP+yPGTJVciSf2pNkFvj16Smqt+0I/kMXp79JLUVOWDfnS4mHXzuj16SWqq8kG/dHgI\ncNaNJLVS+aAfGWow0AiHbiSphcoHfUQwumjAk7GS1ELlgx5g6cgQRxy6kaSmahH0o8MDvHHcoJek\nZmoR9EuGB3ndoJekpmoR9KMGvSS1VIugXzpi0EtSK7UI+iXDg86jl6QWahH0o8ODnoyVpBY6Bn1E\nfDEiJiPiyVltKyLigYh4Jj++M7dHRNweERMRsSsiLu9l8TOWDg/y+olTJO8bK0lv0U2P/r8CH5vT\ndgvwUEppE/BQfg5wDbApf20DPr8wZbY3OjxISnD0hB+akqS5OgZ9SukbwMtzmrcC2/PyduC6We1f\nSoVvAssjYs1CFdvK8ncU17t55eiJXr+VJFXOuY7Rr04pHcjLLwKr8/Ja4IVZ2+3LbT21aukIAJNH\njvf6rSSpcuZ9MjYVA+NnPTgeEdsiYjwixqempuZVw9jSYQAmXzPoJWmucw36gzNDMvlxMrfvBy6a\ntd263PYWKaU7UkqbU0qbx8bGzrGMwqoLiqCfOnJsXq8jSXV0rkF/H3BjXr4RuHdW+yfy7JsrgcOz\nhnh65sLRYRrh0I0kNTPYaYOI+DPgp4GVEbEP+HfAZ4C7I+Im4Hng5/PmXwOuBSaAo8Ane1DzWww0\ngpVLhjn4mj16SZqrY9CnlH6hxaotTbZNwKfmW9S5WHXBsD16SWqiFp+MhWLmjSdjJemtahT09ugl\nqZlaBf2hN45z6vR02aVIUl+pTdCvXjZCSs68kaS5ahP0F68YBeC5Q2+UXIkk9ZfaBP2GsRz0Lx0t\nuRJJ6i+1Cfo1F4wwPNjgey+9XnYpktRXahP0jUaw/sJRvveSQzeSNFttgh5gw0qDXpLmqlXQr185\nyvdfPuoUS0mapVZBv3FslJOnE99/2ROykjSjVkF/6ZoLANhz4EjJlUhS/6hV0F+yagmDjeCpA4fL\nLkWS+katgn5kaIBLVi3hqb97rexSJKlv1CroAS591wXsNugl6Yz6Bf2aC5g8cpxJbysoSUANg/6K\nDSsA+JvvHiq5EknqD7UL+ve+axkrRhfxP/dOlV2KJPWF2gV9oxH81KaVfOOZKaanU9nlSFLpahf0\nAD/998Z46fUTfOeFV8suRZJKV8ug3/KjqxkebHDfzv1llyJJpatl0F8wMsRHf3Q1X911gJNe90bS\nea6WQQ/wc5et5dAbJ3hoz8GyS5GkUtU26H/mPatYu3wx2x95vuxSJKlUtQ36gUbwS1dezN88e4jd\nf+e1bySdv2ob9AD/7CffzdKRQT734DNllyJJpal10C9bPMQ//6mN3P/UQZ7YZ69e0vmp1kEP8Mmr\n1rNs8RB//ODTZZciSaWofdAvHRli24c38tD/neSRiZfKLkeS3na1D3qAX7lqAxtXjvLpP3+cw0dP\nll2OJL2tzougX7xogM/dcBlTR47zu//9CVLyGjiSzh/nRdADvG/dMn7r6h/hr3Yd4D8+PFF2OZL0\nthksu4C307/4+z/Ms1NvcOsDxYnZ39iyqeSKJKn3zqugH2gEf3T9+wG49YGnSQlu/qhhL6nezqug\nhzfDPgJuy1MuDXtJdXbeBT0UYf/Zf1r07G978GmemTzCv9/6XlaMLiq5MklaeOfNydi5ZsL+01f/\nCPfvPsjP3v6/2OmNSiTV0Hkb9FCE/a9v2cQ9v/pBIoKPf+ERfuuundy/+0VOeR17STVxXgf9jPev\nW85f/caHuP7HL2LH3km2ffkxrvvP/4cdeyc57X1nJVVc9OLDQxHxMeBzwADwJymlz7TbfvPmzWl8\nfHzB6zgXJ09P89dPvsgffvUpJo8cZ+3yxVzz3h/iZ96zip9Yv4JFg/5ulNQfIuKxlNLmjtstdNBH\nxADwNHA1sA/4W+AXUkpPtfqefgr6GSdOTfPgnoPcPf4Cj0wc4sTpaUYXDXDVJSvZMDbKxStGef+6\nZWwcG+Udi87Lc9qSStZt0Pcioa4AJlJKz+ZCvgJsBVoGfT9aNNjg2vet4dr3reHoiVM8MnGIHXsn\n+cYzU3x97xQnZo3hLxkeZNXSYVZdMMyqpSOsWjrMssVDjAwNMDLUYHhooFgebDAyNMCiwQaDjaDR\nCAYbwUD+KpYbDEQwMFA8b0SxLoBGBATkByIiP+Z1zKyLpttE3kbS+aUXQb8WeGHW833AT/bgfd42\n71g0yEcvXc1HL10NQEqJ7798lF37DrPvlf/H5JFjTL52nMkjx3h836scfO0Yx07258ncVr8AzizP\n/SUxa7nZa72lrel7NvneFrV184rdv2+z7RZ2P5pp+nol1XIuetod6HFfo5cv36t/85u3bOIf/9i7\nevLaM0obc4iIbcA2gHe/+91llXFOIoKLLxzl4gtHW25z8vQ0x06e5tjJ4vH4qTeXT5ya5nRKnJpO\nTE8Xj6dnff1g+zSnpxMJSAmm81BbSpBI+XHuuh9sP7NdSm9tm3l+Zl3z752r2ZBfs0HApt/bZMvm\n23X3es22bF5z72tp9npdNs3r33Sh9HLqQa8vJtjTV+/hiy9bPNS7F896EfT7gYtmPV+X235ASukO\n4A4oxuh7UEephgYaDA00WDpSdiWSzne9mELyt8CmiNgQEYuAG4D7evA+kqQuLHiPPqV0KiJ+Dfgf\nFNMrv5hS2r3Q7yNJ6k5PxuhTSl8DvtaL15YknR0//SNJNWfQS1LNGfSSVHMGvSTVnEEvSTXXk6tX\nnnUREVPA8+f47SuBlxawnDK5L/3JfelP7gtcnFIa67RRXwT9fETEeDdXb6sC96U/uS/9yX3pnkM3\nklRzBr0k1Vwdgv6OsgtYQO5Lf3Jf+pP70qXKj9FLktqrQ49ektRGpYM+Ij4WEXsjYiIibim7nrMV\nEc9FxBMRsTMixnPbioh4ICKeyY/vLLvOZiLiixExGRFPzmprWnsUbs/HaVdEXF5e5W/VYl9+PyL2\n52OzMyKunbXud/K+7I2If1hO1W8VERdFxI6IeCoidkfEzbm9cselzb5U8biMRMS3IuLxvC9/kNs3\nRMSjuea78mXdiYjh/Hwir18/7yKKOwpV74viEsjfBTYCi4DHgUvLruss9+E5YOWctj8CbsnLtwCf\nLbvOFrV/GLgceLJT7cC1wF9T3OntSuDRsuvvYl9+H/hXTba9NP+sDQMb8s/gQNn7kGtbA1yel5cC\nT+d6K3dc2uxLFY9LAEvy8hDwaP73vhu4Ibd/AfjVvPwvgS/k5RuAu+ZbQ5V79GduQp5SOgHM3IS8\n6rYC2/PyduC6EmtpKaX0DeDlOc2tat8KfCkVvgksj4g1b0+lnbXYl1a2Al9JKR1PKX0PmKD4WSxd\nSulASunbefkIsIfiHs6VOy5t9qWVfj4uKaX0en46lL8S8BHgntw+97jMHK97gC0xzxvWVjnom92E\nvN0PQj9KwP0R8Vi+hy7A6pTSgbz8IrC6nNLOSavaq3qsfi0PaXxx1hBaJfYl/7l/GUXvsdLHZc6+\nQAWPS0QMRMROYBJ4gOIvjldTSqfyJrPrPbMvef1h4ML5vH+Vg74OPpRSuhy4BvhURHx49spU/O1W\nyWlRVa49+zzww8AHgAPAfyi3nO5FxBLgL4DfTCm9Nntd1Y5Lk32p5HFJKZ1OKX2A4h7aVwDveTvf\nv8pB39VNyPtZSml/fpwE/hvFD8DBmT+f8+NkeRWetVa1V+5YpZQO5v+c08B/4c1hgL7el4gYogjG\nP00p/WVuruRxabYvVT0uM1JKrwI7gA9SDJXN3OVvdr1n9iWvXwYcms/7VjnoK30T8ogYjYilM8vA\nPwCepNiHG/NmNwL3llPhOWlV+33AJ/IsjyuBw7OGEvrSnLHqn6M4NlDsyw15ZsQGYBPwrbe7vmby\nOO6dwJ6U0q2zVlXuuLTal4oel7GIWJ6XFwNXU5xz2AFcnzebe1xmjtf1wMP5L7FzV/YZ6fl8Ucwa\neJpivOv3yq7nLGvfSDFL4HFg90z9FGNxDwHPAA8CK8qutUX9f0bxp/NJivHFm1rVTjHr4D/l4/QE\nsLns+rvYly/nWnfl/3hrZm3/e3lf9gLXlF3/rLo+RDEsswvYmb+ureJxabMvVTwu7we+k2t+Evi3\nuX0jxS+jCeDPgeHcPpKfT+T1G+dbg5+MlaSaq/LQjSSpCwa9JNWcQS9JNWfQS1LNGfSSVHMGvSTV\nnEEvSTVn0EtSzf1/Ewo34WyoaW0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f3a701d92b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "score_homework()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}