t81_558_class_03_3_save_load.ipynb

{"cells": [{"cell_type": "markdown", "metadata": {}, "source": ["<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_3_save_load.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# T81-558: Applications of Deep Neural Networks\n", "**Module 3: Introduction to TensorFlow**\n", "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n", "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Module 3 Material\n", "\n", "* Part 3.1: Deep Learning and Neural Network Introduction [[Video]](https://www.youtube.com/watch?v=zYnI4iWRmpc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_1_neural_net.ipynb)\n", "* Part 3.2: Introduction to Tensorflow and Keras [[Video]](https://www.youtube.com/watch?v=PsE73jk55cE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_2_keras.ipynb)\n", "* **Part 3.3: Saving and Loading a Keras Neural Network** [[Video]](https://www.youtube.com/watch?v=-9QfbGM1qGw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_3_save_load.ipynb)\n", "* Part 3.4: Early Stopping in Keras to Prevent Overfitting [[Video]](https://www.youtube.com/watch?v=m1LNunuI2fk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_4_early_stop.ipynb)\n", "* Part 3.5: Extracting Weights and Manual Calculation [[Video]](https://www.youtube.com/watch?v=7PWgx16kH8s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_5_weights.ipynb)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Google CoLab Instructions\n", "\n", "The following code ensures that Google CoLab is running the correct version of TensorFlow.\n", "  Running the following code will map your GDrive to ```/content/drive```."]}, {"cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Note: not using Google CoLab\n"]}], "source": ["try:\n", "    from google.colab import drive\n", "    drive.mount('/content/drive', force_remount=True)\n", "    COLAB = True\n", "    print(\"Note: using Google CoLab\")\n", "    %tensorflow_version 2.x\n", "except:\n", "    print(\"Note: not using Google CoLab\")\n", "    COLAB = False"]}, {"cell_type": "markdown", "metadata": {}, "source": ["# Part 3.3: Saving and Loading a Keras Neural Network\n", "\n", "Complex neural networks will take a long time to fit/train.  It is helpful to be able to save these neural networks so that they can be reloaded later.  A reloaded neural network will not require retraining.  Keras provides three formats for neural network saving.\n", "\n", "* **YAML** - Stores the neural network structure (no weights) in the [YAML file format](https://en.wikipedia.org/wiki/YAML).\n", "* **JSON** - Stores the neural network structure (no weights) in the [JSON file format](https://en.wikipedia.org/wiki/JSON).\n", "* **HDF5** - Stores the complete neural network (with weights) in the [HDF5 file format](https://en.wikipedia.org/wiki/Hierarchical_Data_Format). Do not confuse HDF5 with [HDFS](https://en.wikipedia.org/wiki/Apache_Hadoop).  They are different.  We do not use HDFS in this class.\n", "\n", "Usually you will want to save in HDF5."]}, {"cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Train on 398 samples\n", "Epoch 1/100\n", "398/398 - 1s - loss: 157998.8970\n", "Epoch 2/100\n", "398/398 - 0s - loss: 8659.0171\n", "Epoch 3/100\n", "398/398 - 0s - loss: 6044.9820\n", "Epoch 4/100\n", "398/398 - 0s - loss: 1397.8733\n", "Epoch 5/100\n", "398/398 - 0s - loss: 961.8155\n", "Epoch 6/100\n", "398/398 - 0s - loss: 629.2083\n", "Epoch 7/100\n", "398/398 - 0s - loss: 508.9476\n", "Epoch 8/100\n", "398/398 - 0s - loss: 416.8677\n", "Epoch 9/100\n", "398/398 - 0s - loss: 384.6891\n", "Epoch 10/100\n", "398/398 - 0s - loss: 357.8706\n", "Epoch 11/100\n", "398/398 - 0s - loss: 336.1742\n", "Epoch 12/100\n", "398/398 - 0s - loss: 321.2081\n", "Epoch 13/100\n", "398/398 - 0s - loss: 300.8533\n", "Epoch 14/100\n", "398/398 - 0s - loss: 285.0313\n", "Epoch 15/100\n", "398/398 - 0s - loss: 269.9887\n", "Epoch 16/100\n", "398/398 - 0s - loss: 255.0380\n", "Epoch 17/100\n", "398/398 - 0s - loss: 238.8779\n", "Epoch 18/100\n", "398/398 - 0s - loss: 226.0044\n", "Epoch 19/100\n", "398/398 - 0s - loss: 212.6679\n", "Epoch 20/100\n", "398/398 - 0s - loss: 200.1701\n", "Epoch 21/100\n", "398/398 - 0s - loss: 188.6579\n", "Epoch 22/100\n", "398/398 - 0s - loss: 177.9696\n", "Epoch 23/100\n", "398/398 - 0s - loss: 167.8847\n", "Epoch 24/100\n", "398/398 - 0s - loss: 158.5341\n", "Epoch 25/100\n", "398/398 - 0s - loss: 150.4487\n", "Epoch 26/100\n", "398/398 - 0s - loss: 140.6709\n", "Epoch 27/100\n", "398/398 - 0s - loss: 133.0645\n", "Epoch 28/100\n", "398/398 - 0s - loss: 125.9491\n", "Epoch 29/100\n", "398/398 - 0s - loss: 120.9782\n", "Epoch 30/100\n", "398/398 - 0s - loss: 114.9785\n", "Epoch 31/100\n", "398/398 - 0s - loss: 108.6741\n", "Epoch 32/100\n", "398/398 - 0s - loss: 104.3519\n", "Epoch 33/100\n", "398/398 - 0s - loss: 98.8682\n", "Epoch 34/100\n", "398/398 - 0s - loss: 94.9151\n", "Epoch 35/100\n", "398/398 - 0s - loss: 91.6754\n", "Epoch 36/100\n", "398/398 - 0s - loss: 88.2521\n", "Epoch 37/100\n", "398/398 - 0s - loss: 87.6504\n", "Epoch 38/100\n", "398/398 - 0s - loss: 83.6231\n", "Epoch 39/100\n", "398/398 - 0s - loss: 80.9388\n", "Epoch 40/100\n", "398/398 - 0s - loss: 79.8043\n", "Epoch 41/100\n", "398/398 - 0s - loss: 77.2335\n", "Epoch 42/100\n", "398/398 - 0s - loss: 73.2088\n", "Epoch 43/100\n", "398/398 - 0s - loss: 70.6557\n", "Epoch 44/100\n", "398/398 - 0s - loss: 68.9325\n", "Epoch 45/100\n", "398/398 - 0s - loss: 67.7848\n", "Epoch 46/100\n", "398/398 - 0s - loss: 69.3400\n", "Epoch 47/100\n", "398/398 - 0s - loss: 67.1279\n", "Epoch 48/100\n", "398/398 - 0s - loss: 64.1726\n", "Epoch 49/100\n", "398/398 - 0s - loss: 63.5665\n", "Epoch 50/100\n", "398/398 - 0s - loss: 61.3269\n", "Epoch 51/100\n", "398/398 - 0s - loss: 60.6728\n", "Epoch 52/100\n", "398/398 - 0s - loss: 60.8373\n", "Epoch 53/100\n", "398/398 - 0s - loss: 59.1935\n", "Epoch 54/100\n", "398/398 - 0s - loss: 58.6958\n", "Epoch 55/100\n", "398/398 - 0s - loss: 57.9942\n", "Epoch 56/100\n", "398/398 - 0s - loss: 57.0906\n", "Epoch 57/100\n", "398/398 - 0s - loss: 56.8204\n", "Epoch 58/100\n", "398/398 - 0s - loss: 55.8276\n", "Epoch 59/100\n", "398/398 - 0s - loss: 54.7317\n", "Epoch 60/100\n", "398/398 - 0s - loss: 54.7112\n", "Epoch 61/100\n", "398/398 - 0s - loss: 53.9821\n", "Epoch 62/100\n", "398/398 - 0s - loss: 54.8269\n", "Epoch 63/100\n", "398/398 - 0s - loss: 54.3242\n", "Epoch 64/100\n", "398/398 - 0s - loss: 53.6749\n", "Epoch 65/100\n", "398/398 - 0s - loss: 51.4106\n", "Epoch 66/100\n", "398/398 - 0s - loss: 52.1452\n", "Epoch 67/100\n", "398/398 - 0s - loss: 51.5271\n", "Epoch 68/100\n", "398/398 - 0s - loss: 52.4211\n", "Epoch 69/100\n", "398/398 - 0s - loss: 53.4796\n", "Epoch 70/100\n", "398/398 - 0s - loss: 50.3901\n", "Epoch 71/100\n", "398/398 - 0s - loss: 51.2261\n", "Epoch 72/100\n", "398/398 - 0s - loss: 53.3287\n", "Epoch 73/100\n", "398/398 - 0s - loss: 49.5418\n", "Epoch 74/100\n", "398/398 - 0s - loss: 47.5901\n", "Epoch 75/100\n", "398/398 - 0s - loss: 47.2035\n", "Epoch 76/100\n", "398/398 - 0s - loss: 46.8783\n", "Epoch 77/100\n", "398/398 - 0s - loss: 47.4590\n", "Epoch 78/100\n", "398/398 - 0s - loss: 47.6670\n", "Epoch 79/100\n", "398/398 - 0s - loss: 48.9362\n", "Epoch 80/100\n", "398/398 - 0s - loss: 45.8346\n", "Epoch 81/100\n", "398/398 - 0s - loss: 45.3254\n", "Epoch 82/100\n", "398/398 - 0s - loss: 45.2809\n", "Epoch 83/100\n", "398/398 - 0s - loss: 44.5924\n", "Epoch 84/100\n", "398/398 - 0s - loss: 43.4867\n", "Epoch 85/100\n", "398/398 - 0s - loss: 44.5989\n", "Epoch 86/100\n", "398/398 - 0s - loss: 44.3006\n", "Epoch 87/100\n", "398/398 - 0s - loss: 43.9893\n", "Epoch 88/100\n", "398/398 - 0s - loss: 44.8011\n", "Epoch 89/100\n", "398/398 - 0s - loss: 43.9355\n", "Epoch 90/100\n", "398/398 - 0s - loss: 41.9677\n", "Epoch 91/100\n", "398/398 - 0s - loss: 41.7740\n", "Epoch 92/100\n", "398/398 - 0s - loss: 41.8235\n", "Epoch 93/100\n", "398/398 - 0s - loss: 41.7165\n", "Epoch 94/100\n", "398/398 - 0s - loss: 41.2443\n", "Epoch 95/100\n", "398/398 - 0s - loss: 42.9613\n", "Epoch 96/100\n", "398/398 - 0s - loss: 41.2405\n", "Epoch 97/100\n", "398/398 - 0s - loss: 41.7265\n", "Epoch 98/100\n", "398/398 - 0s - loss: 40.3650\n", "Epoch 99/100\n", "398/398 - 0s - loss: 39.7134\n", "Epoch 100/100\n", "398/398 - 0s - loss: 40.6860\n", "Before save score (RMSE): 6.3568661509704905\n"]}], "source": ["from tensorflow.keras.models import Sequential\n", "from tensorflow.keras.layers import Dense, Activation\n", "import pandas as pd\n", "import io\n", "import os\n", "import requests\n", "import numpy as np\n", "from sklearn import metrics\n", "\n", "save_path = \".\"\n", "\n", "df = pd.read_csv(\n", "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\", \n", "    na_values=['NA', '?'])\n", "\n", "cars = df['name']\n", "\n", "# Handle missing value\n", "df['horsepower'] = df['horsepower'].fillna(df['horsepower'].median())\n", "\n", "# Pandas to Numpy\n", "x = df[['cylinders', 'displacement', 'horsepower', 'weight',\n", "       'acceleration', 'year', 'origin']].values\n", "y = df['mpg'].values # regression\n", "\n", "# Build the neural network\n", "model = Sequential()\n", "model.add(Dense(25, input_dim=x.shape[1], activation='relu')) # Hidden 1\n", "model.add(Dense(10, activation='relu')) # Hidden 2\n", "model.add(Dense(1)) # Output\n", "model.compile(loss='mean_squared_error', optimizer='adam')\n", "model.fit(x,y,verbose=2,epochs=100)\n", "\n", "# Predict\n", "pred = model.predict(x)\n", "\n", "# Measure RMSE error.  RMSE is common for regression.\n", "score = np.sqrt(metrics.mean_squared_error(pred,y))\n", "print(f\"Before save score (RMSE): {score}\")\n", "\n", "# save neural network structure to JSON (no weights)\n", "model_json = model.to_json()\n", "with open(os.path.join(save_path,\"network.json\"), \"w\") as json_file:\n", "    json_file.write(model_json)\n", "\n", "# save neural network structure to YAML (no weights)\n", "model_yaml = model.to_yaml()\n", "with open(os.path.join(save_path,\"network.yaml\"), \"w\") as yaml_file:\n", "    yaml_file.write(model_yaml)\n", "\n", "# save entire network to HDF5 (save everything, suggested)\n", "model.save(os.path.join(save_path,\"network.h5\"))"]}, {"cell_type": "markdown", "metadata": {}, "source": ["The code below sets up a neural network and reads the data (for predictions), but it does not clear the model directory or fit the neural network.  The weights from the previous fit are used."]}, {"cell_type": "markdown", "metadata": {}, "source": ["Now we reload the network and perform another prediction.  The RMSE should match the previous one exactly if the neural network was really saved and reloaded."]}, {"cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["After load score (RMSE): 6.3568661509704905\n"]}], "source": ["from tensorflow.keras.models import load_model\n", "model2 = load_model(os.path.join(save_path,\"network.h5\"))\n", "pred = model2.predict(x)\n", "# Measure RMSE error.  RMSE is common for regression.\n", "score = np.sqrt(metrics.mean_squared_error(pred,y))\n", "print(f\"After load score (RMSE): {score}\")"]}], "metadata": {"anaconda-cloud": {}, "kernelspec": {"display_name": "Python 3.6 (tensorflow)", "language": "python", "name": "tensorflow"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.5"}}, "nbformat": 4, "nbformat_minor": 4}