TF-Serving-Learner.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 align=\"center\"> Deploy Models with TensorFlow Serving and Docker</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2: Load and Preprocess Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 36
    },
    "colab_type": "code",
    "id": "ZQhSNW-CZUZh",
    "outputId": "0a0dd94e-b78f-4ffb-c819-5eae6efb6255"
   },
   "outputs": [],
   "source": [
    "#%%writefile -a train.py\n",
    "import os\n",
    "import time\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Souce: https://www.kaggle.com/snap/amazon-fine-food-reviews/data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%%writefile -a train.py\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 3: Build the Classification Model using TF Hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%%writefile -a train.py\n",
    "\n",
    "## https://tfhub.dev/google/tf2-preview/nnlm-en-dim50/1\n",
    "## https://tfhub.dev/google/tf2-preview/nnlm-en-dim128/1\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 4: Define Training Procedure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%%writefile -a train.py\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 5: Train and Export Model as Protobuf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%%writefile -a train.py\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 6: Test Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Negative Review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Positive Review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 7: TensorFlow Serving with Docker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`docker pull tensorflow/serving`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`docker run -p 8500:8500 \\\n",
    "            -p 8501:8501 \\\n",
    "            --mount type=bind,\\\n",
    "            source=amazon_review/,\\\n",
    "            target=/models/amazon_review \\\n",
    "            -e MODEL_NAME=amazon_review \\\n",
    "            -t tensorflow/serving`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 8: Setup a REST Client to Perform Model Predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Perform Model Prediction\n",
    "\n",
    "##### Support for gRPC and REST\n",
    "\n",
    "- TensorFlow Serving supports\n",
    "    - Remote Procedure Protocal (gRPC)\n",
    "    - Representational State Transfer (REST)\n",
    "- Consistent API structures\n",
    "- Server supports both standards simultaneously\n",
    "- Default ports:\n",
    "    - RPC: 8500\n",
    "    - REST: 8501"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Predictions via REST\n",
    "\n",
    "- Standard HTTP POST requests\n",
    "- Response is a JSON body with the prediction\n",
    "- Request from the default or specific model\n",
    "\n",
    "Default URI scheme:\n",
    "\n",
    "`http://{HOST}:{PORT}/v1/models/{MODEL_NAME}`\n",
    "\n",
    "Specific model versions:\n",
    "\n",
    "`http://{HOST}:{PORT}/v1/models/{MODEL_NAME}[/versions/{MODEL_VERSION}]:predict`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3NND75SMZUsP"
   },
   "outputs": [],
   "source": [
    "%%writefile tf_serving_rest_client.py\n",
    "import json\n",
    "import requests\n",
    "import sys\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 9: Setup a gRPC Client to Perform Model Predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Modified from [https://github.com/tensorflow/serving/blob/master/tensorflow_serving/example/mnist_client.py](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/example/mnist_client.py#L152)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Predictions via gRPC\n",
    "\n",
    "More sophisticated client-server connections\n",
    "\n",
    "- Prediction data has to be converted to the Protobuf format\n",
    "- Request types have designated types, e.g. float, int, bytes\n",
    "- Payloads need to be converted to base64\n",
    "- Connect to the server via gRPC stubs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### gRPC vs REST: When to use which API standard\n",
    "\n",
    "- Rest is easy to implement and debug\n",
    "- RPC is more network efficient, smaller payloads\n",
    "- RPC can provide much faster inferences!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jKJVOjDlZUvc"
   },
   "outputs": [],
   "source": [
    "%%writefile tf_serving_grpc_client.py\n",
    "'''import sys\n",
    "import grpc\n",
    "from grpc.beta import implementations\n",
    "import tensorflow as tf\n",
    "from tensorflow_serving.apis import predict_pb2\n",
    "from tensorflow_serving.apis import prediction_service_pb2, get_model_metadata_pb2\n",
    "from tensorflow_serving.apis import prediction_service_pb2_grpc\n",
    "\n",
    "\n",
    "def get_stub(host='127.0.0.1', port='8500'):\n",
    "    channel = grpc.insecure_channel('127.0.0.1:8500') \n",
    "    stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)\n",
    "    return stub\n",
    "\n",
    "\n",
    "def get_model_prediction(model_input, stub, model_name='amazon_review', signature_name='serving_default'):\n",
    "    \"\"\" no error handling at all, just poc\"\"\"\n",
    "    request = predict_pb2.PredictRequest()\n",
    "    request.model_spec.name = model_name\n",
    "    request.model_spec.signature_name = signature_name\n",
    "    request.inputs['input_input'].CopyFrom(tf.make_tensor_proto(model_input))\n",
    "    response = stub.Predict.future(request, 5.0)  # 5 seconds\n",
    "    return response.result().outputs[\"output\"].float_val\n",
    "\n",
    "\n",
    "def get_model_version(model_name, stub):\n",
    "    request = get_model_metadata_pb2.GetModelMetadataRequest()\n",
    "    request.model_spec.name = 'amazon_review'\n",
    "    request.metadata_field.append(\"signature_def\")\n",
    "    response = stub.GetModelMetadata(request, 10)\n",
    "    # signature of loaded model is available here: response.metadata['signature_def']\n",
    "    return response.model_spec.version.value\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    print(\"\\nCreate RPC connection ...\")\n",
    "    stub = get_stub()\n",
    "    while True:\n",
    "        print(\"\\nEnter an Amazon review [:q for Quit]\")\n",
    "        if sys.version_info[0] <= 3:\n",
    "            sentence = raw_input() if sys.version_info[0] < 3 else input()\n",
    "        if sentence == ':q':\n",
    "            break\n",
    "        model_input = [sentence]\n",
    "        model_prediction = get_model_prediction(model_input, stub)\n",
    "        print(\"The model predicted ...\")\n",
    "        print(model_prediction)'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TidfRe2VZU39"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xA3wNmDSZU66"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "deploy-TF-Serving.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}