notebooks/speculative-sampling/speculative-sampling.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ef2ed242-3561-464c-8d1c-cc3862e23702",
   "metadata": {},
   "source": [
    "# Text Generation via Speculative Sampling, KV Caching, and OpenVINO™\n",
    "\n",
    "As model sizes grow, Generative AI implementations require significant inference resources. This not only increases the cost per generation from a prompt, but also increases the power consumption used to serve such requests.\n",
    "\n",
    "Inference optimizations for text generation are essential for reducing costs and power consumption. When optimizing the inference process, the amount of time and energy required to generate text can be significantly reduced. This can lead to cost savings in terms of hardware and software, as well as reduced power consumption. Additionally, inference optimizations can help improve the accuracy of text generation as well as the speed at which it can be generated. This can lead to an improved user experience and increased efficiency in text-generation tasks. In summary, inference optimizations for text generation are essential to reduce costs and power consumption, while also improving the accuracy and speed of text generation.\n",
    "\n",
    "Another necessary condition is that the optimizations are compatible with each other. That is, implementing a certain optimization should not preclude other optimizations. There are several levels of optimizations that can provide significant speedup without \"bumping into each other\" in a way that will compromise overall efficiency.\n",
    "\n",
    "For details on this method, please refer to the paper by Chen et al, http://arxiv.org/abs/2302.01318. Additionally, there's an interesting proof of correctness of speculative sampling (showing that the original distribution is preserved) by Leviathan et al, http://arxiv.org/abs/2211.17192\n",
    "\n",
    "Our blog article describing this implementation with OpenVino is available at openvino.ai\n",
    "\n",
    "<img referrerpolicy=\"no-referrer-when-downgrade\" src=\"https://static.scarf.sh/a.png?x-pxid=5b5a4db0-7875-4bfb-bdbd-01698b5b1a77&file=notebooks/speculative-sampling/speculative-sampling.ipynb\" />\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f97c435a",
   "metadata": {},
   "source": [
    "#### Table of contents:\n",
    "\n",
    "- [Prerequisites](#Prerequisites)\n",
    "    - [Select inference device](#Select-inference-device)\n",
    "- [Create autoregressive and speculative forms of sampling with KV Cache support](#Create-autoregressive-and-speculative-forms-of-sampling-with-KV-Cache-support)\n",
    "    - [Setup imports](#Setup-imports)\n",
    "    - [Prepare autoregressive sampling](#Prepare-autoregressive-sampling)\n",
    "    - [Prepare speculative sampling](#Prepare-speculative-sampling)\n",
    "- [Main generation function](#Main-generation-function)\n",
    "    - [Download and Convert Model](#Download-and-Convert-Model)\n",
    "### Installation Instructions\n",
    "\n",
    "This is a self-contained example that relies solely on its own code.\n",
    "\n",
    "We recommend  running the notebook in a virtual environment. You only need a Jupyter server to start.\n",
    "For details, please refer to [Installation Guide](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/README.md#-installation-guide)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "08aa16b1-d2f6-4a3a-abfb-5ec278133c80",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "\n",
    "First, we should install the [Hugging Face Optimum](https://huggingface.co/docs/optimum/installation) library accelerated by OpenVINO integration.\n",
    "The Hugging Face Optimum Intel API is a high-level API that enables us to convert and quantize models from the Hugging Face Transformers library to the OpenVINO™ IR format. For more details, refer to the [Hugging Face Optimum Intel documentation](https://huggingface.co/docs/optimum/intel/inference).\n",
    "\n",
    "We will also need to install transformers (HuggingFace) and some other useful modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4421fc85-bed6-4a62-b8fa-19c7ba474891",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install -Uq pip\n",
    "%pip uninstall -q -y optimum optimum-intel\n",
    "%pip install --pre -Uq \"openvino>=2024.2.0\" openvino-tokenizers[transformers] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly\n",
    "%pip install -q --upgrade transformers \"torch>=2.1\" \"torchvision\" \"gradio>=4.19\" accelerate \"onnx<1.16.2\" ipywidgets --extra-index-url https://download.pytorch.org/whl/cpu\n",
    "%pip install -q \"git+https://github.com/huggingface/optimum-intel.git\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "367f84f8-33e8-4ad6-bd40-e6fd41d2d703",
   "metadata": {},
   "source": [
    "### Select inference device\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "\n",
    "Select the device from dropdown list for running inference using OpenVINO."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6ddd57de-9f41-403c-bccc-8d3118654a24",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "271cc0d2e97b494cb57df890114cb682",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Device:', options=('CPU', 'GPU.0', 'GPU.1', 'AUTO'), value='CPU')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "r = requests.get(\n",
    "    url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py\",\n",
    ")\n",
    "open(\"notebook_utils.py\", \"w\").write(r.text)\n",
    "\n",
    "from notebook_utils import device_widget\n",
    "\n",
    "device = device_widget()\n",
    "\n",
    "device"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b6d9c4a5-ef75-4076-9f1c-f45a2259ec46",
   "metadata": {},
   "source": [
    "## Create autoregressive and speculative forms of sampling with KV Cache support\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    " \n",
    "Text generation is often done in an autoregressive fashion. We will all support a KV cache (aka Past Value Cache) in the code. Note that we are using greedy sampling. We do not adjust other text generation parameters (e.g. temperature) so keep this illustration of speculative sampling as simple and understandable as possible."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b9b5da4d-d2fd-440b-b204-7fbc6966dd1f",
   "metadata": {},
   "source": [
    "### Setup imports\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6f976094-8603-42c4-8f18-a32ba6d7192e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import time\n",
    "import numpy as np\n",
    "import openvino as ov"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c58611d6-0a91-4efd-976e-4221acbb43cd",
   "metadata": {},
   "source": [
    "### Prepare autoregressive sampling\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "52ac10a5-3141-4227-8f0b-0617acd027c8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def autoregressive_sampling_with_pkv(input, model, N=30):\n",
    "    input_ids, attention_mask = input.input_ids, input.attention_mask\n",
    "    seq_len = input_ids.shape[-1]\n",
    "    position_ids = np.arange(0, seq_len, dtype=np.int64).reshape([-1, seq_len])\n",
    "\n",
    "    # in all subsequent inferences we feed tokens one by one,\n",
    "    # but for the first one we feed the whole encoded prompt\n",
    "    request = model.create_infer_request()\n",
    "    request.infer((input_ids, attention_mask, position_ids, np.array([0])))\n",
    "    next_token = np.argmax(request.results[\"logits\"][:, -1]).reshape([1])\n",
    "\n",
    "    all_tokens = []\n",
    "    all_tokens.extend(input_ids[0])\n",
    "    all_tokens.append(next_token[0])\n",
    "\n",
    "    while seq_len < N:\n",
    "        input_ids = next_token.reshape([1, 1])\n",
    "        attention_mask = np.concatenate((attention_mask, np.array([1]).reshape([1, 1])), axis=1)\n",
    "        position_ids = np.array([attention_mask.shape[1]]).reshape([1, 1])\n",
    "\n",
    "        request.infer((input_ids, attention_mask, position_ids, np.array([0])))\n",
    "        next_token = np.argmax(request.results[\"logits\"][:, -1])\n",
    "        all_tokens.append(next_token)\n",
    "        seq_len += 1\n",
    "\n",
    "    return all_tokens"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "27a01739-1363-42ef-927f-6a340bdbe7ba",
   "metadata": {},
   "source": [
    "### Prepare speculative sampling\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "* Step 1: With speculative sampling, we first generate K samples from the draft model (in an autoregressive manner).\n",
    "* Step 2: These are now candidates to examine using the main model (step 2) using a batch size of K.\n",
    "* Step 3: We go through each K predicted tokens, and if tokens differ, we stop and keep the last token predicted by the main model.\n",
    "* Step 4: We update KV-cache dropping keys & values for differing tokens and repeat Step 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9fde1b3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_state(request, seq_len):\n",
    "    for state in request.query_state():\n",
    "        old_seq_len = state.state.shape[2]\n",
    "        if seq_len >= old_seq_len:\n",
    "            continue\n",
    "        # After the inference request, key/values have shape [BATCH_SIZE, seq_len + K, vocab_size].\n",
    "        # Increment the sequence length by the number of matched tokens, and\n",
    "        # trim the KV cache to match the new sequence length.\n",
    "        state.state = ov.Tensor(state.state.data[:, :, :seq_len])\n",
    "\n",
    "\n",
    "def speculative_sampling_with_pkv(input, draft_model, main_model, K, N=30, **kwargs):\n",
    "    input_ids, attention_mask = input.input_ids, input.attention_mask\n",
    "    # seq_len number of key/values or number of already processed input tokens\n",
    "    seq_len = input_ids.shape[-1]\n",
    "    position_ids = np.arange(0, seq_len, dtype=np.int64).reshape([-1, seq_len])\n",
    "\n",
    "    draft_request = draft_model.create_infer_request()\n",
    "    draft_request.infer((input_ids, attention_mask, position_ids, np.array([0])))\n",
    "\n",
    "    main_request = main_model.create_infer_request()\n",
    "    main_request.infer((input_ids, attention_mask, position_ids, np.array([0])))\n",
    "    first_token = np.argmax(main_request.results[\"logits\"][:, -1]).reshape([1])\n",
    "\n",
    "    all_tokens = []\n",
    "    all_tokens.extend(input_ids[0])\n",
    "    all_tokens.append(first_token[0])\n",
    "\n",
    "    accum_draft_tokens = []\n",
    "    while seq_len < N:\n",
    "        next_token = first_token\n",
    "        for i in range(K):\n",
    "            input_ids = next_token.reshape([1, 1])\n",
    "            attention_mask = np.concatenate((attention_mask, np.array([1]).reshape([1, 1])), axis=1)\n",
    "            position_ids = np.array([attention_mask.shape[1]]).reshape([1, 1])\n",
    "\n",
    "            draft_request.infer((input_ids, attention_mask, position_ids, np.array([0])))\n",
    "            next_token = np.argmax(draft_request.results[\"logits\"][:, -1])\n",
    "            accum_draft_tokens.append(next_token)\n",
    "\n",
    "        # main model will give also K out tokens\n",
    "        # feed the same first token to the main model and do not give the last token generated by the draft\n",
    "        input_ids = np.concatenate((first_token.reshape([1]), accum_draft_tokens[:-1])).reshape([1, -1])\n",
    "        attention_mask = np.ones((1, seq_len + K))\n",
    "        position_ids = np.arange(seq_len, seq_len + K, dtype=np.int64).reshape([1, -1])\n",
    "\n",
    "        main_request.infer((input_ids, attention_mask, position_ids, np.array([0])))\n",
    "        next_tokens = np.argmax(main_request.results[\"logits\"], axis=-1)[0]\n",
    "\n",
    "        # if disagrees from the very beggining then context will be expanded only for one element\n",
    "        # all elements match then context will be expanded to K elements\n",
    "        for disagree_idx, (t1, t2) in enumerate(zip(accum_draft_tokens, next_tokens)):\n",
    "            if t1 != t2:\n",
    "                break\n",
    "\n",
    "        first_token = next_tokens[disagree_idx]\n",
    "        all_tokens.extend(next_tokens[: disagree_idx + 1])\n",
    "        seq_len += disagree_idx + 1\n",
    "\n",
    "        # cut key/values depending on the position where disagreement starts\n",
    "        update_state(draft_request, seq_len)\n",
    "        update_state(main_request, seq_len)\n",
    "\n",
    "        attention_mask = np.ones((1, seq_len))\n",
    "        accum_draft_tokens = []\n",
    "    all_tokens.extend(accum_draft_tokens)\n",
    "    return all_tokens"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "583202d2-6d29-4729-af2e-232d3ee0bc2c",
   "metadata": {},
   "source": [
    "## Main generation function\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d6993840-e8b2-4c26-8da4-e1b046c3f3cc",
   "metadata": {},
   "source": [
    "### Download and Convert Model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Optimum Intel can be used to load optimized models from the [Hugging Face Hub](https://huggingface.co/docs/optimum/intel/hf.co/models) and create pipelines to run an inference with OpenVINO Runtime using Hugging Face APIs. For speculative decoding we need to manually update states, therefore we will use directly openvino inference api, and optimum only for model conversion.\n",
    ">To download Llama-2-7b-chat-hf, you will need to accept license agreement. You must be a registered user in 🤗 Hugging Face Hub. Please visit HuggingFace model [card](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf), carefully read terms of usage and click accept button. You will need to use an access token for the code below to run. For more information on access tokens, refer to this section of the documentation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "82596e66",
   "metadata": {
    "test_replace": {
     "main_model_id = \"meta-llama/Llama-2-7b-chat-hf\"": "main_model_id =\"TinyLlama/TinyLlama-1.1B-Chat-v1.0\""
    }
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "main_model_id = \"meta-llama/Llama-2-7b-chat-hf\"\n",
    "main_model_path = Path(\"Llama-2-7b-chat-hf\")\n",
    "draft_model_id = \"TinyLlama/TinyLlama-1.1B-Chat-v1.0\"\n",
    "draft_model_path = Path(\"TinyLlama-1.1B-Chat-v1.0\")\n",
    "\n",
    "from transformers import AutoTokenizer\n",
    "\n",
    "main_tokenizer = AutoTokenizer.from_pretrained(main_model_id)\n",
    "draft_tokenizer = AutoTokenizer.from_pretrained(draft_model_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "995dc418",
   "metadata": {},
   "outputs": [],
   "source": [
    "# In order for speculative sampling to work, both main and draft tokenizers should be the same.\n",
    "token_test_txt = \"text to ensure tokenizers work the same, as of 2024\"\n",
    "tokens_1 = draft_tokenizer(token_test_txt, return_tensors=\"pt\").input_ids\n",
    "tokens_2 = main_tokenizer(token_test_txt, return_tensors=\"pt\").input_ids\n",
    "\n",
    "assert all((tokens_1 - tokens_2)[0] == 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "78113302",
   "metadata": {},
   "outputs": [],
   "source": [
    "if not main_model_path.exists():\n",
    "    !optimum-cli export openvino --model $main_model_id --weight-format fp16 {main_model_path}\n",
    "if not draft_model_path.exists():\n",
    "    !optimum-cli export openvino --model $draft_model_id --weight-format fp16 {draft_model_path}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "824a2f25",
   "metadata": {},
   "source": [
    "Infer directly using OpenVINO Inference Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "67fb4f9d-5877-48d8-8eff-c30ff6974d7a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "core = ov.Core()\n",
    "draft_ov_model = core.read_model(draft_model_path / \"openvino_model.xml\")\n",
    "draft_model = core.compile_model(draft_ov_model, device_name=device.value)\n",
    "\n",
    "main_ov_model = core.read_model(main_model_path / \"openvino_model.xml\")\n",
    "main_model = core.compile_model(main_ov_model, device_name=device.value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4469807a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def main(\n",
    "    prompt: str,\n",
    "    n_tokens_to_generate: int = 75,\n",
    "    K: int = 5,\n",
    "    seed: int = 5555,\n",
    "):\n",
    "    # seed numpy rng\n",
    "    np.random.seed(seed)\n",
    "    tokenized = main_tokenizer(prompt, return_tensors=\"pt\")\n",
    "\n",
    "    def run_autoregressive_sampling_fn(decode_fn, tokenized, **kwargs):\n",
    "        start = time.perf_counter()\n",
    "        output_ids = decode_fn(tokenized, **kwargs)\n",
    "        text = main_tokenizer.decode(output_ids, skip_special_tokens=True)\n",
    "        elapsed_time = time.perf_counter() - start\n",
    "        return text, elapsed_time\n",
    "\n",
    "    def run_speculative_sampling_fn(decode_fn, input_ids, **kwargs):\n",
    "        start = time.perf_counter()\n",
    "        output_ids = decode_fn(input_ids, **kwargs)\n",
    "        text = main_tokenizer.decode(output_ids, skip_special_tokens=True)\n",
    "        elapsed_time = time.perf_counter() - start\n",
    "        return text, elapsed_time\n",
    "\n",
    "    autoregressive_text, autoregressive_time = run_autoregressive_sampling_fn(\n",
    "        autoregressive_sampling_with_pkv,\n",
    "        tokenized,\n",
    "        model=main_model,\n",
    "        N=n_tokens_to_generate,\n",
    "    )\n",
    "\n",
    "    speculative_text, speculative_time = run_speculative_sampling_fn(\n",
    "        speculative_sampling_with_pkv,\n",
    "        tokenized,\n",
    "        main_model=main_model,\n",
    "        draft_model=draft_model,\n",
    "        N=n_tokens_to_generate,\n",
    "        K=K,\n",
    "    )\n",
    "\n",
    "    # Format results for output in gradio\n",
    "    out = \"\\n\" + \"Autoregressive Decode\" + \"\\n\" + \"---------------------\" + \"\\n\"\n",
    "    out = out + f\"Time = {autoregressive_time:.2f}s\" + \"\\n\" + f\"Text = {autoregressive_text}\" + \"\\n\"\n",
    "    out = out + \"\\n\" + \"Speculative Decode\" + \"\\n\" + \"------------------\" + \"\\n\"\n",
    "    out = out + f\"Time = {speculative_time:.2f}s\" + \"\\n\" + f\"Text = {speculative_text}\"\n",
    "    return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "18eb331b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-04-17 10:21:41.642283: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n",
      "2024-04-17 10:21:41.644834: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.\n",
      "2024-04-17 10:21:41.677055: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2024-04-17 10:21:41.677093: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2024-04-17 10:21:41.677119: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2024-04-17 10:21:41.683198: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.\n",
      "2024-04-17 10:21:41.683977: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2024-04-17 10:21:42.477656: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Autoregressive Decode\n",
      "---------------------\n",
      "Time = 44.39s\n",
      "Text = Alan Turing was a British mathematician, computer scientist, and codebreaker who played a pivotal role in cracking the German Enigma code during World War II. He was also a pioneer in the field of artificial intelligence and made significant contributions to the development of computer science.\n",
      "\n",
      "Turing was born on June 23, 1912, in London, England. He was educated at Cambridge University, where he earned a degree in mathematics in \n",
      "\n",
      "Speculative Decode\n",
      "------------------\n",
      "Time = 22.96s\n",
      "Text = Alan Turing was a British mathematician, computer scientist, and codebreaker who played a pivotal role in cracking the German Enigma code during World War II. He was also a pioneer in the field of artificial intelligence and made significant contributions to the development of computer science.\n",
      "\n",
      "Turing was born on June 23, 1912, in London, England. He was educated at Cambridge University, where he earned a degree in mathematics in 1\n"
     ]
    }
   ],
   "source": [
    "res = main(\"Alan Turing was a\", n_tokens_to_generate=100)\n",
    "print(res)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "757d1d26",
   "metadata": {},
   "outputs": [],
   "source": [
    "if not Path(\"gradio_helper.py\").exists():\n",
    "    r = requests.get(url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/speculative-sampling/gradio_helper.py\")\n",
    "    open(\"gradio_helper.py\", \"w\").write(r.text)\n",
    "\n",
    "from gradio_helper import make_demo\n",
    "\n",
    "demo = make_demo(fn=main)\n",
    "\n",
    "try:\n",
    "    demo.launch(debug=True)\n",
    "except Exception:\n",
    "    demo.launch(share=True, debug=True)\n",
    "# If you are launching remotely, specify server_name and server_port\n",
    "# EXAMPLE: `demo.launch(server_name='your server name', server_port='server port in int')`\n",
    "# To learn more please refer to the Gradio docs: https://gradio.app/docs/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba8aa6cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# please uncomment and run this cell for stopping gradio interface\n",
    "# demo.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  },
  "openvino_notebooks": {
   "imageUrl": "",
   "tags": {
    "categories": [
     "Model Demos"
    ],
    "libraries": [],
    "other": [
     "LLM"
    ],
    "tasks": [
     "Text Generation"
    ]
   }
  },
  "vscode": {
   "interpreter": {
    "hash": "cec18e25feb9469b5ff1085a8097bdcd86db6a4ac301d6aeff87d0f3e7ce4ca5"
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}