notebooks/213-question-answering/213-question-answering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Interactive question answering with OpenVINO™\n",
    "\n",
    "This demo shows interactive question answering with OpenVINO, using [small BERT-large-like model](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/intel/bert-small-uncased-whole-word-masking-squad-int8-0002) distilled and quantized to `INT8` on SQuAD v1.1 training set from larger BERT-large model. The model comes from [Open Model Zoo](https://github.com/openvinotoolkit/open_model_zoo/). Final part of this notebook provides live inference results from your inputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import operator\n",
    "import time\n",
    "from urllib import parse\n",
    "\n",
    "import numpy as np\n",
    "from openvino.runtime import Core\n",
    "\n",
    "import html_reader as reader\n",
    "import tokens_bert as tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## The model\n",
    "\n",
    "### Download the model\n",
    "\n",
    "Use `omz_downloader`, which is a command-line tool from the `openvino-dev` package. The `omz_downloader` tool automatically creates a directory structure and downloads the selected model. If the model is already downloaded, this step is skipped.\n",
    "\n",
    "You can download and use any of the following models: `bert-large-uncased-whole-word-masking-squad-0001`, `bert-large-uncased-whole-word-masking-squad-int8-0001`, `bert-small-uncased-whole-word-masking-squad-0001`, `bert-small-uncased-whole-word-masking-squad-0002`, `bert-small-uncased-whole-word-masking-squad-int8-0002`, just change the model name in the code below. All of these models are already converted to OpenVINO Intermediate Representation (OpenVINO IR), so there is no need to use `omz_converter`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# A directory where the model will be downloaded.\n",
    "base_model_dir = \"model\"\n",
    "\n",
    "# Selected precision (FP32, FP16, FP16-INT8).\n",
    "precision = \"FP16-INT8\"\n",
    "\n",
    "# The name of the model from Open Model Zoo.\n",
    "model_name = \"bert-small-uncased-whole-word-masking-squad-int8-0002\"\n",
    "\n",
    "model_path = f\"model/intel/{model_name}/{precision}/{model_name}.xml\"\n",
    "model_weights_path = f\"model/intel/{model_name}/{precision}/{model_name}.bin\"\n",
    "\n",
    "download_command = f\"omz_downloader \" \\\n",
    "                   f\"--name {model_name} \" \\\n",
    "                   f\"--precision {precision} \" \\\n",
    "                   f\"--output_dir {base_model_dir} \" \\\n",
    "                   f\"--cache_dir {base_model_dir}\"\n",
    "! $download_command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Load the model\n",
    "\n",
    "Downloaded models are located in a fixed structure, which indicates a vendor, a model name and a precision. Only a few lines of code are required to run the model. First, create an OpenVINO Runtime object. Then, read the network architecture and model weights from the `.xml` and `.bin` files. Finally, compile the network for the desired device. You can choose `CPU` or `GPU` for this model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Initialize OpenVINO Runtime.\n",
    "core = Core()\n",
    "# Read the network and corresponding weights from a file.\n",
    "model = core.read_model(model_path)\n",
    "# Load the model on CPU (you can use GPU as well).\n",
    "compiled_model = core.compile_model(model=model, device_name=\"CPU\")\n",
    "\n",
    "# Get input and output names of nodes.\n",
    "input_keys = list(compiled_model.inputs)\n",
    "output_keys = list(compiled_model.outputs)\n",
    "\n",
    "# Get the network input size.\n",
    "input_size = compiled_model.input(0).shape[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Input keys are the names of the input nodes and output keys contain names of output nodes of the network. There are 4 inputs and 2 outputs for BERT-large-like model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "[i.any_name for i in input_keys], [o.any_name for o in output_keys]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Processing\n",
    "\n",
    "NLP models usually take a list of tokens as a standard input. A token is a single word converted to some integer. To provide the proper input, you need the vocabulary for such mapping. You also need to define some special tokens, such as separators or padding and a function to load the content from provided URLs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# The path to the vocabulary file.\n",
    "vocab_file_path = \"data/vocab.txt\"\n",
    "\n",
    "# Create a dictionary with words and their indices.\n",
    "vocab = tokens.load_vocab_file(vocab_file_path)\n",
    "\n",
    "# Define special tokens.\n",
    "cls_token = vocab[\"[CLS]\"]\n",
    "pad_token = vocab[\"[PAD]\"]\n",
    "sep_token = vocab[\"[SEP]\"]\n",
    "\n",
    "\n",
    "# A function to load text from given urls.\n",
    "def load_context(sources):\n",
    "    input_urls = []\n",
    "    paragraphs = []\n",
    "    for source in sources:\n",
    "        result = parse.urlparse(source)\n",
    "        if all([result.scheme, result.netloc]):\n",
    "            input_urls.append(source)\n",
    "        else:\n",
    "            paragraphs.append(source)\n",
    "\n",
    "    paragraphs.extend(reader.get_paragraphs(input_urls))\n",
    "    # Produce one big context string.\n",
    "    return \"\\n\".join(paragraphs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "\n",
    "### Preprocessing\n",
    "\n",
    "The input size in this case is 384 tokens long. The main input (`input_ids`) to used BERT model consists of two parts: question tokens and context tokens separated by some special tokens. \n",
    "\n",
    "If `question + context` are shorter than 384 tokens, padding tokens are added. If `question + context` is longer than 384 tokens, the context must be split into parts and the question with different parts of context must be fed to the network many times. \n",
    "\n",
    "Use overlapping, so neighbor parts of the context are overlapped by half size of the context part (if the context part equals 300 tokens, neighbor context parts overlap with 150 tokens). You also need to provide the following sequences of integer values: \n",
    "\n",
    "- `attention_mask` - a sequence of integer values representing the mask of valid values in the input. \n",
    "- `token_type_ids` - a sequence of integer values representing the segmentation of `input_ids` into question and context. \n",
    "- `position_ids` - a sequence of integer values from 0 to 383 representing the position index for each input token. \n",
    "\n",
    "For more information, refer to the **Input** section of [BERT model documentation](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/intel/bert-small-uncased-whole-word-masking-squad-int8-0002#input)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# A generator of a sequence of inputs.\n",
    "def prepare_input(question_tokens, context_tokens):\n",
    "    # A length of question in tokens.\n",
    "    question_len = len(question_tokens)\n",
    "    # The context part size.\n",
    "    context_len = input_size - question_len - 3\n",
    "\n",
    "    if context_len < 16:\n",
    "        raise RuntimeError(\"Question is too long in comparison to input size. No space for context\")\n",
    "\n",
    "    # Take parts of the context with overlapping by 0.5.\n",
    "    for start in range(0, max(1, len(context_tokens) - context_len), context_len // 2):\n",
    "        # A part of the context.\n",
    "        part_context_tokens = context_tokens[start:start + context_len]\n",
    "        # The input: a question and the context separated by special tokens.\n",
    "        input_ids = [cls_token] + question_tokens + [sep_token] + part_context_tokens + [sep_token]\n",
    "        # 1 for any index if there is no padding token, 0 otherwise.\n",
    "        attention_mask = [1] * len(input_ids)\n",
    "        # 0 for question tokens, 1 for context part.\n",
    "        token_type_ids = [0] * (question_len + 2) + [1] * (len(part_context_tokens) + 1)\n",
    "\n",
    "        # Add padding at the end.\n",
    "        (input_ids, attention_mask, token_type_ids), pad_number = pad(input_ids=input_ids,\n",
    "                                                                      attention_mask=attention_mask,\n",
    "                                                                      token_type_ids=token_type_ids)\n",
    "\n",
    "        # Create an input to feed the model.\n",
    "        input_dict = {\n",
    "            \"input_ids\": np.array([input_ids], dtype=np.int32),\n",
    "            \"attention_mask\": np.array([attention_mask], dtype=np.int32),\n",
    "            \"token_type_ids\": np.array([token_type_ids], dtype=np.int32),\n",
    "        }\n",
    "\n",
    "        # Some models require additional position_ids.\n",
    "        if \"position_ids\" in [i_key.any_name for i_key in input_keys]:\n",
    "            position_ids = np.arange(len(input_ids))\n",
    "            input_dict[\"position_ids\"] = np.array([position_ids], dtype=np.int32)\n",
    "\n",
    "        yield input_dict, pad_number, start\n",
    "\n",
    "\n",
    "# A function to add padding.\n",
    "def pad(input_ids, attention_mask, token_type_ids):\n",
    "    # How many padding tokens.\n",
    "    diff_input_size = input_size - len(input_ids)\n",
    "\n",
    "    if diff_input_size > 0:\n",
    "        # Add padding to all the inputs.\n",
    "        input_ids = input_ids + [pad_token] * diff_input_size\n",
    "        attention_mask = attention_mask + [0] * diff_input_size\n",
    "        token_type_ids = token_type_ids + [0] * diff_input_size\n",
    "\n",
    "    return (input_ids, attention_mask, token_type_ids), diff_input_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Postprocessing\n",
    "\n",
    "The results from the network are raw (logits). Use the softmax function to get the probability distribution. Then, find the best answer in the current part of the context (the highest score) and return the score and the context range for the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Based on https://github.com/openvinotoolkit/open_model_zoo/blob/bf03f505a650bafe8da03d2747a8b55c5cb2ef16/demos/common/python/openvino/model_zoo/model_api/models/bert.py#L163\n",
    "def postprocess(output_start, output_end, question_tokens, context_tokens_start_end, padding, start_idx):\n",
    "\n",
    "    def get_score(logits):\n",
    "        out = np.exp(logits)\n",
    "        return out / out.sum(axis=-1)\n",
    "\n",
    "    # Get start-end scores for the context.\n",
    "    score_start = get_score(output_start)\n",
    "    score_end = get_score(output_end)\n",
    "\n",
    "    # An index of the first context token in a tensor.\n",
    "    context_start_idx = len(question_tokens) + 2\n",
    "    # An index of the last+1 context token in a tensor.\n",
    "    context_end_idx = input_size - padding - 1\n",
    "\n",
    "    # Find product of all start-end combinations to find the best one.\n",
    "    max_score, max_start, max_end = find_best_answer_window(start_score=score_start,\n",
    "                                                            end_score=score_end,\n",
    "                                                            context_start_idx=context_start_idx,\n",
    "                                                            context_end_idx=context_end_idx)\n",
    "\n",
    "    # Convert to context text start-end index.\n",
    "    max_start = context_tokens_start_end[max_start + start_idx][0]\n",
    "    max_end = context_tokens_start_end[max_end + start_idx][1]\n",
    "\n",
    "    return max_score, max_start, max_end\n",
    "\n",
    "\n",
    "# Based on https://github.com/openvinotoolkit/open_model_zoo/blob/bf03f505a650bafe8da03d2747a8b55c5cb2ef16/demos/common/python/openvino/model_zoo/model_api/models/bert.py#L188\n",
    "def find_best_answer_window(start_score, end_score, context_start_idx, context_end_idx):\n",
    "    context_len = context_end_idx - context_start_idx\n",
    "    score_mat = np.matmul(\n",
    "        start_score[context_start_idx:context_end_idx].reshape((context_len, 1)),\n",
    "        end_score[context_start_idx:context_end_idx].reshape((1, context_len)),\n",
    "    )\n",
    "    # Reset candidates with end before start.\n",
    "    score_mat = np.triu(score_mat)\n",
    "    # Reset long candidates (>16 words).\n",
    "    score_mat = np.tril(score_mat, 16)\n",
    "    # Find the best start-end pair.\n",
    "    max_s, max_e = divmod(score_mat.flatten().argmax(), score_mat.shape[1])\n",
    "    max_score = score_mat[max_s, max_e]\n",
    "\n",
    "    return max_score, max_s, max_e"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    " First, create a list of tokens from the context and the question. Then, find the best answer by trying different parts of the context. The best answer should come with the highest score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def get_best_answer(question, context):\n",
    "    # Convert the context string to tokens.\n",
    "    context_tokens, context_tokens_start_end = tokens.text_to_tokens(text=context.lower(),\n",
    "                                                                     vocab=vocab)\n",
    "    # Convert the question string to tokens.\n",
    "    question_tokens, _ = tokens.text_to_tokens(text=question.lower(), vocab=vocab)\n",
    "\n",
    "    results = []\n",
    "    # Iterate through different parts of the context.\n",
    "    for network_input, padding, start_idx in prepare_input(question_tokens=question_tokens,\n",
    "                                                           context_tokens=context_tokens):\n",
    "        # Get output layers.\n",
    "        output_start_key = compiled_model.output(\"output_s\")\n",
    "        output_end_key = compiled_model.output(\"output_e\")\n",
    "\n",
    "        # OpenVINO inference.\n",
    "        result = compiled_model(network_input)\n",
    "        # Postprocess the result, getting the score and context range for the answer.\n",
    "        score_start_end = postprocess(output_start=result[output_start_key][0],\n",
    "                                      output_end=result[output_end_key][0],\n",
    "                                      question_tokens=question_tokens,\n",
    "                                      context_tokens_start_end=context_tokens_start_end,\n",
    "                                      padding=padding,\n",
    "                                      start_idx=start_idx)\n",
    "        results.append(score_start_end)\n",
    "\n",
    "    # Find the highest score.\n",
    "    answer = max(results, key=operator.itemgetter(0))\n",
    "    # Return the part of the context, which is already an answer.\n",
    "    return context[answer[1]:answer[2]], answer[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Main Processing Function\n",
    "\n",
    "Run question answering on a specific knowledge base (websites) and iterate through the questions. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "test_replace": {
     "input()": "\"What does OpenVINO mean?\"",
     "while True": "for i in range(1)"
    }
   },
   "outputs": [],
   "source": [
    "def run_question_answering(sources, example_question=None):\n",
    "    print(f\"Context: {sources}\", flush=True)\n",
    "    context = load_context(sources)\n",
    "\n",
    "    if len(context) == 0:\n",
    "        print(\"Error: Empty context or outside paragraphs\")\n",
    "        return\n",
    "\n",
    "    if example_question is not None:\n",
    "        start_time = time.perf_counter()\n",
    "        answer, score = get_best_answer(question=example_question, context=context)\n",
    "        end_time = time.perf_counter()\n",
    "\n",
    "        print(f\"Question: {example_question}\")\n",
    "        print(f\"Answer: {answer}\")\n",
    "        print(f\"Score: {score:.2f}\")\n",
    "        print(f\"Time: {end_time - start_time:.2f}s\")\n",
    "    else:\n",
    "        while True:\n",
    "            question = input()\n",
    "            # if no question - break\n",
    "            if question == \"\":\n",
    "                break\n",
    "\n",
    "            # measure processing time\n",
    "            start_time = time.perf_counter()\n",
    "            answer, score = get_best_answer(question=question, context=context)\n",
    "            end_time = time.perf_counter()\n",
    "\n",
    "            print(f\"Question: {question}\")\n",
    "            print(f\"Answer: {answer}\")\n",
    "            print(f\"Score: {score:.2f}\")\n",
    "            print(f\"Time: {end_time - start_time:.2f}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Run\n",
    "\n",
    "### Run on local paragraphs\n",
    "\n",
    "Change sources to your own to answer your questions. You can use as many sources as you want. Usually, you need to wait a few seconds for the answer, but the longer the context, the longer the waiting time. The model is very limited and sensitive for the input. The answer can depend on whether there is a question mark at the end. The model will try to answer any of your questions even if there is no good answer in the context. Therefore, in such cases, you can see random results.\n",
    "\n",
    "Sample source: a paragraph from [Computational complexity theory](https://rajpurkar.github.io/SQuAD-explorer/explore/v2.0/dev/Computational_complexity_theory.html)\n",
    "\n",
    "Sample questions:\n",
    "- What is the term for a task that generally lends itself to being solved by a computer?\n",
    "- By what main attribute are computational problems classified utilizing computational complexity theory?\n",
    "- What branch of theoretical computer science deals with broadly classifying computational problems by difficulty and class of relationship?\n",
    "\n",
    "If you want to stop the processing just put an empty string.\n",
    "\n",
    "**First, run the code below. If you want to run it in interactive mode set `example_question` as `None`, run the code, and then put your questions in the box.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "sources = [\"Computational complexity theory is a branch of the theory of computation in theoretical computer \"\n",
    "           \"science that focuses on classifying computational problems according to their inherent difficulty, \"\n",
    "           \"and relating those classes to each other. A computational problem is understood to be a task that \"\n",
    "           \"is in principle amenable to being solved by a computer, which is equivalent to stating that the \"\n",
    "           \"problem may be solved by mechanical application of mathematical steps, such as an algorithm.\"]\n",
    "\n",
    "question = \"What is the term for a task that generally lends itself to being solved by a computer?\"\n",
    "\n",
    "run_question_answering(sources, example_question=question)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run on websites\n",
    "\n",
    "You can also provide urls. Note that the context (a knowledge base) is built from paragraphs on websites. If some information is outside the paragraphs, the algorithm wil not be able to find it.\n",
    "\n",
    "Sample source: [OpenVINO wiki](https://en.wikipedia.org/wiki/OpenVINO)\n",
    "\n",
    "Sample questions:\n",
    "- What does OpenVINO mean?\n",
    "- What is the license for OpenVINO?\n",
    "- Where can you deploy OpenVINO code?\n",
    "\n",
    "If you want to stop the processing just put an empty string.\n",
    "\n",
    "**First, run the code below and then put your questions in the box.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "sources = [\"https://en.wikipedia.org/wiki/OpenVINO\"]\n",
    "\n",
    "question = \"What does OpenVINO mean?\"\n",
    "\n",
    "run_question_answering(sources, example_question=None)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}