sdk/python/foundation-models/huggingface/inference/question-answering/question-answering-online-endpoint.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy question-answering models from HuggingFaceHub to AzureML Online Endpoints\n",
    "\n",
    "This sample shows how to deploy `question-answering` models from the HuggingFaceHub to an online endpoint for inference. Learn more about `question-answering` task: https://huggingface.co/tasks/question-answering\n",
    "\n",
    "A large set of models hosted on [Hugging Face Hub](https://huggingface.co/models) are available in the Hugging Face Hub collection in AzureML Model Catalog. This collection is powered by the Hugging Face Hub community registry. Integration with the AzureML Model Catalog enables seamless deployment of Hugging Face Hub models in AzureML. _todo: learn more link_\n",
    "\n",
    "### Outline\n",
    "* Set up pre-requisites.\n",
    "* Pick a model to deploy.\n",
    "* Deploy the model for real time inference.\n",
    "* Try sample inference.\n",
    "* Clean up resources."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up pre-requisites\n",
    "* Install dependencies\n",
    "* Connect to AzureML Workspace. Learn more at [set up SDK authentication](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-setup-authentication?tabs=sdk). Replace  `<WORKSPACE_NAME>`, `<RESOURCE_GROUP>` and `<SUBSCRIPTION_ID>` below.\n",
    "* Connect to `HuggingFaceHub` community registry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml import MLClient\n",
    "from azure.identity import (\n",
    "    DefaultAzureCredential,\n",
    "    InteractiveBrowserCredential,\n",
    "    ClientSecretCredential,\n",
    ")\n",
    "from azure.ai.ml.entities import AmlCompute\n",
    "import time\n",
    "\n",
    "try:\n",
    "    credential = DefaultAzureCredential()\n",
    "    credential.get_token(\"https://management.azure.com/.default\")\n",
    "except Exception as ex:\n",
    "    credential = InteractiveBrowserCredential()\n",
    "\n",
    "# connect to a workspace\n",
    "workspace_ml_client = None\n",
    "try:\n",
    "    workspace_ml_client = MLClient.from_config(credential)\n",
    "    subscription_id = workspace_ml_client.subscription_id\n",
    "    workspace = workspace_ml_client.workspace_name\n",
    "    resource_group = workspace_ml_client.resource_group_name\n",
    "except Exception as ex:\n",
    "    print(ex)\n",
    "    # Enter details of your workspace\n",
    "    subscription_id = \"<SUBSCRIPTION_ID>\"\n",
    "    resource_group = \"<RESOURCE_GROUP>\"\n",
    "    workspace = \"<WORKSPACE_NAME>\"\n",
    "    workspace_ml_client = MLClient(\n",
    "        credential, subscription_id, resource_group, workspace\n",
    "    )\n",
    "# Connect to the HuggingFaceHub registry\n",
    "registry_ml_client = MLClient(credential, registry_name=\"HuggingFace\")\n",
    "print(registry_ml_client)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pick a model to deploy\n",
    "\n",
    "Open the Model Catalog in AzureML Studio and choose the Hugging Face Hub collection. Filter by the `question-answering` task and search any specific models you are interested in. In this example, we use the `deepset-roberta-base-squad2\"` model. If you plan to deploy a different model, replace the model name and version accordingly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = \"deepset-roberta-base-squad2\"\n",
    "foundation_model = registry_ml_client.models.get(model_name, version=\"17\")\n",
    "print(\n",
    "    \"\\n\\nUsing model name: {0}, version: {1}, id: {2} for inferencing\".format(\n",
    "        foundation_model.name, foundation_model.version, foundation_model.id\n",
    "    )\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy the model to an online endpoint\n",
    "Online endpoints give a durable REST API that can be used to integrate with applications that need to use the model. Create an online endpoint and then create an online deployment. You need to specify the Virtual Machine instance or SKU when creating the deployment. You can find the optimal CPU or GPU SKU for a model by opening the quick deployment dialog from the model page in the AzureML Model Catalog. Specify the SKU in the `instance_type` input in deployment settings below.\n",
    "\n",
    "Typically Online Endpoints require you to provide scoring script and a docker container image (through an AzureML environment), in addition to the model. You don't need to worry about them for HuggingFace Hub models available in AzureML Model Catalog because we have enabled 'no code deployments' for these models by packaging scoring script and container image along with the model.\n",
    "\n",
    "Learn more about Online Endpoints: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-online-endpoints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time, sys\n",
    "from azure.ai.ml.entities import (\n",
    "    ManagedOnlineEndpoint,\n",
    "    ManagedOnlineDeployment,\n",
    "    OnlineRequestSettings,\n",
    ")\n",
    "\n",
    "# Create online endpoint - endpoint names need to be unique in a region, hence using timestamp to create unique endpoint name\n",
    "timestamp = int(time.time())\n",
    "online_endpoint_name = \"question-answering-\" + str(timestamp)\n",
    "# create an online endpoint\n",
    "endpoint = ManagedOnlineEndpoint(\n",
    "    name=online_endpoint_name,\n",
    "    description=\"Online endpoint for \"\n",
    "    + foundation_model.name\n",
    "    + \", for question-answering task\",\n",
    "    auth_mode=\"key\",\n",
    ")\n",
    "workspace_ml_client.begin_create_or_update(endpoint).wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a deployment\n",
    "demo_deployment = ManagedOnlineDeployment(\n",
    "    name=\"demo\",\n",
    "    endpoint_name=online_endpoint_name,\n",
    "    model=foundation_model.id,\n",
    "    instance_type=\"Standard_DS3_v2\",\n",
    "    instance_count=1,\n",
    ")\n",
    "workspace_ml_client.online_deployments.begin_create_or_update(demo_deployment).wait()\n",
    "# online endpoints can have multiple deployments with traffic split or shadow traffic. Set traffic to 100% for demo deployment\n",
    "endpoint.traffic = {\"demo\": 100}\n",
    "workspace_ml_client.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Try sample inference\n",
    "\n",
    "Online endpoints expose a REST API that can be integrated into your applications. Learn how to fetch the scoring REST API and credentials for online endpoints here: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-authenticate-online-endpoint\n",
    "\n",
    "In this example, we will use the Python SDK helper method to invoke the endpoint. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the model object from HuggingFaceHub. We can use it to check for sample test data\n",
    "import urllib.request, json\n",
    "\n",
    "raw_data = urllib.request.urlopen(\n",
    "    \"https://huggingface.co/api/models/\" + foundation_model.tags[\"modelId\"]\n",
    ")\n",
    "\n",
    "print(\"https://huggingface.co/api/models/\" + foundation_model.tags[\"modelId\"])\n",
    "data = json.load(raw_data)\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data[\"widgetData\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check if there is sample inference data available on HuggingFaceHub for the model, else try with the backup sample data\n",
    "scoring_file = \"./sample_score.json\"\n",
    "inputs = {}\n",
    "input_question = []\n",
    "input_context = []\n",
    "if \"widgetData\" in data:\n",
    "    for input in data[\"widgetData\"]:\n",
    "        input_question.append(input[\"text\"])\n",
    "        input_context.append(input[\"context\"])\n",
    "    inputs[\"question\"] = input_question\n",
    "    inputs[\"context\"] = input_context\n",
    "    # write the sample_score.json file\n",
    "    score_dict = {\"inputs\": inputs}\n",
    "    with open(scoring_file, \"w\") as outfile:\n",
    "        json.dump(score_dict, outfile, indent=2)\n",
    "else:\n",
    "    scoring_file = \"./sample_score_backup.json\"\n",
    "\n",
    "# print the sample scoring file\n",
    "print(\"\\n\\nSample scoring file: \")\n",
    "with open(scoring_file) as json_file:\n",
    "    scoring_data = json.load(json_file)\n",
    "    print(scoring_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# score the sample_score.json file using the online endpoint with the azureml endpoint invoke method\n",
    "\n",
    "response = workspace_ml_client.online_endpoints.invoke(\n",
    "    endpoint_name=online_endpoint_name,\n",
    "    deployment_name=\"demo\",\n",
    "    request_file=scoring_file,\n",
    ")\n",
    "response_json = json.loads(response)\n",
    "print(json.dumps(response_json, indent=2))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make the output prettier and print it in below format:\n",
    "\n",
    "\"Context: \" \n",
    "\n",
    "\"Question: \" \n",
    "\n",
    "\"Answer: \""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(input_question)):\n",
    "    print(\"Context: \" + input_context[i])\n",
    "    print(\"Question: \" + input_question[i])\n",
    "    print(\"Answer: \" + response_json[i][\"answer\"])\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the online endpoint\n",
    "Don't forget to delete the online endpoint, else you will leave the billing meter running for the compute used by the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workspace_ml_client.online_endpoints.begin_delete(name=online_endpoint_name).wait()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK V2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}