sdk/python/endpoints/online/llm/semantic-kernel/1_semantic_http_server.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Semantic Kernel (SK) Integration into Azure Machine Learning (AzureML)\n",
    "\n",
    "**Requirements** - In order to benefit from this tutorial, you will need:\n",
    "* A basic understanding of Machine Learning and Large Language Models\n",
    "* An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n",
    "* An Azure Machine Learning Workspace and Azure Container Registry\n",
    "* An OpenAI API Key which can be found in User Settings in OpenAI\n",
    "\n",
    "**Motivations** - Semantic kernel has a slightly different approach to LLM agents. It offers an interesting Plan->Execute pattern, where it could use LLM to form a plan first, then human could confirm and execute on the plan. In this notebook, we use the [planner](https://github.com/microsoft/semantic-kernel/blob/main/samples/notebooks/python/05-using-the-planner.ipynb) example from Semantic Kernel as a base. But additionally, we've made the following modifications:\n",
    "* Created a **python SemanticKernelHttp server** based on Flask.\n",
    "* Deploy SemanticKernelHttp to an **AzureML Managed Online Endpoint**\n",
    "\n",
    "Managed online endpoints provide an easy to manage inferencing server for your ML workload. It's perfect for LLM based applications. Since we need a REST service, we won't use the default endpoint docker image, we will create a custom docker image instead.\n",
    "\n",
    "**Outline** - \n",
    "1. Prepare Dependencies\n",
    "2. Deploy to Managed Online Endpoint\n",
    "3. Test"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Connect to Azure Machine Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "OPENAI_API_TYPE = \"openai\"  # 'azure' or 'openai'\n",
    "OPENAI_API_KEY = \"<OPENAI-API-KEY>\"\n",
    "\n",
    "# required for OpenAI API\n",
    "OPENAI_ORG_ID = \"\"\n",
    "OPENAI_MODEL_ID = \"gpt-3.5-turbo\"\n",
    "\n",
    "# required for Azure OpenAI API\n",
    "AZURE_OPENAI_API_ENDPOINT = \"https://<AZURE_OPENAI_ENDPOINT>.openai.azure.com/\"\n",
    "AZURE_OPENAI_API_DEPLOYMENT_NAME = \"<DEPLOYMENT_NAME>\"\n",
    "\n",
    "# set to true for chat completion API, false for text completion\n",
    "IS_CHAT_COMPLETION = True\n",
    "\n",
    "# setting up env variables for local server\n",
    "%env OPENAI_API_TYPE=$OPENAI_API_TYPE\n",
    "%env OPENAI_API_KEY=$OPENAI_API_KEY\n",
    "%env OPENAI_MODEL_ID=$OPENAI_MODEL_ID\n",
    "%env OPENAI_ORG_ID=$OPENAI_ORG_ID\n",
    "%env AZURE_OPENAI_API_ENDPOINT=$AZURE_OPENAI_API_ENDPOINT\n",
    "%env AZURE_OPENAI_API_DEPLOYMENT_NAME=$AZURE_OPENAI_API_DEPLOYMENT_NAME\n",
    "%env IS_CHAT_COMPLETION=$IS_CHAT_COMPLETION\n",
    "\n",
    "# Install python dependencies\n",
    "%pip install -r ../src/sk/requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Set workspace details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# enter details of your AML workspace\n",
    "SUBSCRIPTION_ID = \"<SUBSCRIPTION_ID>\"\n",
    "RESOURCE_GROUP = \"<RESOURCE_GROUP>\"\n",
    "AML_WORKSPACE_NAME = \"<AML_WORKSPACE_NAME>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Login to your Azure account"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.identity import (\n",
    "    DefaultAzureCredential,\n",
    "    InteractiveBrowserCredential,\n",
    "    AzureCliCredential,\n",
    ")\n",
    "\n",
    "try:\n",
    "    credential = DefaultAzureCredential(additionally_allowed_tenants=[\"*\"])\n",
    "except Exception as ex:\n",
    "    # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work\n",
    "    credential = InteractiveBrowserCredential(additionally_allowed_tenants=[\"*\"])\n",
    "\n",
    "# If login doesn't work above, uncomment the code below and login using device code\n",
    "# !az login --use-device-code"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Create Container Registry and Docker Image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from azure.ai.ml import (\n",
    "    MLClient,\n",
    ")\n",
    "\n",
    "ml_client = MLClient(credential, SUBSCRIPTION_ID, RESOURCE_GROUP, AML_WORKSPACE_NAME)\n",
    "ws = ml_client.workspaces.get(AML_WORKSPACE_NAME)\n",
    "\n",
    "# Get the Azure Container Registry associated with the workspace\n",
    "acr = ws.container_registry\n",
    "\n",
    "# Parse the ACR resource Id for the ACR name\n",
    "match_object = re.match(r\".+?registries\\/(.+)\", acr)\n",
    "ACR_NAME = match_object.group(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the image in your ACR image\n",
    "ACR_IMAGE_NAME = \"serving\"\n",
    "\n",
    "!az acr build --image {ACR_IMAGE_NAME} --registry {ACR_NAME} ./environment/serving/. --resource-group {RESOURCE_GROUP}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Managed Online Endpoint\n",
    "### 2.1 Create Endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a endpoint\n",
    "import datetime\n",
    "\n",
    "from azure.ai.ml.entities import (\n",
    "    ManagedOnlineEndpoint,\n",
    ")\n",
    "\n",
    "from azure.ai.ml import (\n",
    "    MLClient,\n",
    ")\n",
    "\n",
    "time = str(datetime.datetime.now().strftime(\"%m%d%H%M%f\"))\n",
    "online_endpoint_name = f\"aml-llm-sk-demo-{time}\"\n",
    "\n",
    "ml_client = MLClient(credential, SUBSCRIPTION_ID, RESOURCE_GROUP, AML_WORKSPACE_NAME)\n",
    "\n",
    "# create an online endpoint\n",
    "endpoint = ManagedOnlineEndpoint(\n",
    "    name=online_endpoint_name,\n",
    "    description=\"online endpoint for SemanticKernelHttp server\",\n",
    "    auth_mode=\"key\",\n",
    ")\n",
    "\n",
    "endpoint = ml_client.begin_create_or_update(endpoint).result()\n",
    "\n",
    "print(endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Store the API Key in KeyVault\n",
    "\n",
    "The below code is modelled after the example notebook [online-endpoints-keyvault.ipynb](../../managed/online-endpoints-keyvault.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Import Keyvault Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.mgmt.keyvault import KeyVaultManagementClient\n",
    "from azure.keyvault.secrets import SecretClient\n",
    "from azure.mgmt.keyvault.models import (\n",
    "    VaultCreateOrUpdateParameters,\n",
    "    VaultProperties,\n",
    "    Sku,\n",
    ")\n",
    "from azure.mgmt.keyvault.models import AccessPolicyEntry, Permissions, SecretPermissions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Create a Keyvault Management client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "KEYVAULT_NAME = f\"llmdemokv{time}\"\n",
    "KV_OPENAI_KEY = \"OPENAI-API-KEY\"\n",
    "\n",
    "keyvault_mgmt_client = KeyVaultManagementClient(\n",
    "    credential=credential, subscription_id=SUBSCRIPTION_ID\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Get your Object Id\n",
    "The `oid` in your JWT access token represents the Object ID of the current user or Service Principal logged into the Azure CLI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json, base64\n",
    "\n",
    "cli_credential = AzureCliCredential()\n",
    "token = cli_credential.get_token(\"https://management.azure.com\").token\n",
    "user_or_sp_object_id = json.loads(base64.b64decode(token.split(\".\")[1] + \"===\")).get(\n",
    "    \"oid\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 Define an AccessPolicy for the Endpoint and the current user\n",
    "\n",
    "Allow the endpoint to get secrets in the keyvault and allow all secret permissions for the current user or Service Principal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_access_policy = AccessPolicyEntry(\n",
    "    tenant_id=endpoint.identity.tenant_id,\n",
    "    object_id=endpoint.identity.principal_id,\n",
    "    permissions=Permissions(secrets=[SecretPermissions.GET]),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_or_sp_access_policy = AccessPolicyEntry(\n",
    "    tenant_id=endpoint.identity.tenant_id,\n",
    "    object_id=user_or_sp_object_id,\n",
    "    permissions=Permissions(secrets=[SecretPermissions.ALL]),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5 Create the Keyvault"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "keyvault = keyvault_mgmt_client.vaults.begin_create_or_update(\n",
    "    vault_name=KEYVAULT_NAME,\n",
    "    resource_group_name=RESOURCE_GROUP,\n",
    "    parameters=VaultCreateOrUpdateParameters(\n",
    "        location=endpoint.location,\n",
    "        properties=VaultProperties(\n",
    "            tenant_id=endpoint.identity.tenant_id,\n",
    "            sku=Sku(name=\"Standard\", family=\"A\"),\n",
    "            access_policies=[endpoint_access_policy, user_or_sp_access_policy],\n",
    "        ),\n",
    "    ),\n",
    ").result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6 Add your OPENAI_API_KEY to the Keyvault"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "KEYVAULT_URL = f\"https://{KEYVAULT_NAME}.vault.azure.net\"\n",
    "\n",
    "secret_client = SecretClient(credential=credential, vault_url=KEYVAULT_URL)\n",
    "secret = secret_client.set_secret(name=KV_OPENAI_KEY, value=OPENAI_API_KEY)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Deploy to the Endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml.entities import (\n",
    "    ManagedOnlineDeployment,\n",
    "    OnlineRequestSettings,\n",
    "    Model,\n",
    "    Environment,\n",
    ")\n",
    "\n",
    "deployment_name = f\"deploy-{time}\"\n",
    "sk_deployment = ManagedOnlineDeployment(\n",
    "    name=deployment_name,\n",
    "    model=Model(path=\"../src\"),\n",
    "    request_settings=OnlineRequestSettings(request_timeout_ms=60000),\n",
    "    environment=Environment(\n",
    "        image=f\"{ACR_NAME}.azurecr.io/{ACR_IMAGE_NAME}:latest\",\n",
    "        name=\"serving\",\n",
    "        description=\"A generic serving environment, allowing customer to provide their own entry point to bring up an http server\",\n",
    "        inference_config={\n",
    "            \"liveness_route\": {\"port\": 5001, \"path\": \"/health\"},\n",
    "            \"readiness_route\": {\"port\": 5001, \"path\": \"/health\"},\n",
    "            \"scoring_route\": {\"port\": 5001, \"path\": \"/\"},\n",
    "        },\n",
    "    ),\n",
    "    environment_variables={\n",
    "        \"AZUREML_SERVING_ENTRYPOINT\": \"src/sk/entry.sh\",\n",
    "        \"OPENAI_API_KEY\": f\"keyvaultref:{KEYVAULT_URL}/secrets/{KV_OPENAI_KEY}\",\n",
    "        \"OPENAI_API_TYPE\": OPENAI_API_TYPE,\n",
    "        \"OPENAI_MODEL_ID\": OPENAI_MODEL_ID,\n",
    "        \"OPENAI_ORG_ID\": OPENAI_ORG_ID,\n",
    "        \"AZURE_OPENAI_API_ENDPOINT\": AZURE_OPENAI_API_ENDPOINT,\n",
    "        \"AZURE_OPENAI_API_DEPLOYMENT_NAME\": AZURE_OPENAI_API_DEPLOYMENT_NAME,\n",
    "        \"IS_CHAT_COMPLETION\": True,\n",
    "    },\n",
    "    endpoint_name=online_endpoint_name,\n",
    "    instance_type=\"Standard_F2s_v2\",\n",
    "    instance_count=1,\n",
    ")\n",
    "ml_client.online_deployments.begin_create_or_update(sk_deployment).result()\n",
    "\n",
    "endpoint.traffic = {deployment_name: 100}\n",
    "ml_client.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Test\n",
    "Now endpoint has been deployed, let's test it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests, json\n",
    "from urllib.parse import urlsplit\n",
    "\n",
    "url_parts = urlsplit(endpoint.scoring_uri)\n",
    "url = url_parts.scheme + \"://\" + url_parts.netloc\n",
    "\n",
    "token = ml_client.online_endpoints.get_keys(name=online_endpoint_name).primary_key\n",
    "headers = {\"Authorization\": \"Bearer \" + token, \"Content-Type\": \"application/json\"}\n",
    "payload = json.dumps(\n",
    "    {\n",
    "        \"value\": \"Tomorrow is Valentine's day. I need to come up with a few date ideas. She speaks French so write it in French.\"\n",
    "    }\n",
    ")\n",
    "\n",
    "response = requests.post(f\"{url}/planner/createplan\", headers=headers, data=payload)\n",
    "print(f\"Created Plan:\\n\", response.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "payload = response.text\n",
    "response = requests.request(\n",
    "    \"POST\", f\"{url}/planner/executeplan\", headers=headers, data=payload\n",
    ")\n",
    "print(f\"Execution Result:\\n\", response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Clean up resources\n",
    "\n",
    "### 6.1 Delete the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client.online_endpoints.begin_delete(name=online_endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2 Delete the ACR Image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!az acr repository delete --name {ACR_NAME} --image {ACR_IMAGE_NAME}:latest --yes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.3 Delete the KeyVault"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    }
   ],
   "source": [
    "keyvault_mgmt_client.vaults.delete(RESOURCE_GROUP, KEYVAULT_NAME)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}