Workshop/1_LoadData.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load and Index Data - Vector Store\n",
    "\n",
    "We will use [Azure Cognitive Search](https://learn.microsoft.com/en-us/azure/search/search-what-is-azure-search) to load and index the data.  Azure Cognitive Search is a cloud search service with built-in AI capabilities that enrich all types of information to easily identify and explore relevant content at scale. It uses the same integrated Microsoft natural language stack that Bing and Office have used for more than a decade, and AI services across vision, language, and speech, to deliver knowledge from structured and unstructured data.\n",
    "\n",
    "Cognitive search enabled the vector search feature! When done correctly, vector search is a proven technique for significantly increasing the semantic relevance of search results.  It is a technique that uses machine learning to embed text into a vector space, where the distance between vectors is a measure of semantic similarity.  This allows for the use of vector similarity search to find relevant results.  [Sign up]\n",
    "(https://aka.ms/VectorSearchSignUp) for Private Preview of Vector Search.\n",
    "\n",
    "Cognitive Search can index and store vectors, but it doesn't generate them out of the box. The documents that you push to your search service must contain vectors within the payload. Alternatively, you can use the Indexer to pull vectors from your data sources such as Blob Storage JSON files or CSVs. You can also use a Custom Skill to generate embeddings as part of the AI Enrichment process.\n",
    "\n",
    "\n",
    "[Sample repo](https://github.com/Azure/cognitive-search-vector-pr) to get started with vector search. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pre-requisite:\n",
    "- To run the code, install the following packages from local Wheel file. Alternatively, install azure-search-documents==11.4.0a20230509004 from the Dev Feed. For instructions on how to connect to the dev feed, please visit Azure-Python-SDK Azure Search Documents [Dev Feed](https://dev.azure.com/azure-sdk/public/_artifacts/feed/azure-sdk-for-python/connect/pip).\n",
    "- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/).\n",
    "- An Azure Cognitive Search service (any tier, any region). [Create a service](https://learn.microsoft.com/en-us/azure/search/search-create-service-portal) or find an [existing service](https://portal.azure.com/#blade/HubsExtension/BrowseResourceBlade/resourceType/Microsoft.Search%2FsearchServices) under your current subscription."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install ./azure_search_documents-11.4.0b4-py3-none-any.whl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install langchain\n",
    "%pip install langchain"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set the Environment Variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os  \n",
    "import json  \n",
    "import openai\n",
    "from Utilities.envVars import *\n",
    "\n",
    "# Set Search Service endpoint, index name, and API key from environment variables\n",
    "indexName = SearchIndex\n",
    "\n",
    "# Set OpenAI API key and endpoint\n",
    "openai.api_type = \"azure\"\n",
    "openai.api_version = OpenAiVersion\n",
    "openai_api_key = OpenAiKey\n",
    "assert openai_api_key, \"ERROR: Azure OpenAI Key is missing\"\n",
    "openai.api_key = openai_api_key\n",
    "openAiEndPoint = f\"https://{OpenAiService}.openai.azure.com\"\n",
    "assert openAiEndPoint, \"ERROR: Azure OpenAI Endpoint is missing\"\n",
    "assert \"openai.azure.com\" in openAiEndPoint.lower(), \"ERROR: Azure OpenAI Endpoint should be in the form: \\n\\n\\t<your unique endpoint identifier>.openai.azure.com\"\n",
    "openai.api_base = openAiEndPoint\n",
    "davincimodel = OpenAiDavinci\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import Required Library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import required libraries\n",
    "from langchain.llms.openai import AzureOpenAI, OpenAI\n",
    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "from langchain.document_loaders import (\n",
    "    PDFMinerLoader,\n",
    "    UnstructuredFileLoader,\n",
    ")\n",
    "from Utilities.cogSearch import createSearchIndex, indexSections"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load the PDF, create the chunk and push to Azure Cognitive Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Flexibility to change the call to OpenAI or Azure OpenAI\n",
    "embeddingModelType = \"openai\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the file name and the namespace for the index\n",
    "fileName = \"Fabric Get Started.pdf\"\n",
    "fabricGetStartedPath = \"Data/PDF/\" + fileName\n",
    "# Load the PDF with Document Loader available from Langchain\n",
    "loader = PDFMinerLoader(fabricGetStartedPath)\n",
    "rawDocs = loader.load()\n",
    "# Set the source \n",
    "for doc in rawDocs:\n",
    "    doc.metadata['source'] = fabricGetStartedPath\n",
    "\n",
    "textSplitter = RecursiveCharacterTextSplitter(chunk_size=1500, chunk_overlap=0)\n",
    "docs = textSplitter.split_documents(rawDocs)\n",
    "# Call Helper function to create Index and Index the sections\n",
    "createSearchIndex(SearchService, SearchKey, indexName)\n",
    "indexSections(OpenAiService, OpenAiKey, OpenAiVersion, OpenAiApiKey, SearchService, SearchKey, embeddingModelType, OpenAiEmbedding, fileName, indexName, docs)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Perform Vector Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Id: Fabric_Get_Started_pdf-3\n",
      "Content: Fabric allows creators to concentrate on producing their best work, freeing them from\n",
      "\n",
      "the need to integrate, manage, or understand the underlying infrastructure that\n",
      "\n",
      "supports the experience.\n",
      "\n",
      "Components of Microsoft Fabric\n",
      "\n",
      "Microsoft Fabric offers the comprehensive set of analytics experiences designed to work\n",
      "\n",
      "together seamlessly. Each experience is tailored to a specific persona and a specific task.\n",
      "\n",
      "Fabric includes industry-leading experiences in the following categories for an end-to-\n",
      "\n",
      "end analytical need.\n",
      "\n",
      "Data Engineering - Data Engineering experience provides a world class Spark\n",
      "\n",
      "platform with great authoring experiences, enabling data engineers to perform\n",
      "\n",
      "large scale data transformation and democratize data through the lakehouse.\n",
      "\n",
      "Microsoft Fabric Spark's integration with Data Factory enables notebooks and\n",
      "\n",
      "spark jobs to be scheduled and orchestrated. For more information, see What is\n",
      "\n",
      "Data engineering in Microsoft Fabric?\n",
      "\n",
      "\fData Factory - Azure Data Factory combines the simplicity of Power Query with the\n",
      "\n",
      "scale and power of Azure Data Factory. You can use more than 200 native\n",
      "\n",
      "connectors to connect to data sources on-premises and in the cloud. For more\n",
      "\n",
      "information, see What is Data Factory in Microsoft Fabric?\n",
      "\n",
      "Data Science - Data Science experience enables you to build, deploy, and\n",
      "\n",
      "operationalize machine learning models seamlessly within your Fabric experience.\n",
      "\n",
      "It integrates with Azure Machine Learning to provide built-in experiment tracking\n",
      "Source File: Fabric Get Started.pdf\n",
      "\n",
      "Id: Fabric_Get_Started_pdf-2\n",
      "Content: Microsoft Fabric is currently in PREVIEW. This information relates to a prerelease\n",
      "\n",
      "product that may be substantially modified before it's released. Microsoft makes no\n",
      "\n",
      "warranties, expressed or implied, with respect to the information provided here.\n",
      "\n",
      "SaaS foundation\n",
      "\n",
      "Microsoft Fabric brings together new and existing components from Power BI, Azure\n",
      "\n",
      "Synapse, and Azure Data Explorer into a single integrated environment. These\n",
      "\n",
      "components are then presented in various customized user experiences.\n",
      "\n",
      "Fabric brings together experiences such as Data Engineering, Data Factory, Data Science,\n",
      "\n",
      "Data Warehouse, Real-Time Analytics, and Power BI onto a shared SaaS foundation. This\n",
      "\n",
      "integration provides the following advantages:\n",
      "\n",
      "\fAn extensive range of deeply integrated analytics in the industry.\n",
      "\n",
      "Shared experiences across experiences that are familiar and easy to learn.\n",
      "\n",
      "Developers can easily access and reuse all assets.\n",
      "\n",
      "A unified data lake that allows you to retain the data where it is while using your\n",
      "\n",
      "preferred analytics tools.\n",
      "\n",
      "Centralized administration and governance across all experiences.\n",
      "\n",
      "With the Microsoft Fabric SaaS experience, all the data and the services are seamlessly\n",
      "\n",
      "integrated. IT teams can centrally configure core enterprise capabilities and permissions\n",
      "\n",
      "are automatically applied across all the underlying services. Additionally, data sensitivity\n",
      "\n",
      "labels are inherited automatically across the items in the suite.\n",
      "Source File: Fabric Get Started.pdf\n",
      "\n",
      "Id: Fabric_Get_Started_pdf-1\n",
      "Content: Tell us about your PDF experience.\n",
      "\n",
      "Microsoft Fabric get started\n",
      "documentation\n",
      "\n",
      "Microsoft Fabric is a unified platform that can meet your organization's data and\n",
      "analytics needs. Discover the Fabric shared and platform documentation from this page.\n",
      "\n",
      "About Microsoft Fabric\n",
      "\n",
      "ｅ OVERVIEW\n",
      "\n",
      "What is Fabric?\n",
      "\n",
      "Fabric terminology\n",
      "\n",
      "ｂ GET STARTED\n",
      "\n",
      "Start a Fabric trial\n",
      "\n",
      "Fabric home navigation\n",
      "\n",
      "End-to-end tutorials\n",
      "\n",
      "Context sensitive Help pane\n",
      "\n",
      "Get started with Fabric items\n",
      "\n",
      "ｐ CONCEPT\n",
      "\n",
      "Find items in OneLake data hub\n",
      "\n",
      "Promote and certify items\n",
      "\n",
      "ｃ HOW-TO GUIDE\n",
      "\n",
      "Apply sensitivity labels\n",
      "\n",
      "Workspaces\n",
      "\n",
      "ｐ CONCEPT\n",
      "\n",
      "Fabric workspace\n",
      "\n",
      "\fWorkspace roles\n",
      "\n",
      "ｂ GET STARTED\n",
      "\n",
      "Create a workspace\n",
      "\n",
      "ｃ HOW-TO GUIDE\n",
      "\n",
      "Workspace access control\n",
      "\n",
      "\fWhat is Microsoft Fabric?\n",
      "\n",
      "Article • 05/23/2023\n",
      "\n",
      "Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything\n",
      "\n",
      "from data movement to data science, Real-Time Analytics, and business intelligence. It\n",
      "\n",
      "offers a comprehensive suite of services, including data lake, data engineering, and data\n",
      "\n",
      "integration, all in one place.\n",
      "\n",
      "With Fabric, you don't need to piece together different services from multiple vendors.\n",
      "\n",
      "Instead, you can enjoy a highly integrated, end-to-end, and easy-to-use product that is\n",
      "\n",
      "designed to simplify your analytics needs.\n",
      "\n",
      "The platform is built on a foundation of Software as a Service (SaaS), which takes\n",
      "\n",
      "simplicity and integration to a whole new level.\n",
      "\n",
      "） Important\n",
      "Source File: Fabric Get Started.pdf\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from Utilities.cogSearch import performCogSearch\n",
    "\n",
    "# Pure Vector Search\n",
    "query = \"What is Microsoft Fabric\"  \n",
    "\n",
    "results = performCogSearch(OpenAiService, OpenAiKey, OpenAiVersion, OpenAiApiKey, SearchService, SearchKey, embeddingModelType, OpenAiEmbedding, query, indexName, 3)\n",
    "\n",
    "for result in results:  \n",
    "    print(f\"Id: {result['id']}\")  \n",
    "    print(f\"Content: {result['content']}\")  \n",
    "    print(f\"Source File: {result['sourcefile']}\\n\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Id: Fabric_Get_Started_pdf-1\n",
      "Content: Tell us about your PDF experience.\n",
      "\n",
      "Microsoft Fabric get started\n",
      "documentation\n",
      "\n",
      "Microsoft Fabric is a unified platform that can meet your organization's data and\n",
      "analytics needs. Discover the Fabric shared and platform documentation from this page.\n",
      "\n",
      "About Microsoft Fabric\n",
      "\n",
      "ｅ OVERVIEW\n",
      "\n",
      "What is Fabric?\n",
      "\n",
      "Fabric terminology\n",
      "\n",
      "ｂ GET STARTED\n",
      "\n",
      "Start a Fabric trial\n",
      "\n",
      "Fabric home navigation\n",
      "\n",
      "End-to-end tutorials\n",
      "\n",
      "Context sensitive Help pane\n",
      "\n",
      "Get started with Fabric items\n",
      "\n",
      "ｐ CONCEPT\n",
      "\n",
      "Find items in OneLake data hub\n",
      "\n",
      "Promote and certify items\n",
      "\n",
      "ｃ HOW-TO GUIDE\n",
      "\n",
      "Apply sensitivity labels\n",
      "\n",
      "Workspaces\n",
      "\n",
      "ｐ CONCEPT\n",
      "\n",
      "Fabric workspace\n",
      "\n",
      "\fWorkspace roles\n",
      "\n",
      "ｂ GET STARTED\n",
      "\n",
      "Create a workspace\n",
      "\n",
      "ｃ HOW-TO GUIDE\n",
      "\n",
      "Workspace access control\n",
      "\n",
      "\fWhat is Microsoft Fabric?\n",
      "\n",
      "Article • 05/23/2023\n",
      "\n",
      "Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything\n",
      "\n",
      "from data movement to data science, Real-Time Analytics, and business intelligence. It\n",
      "\n",
      "offers a comprehensive suite of services, including data lake, data engineering, and data\n",
      "\n",
      "integration, all in one place.\n",
      "\n",
      "With Fabric, you don't need to piece together different services from multiple vendors.\n",
      "\n",
      "Instead, you can enjoy a highly integrated, end-to-end, and easy-to-use product that is\n",
      "\n",
      "designed to simplify your analytics needs.\n",
      "\n",
      "The platform is built on a foundation of Software as a Service (SaaS), which takes\n",
      "\n",
      "simplicity and integration to a whole new level.\n",
      "\n",
      "） Important\n",
      "Source File: Fabric Get Started.pdf\n",
      "\n",
      "Id: Fabric_Get_Started_pdf-3\n",
      "Content: Fabric allows creators to concentrate on producing their best work, freeing them from\n",
      "\n",
      "the need to integrate, manage, or understand the underlying infrastructure that\n",
      "\n",
      "supports the experience.\n",
      "\n",
      "Components of Microsoft Fabric\n",
      "\n",
      "Microsoft Fabric offers the comprehensive set of analytics experiences designed to work\n",
      "\n",
      "together seamlessly. Each experience is tailored to a specific persona and a specific task.\n",
      "\n",
      "Fabric includes industry-leading experiences in the following categories for an end-to-\n",
      "\n",
      "end analytical need.\n",
      "\n",
      "Data Engineering - Data Engineering experience provides a world class Spark\n",
      "\n",
      "platform with great authoring experiences, enabling data engineers to perform\n",
      "\n",
      "large scale data transformation and democratize data through the lakehouse.\n",
      "\n",
      "Microsoft Fabric Spark's integration with Data Factory enables notebooks and\n",
      "\n",
      "spark jobs to be scheduled and orchestrated. For more information, see What is\n",
      "\n",
      "Data engineering in Microsoft Fabric?\n",
      "\n",
      "\fData Factory - Azure Data Factory combines the simplicity of Power Query with the\n",
      "\n",
      "scale and power of Azure Data Factory. You can use more than 200 native\n",
      "\n",
      "connectors to connect to data sources on-premises and in the cloud. For more\n",
      "\n",
      "information, see What is Data Factory in Microsoft Fabric?\n",
      "\n",
      "Data Science - Data Science experience enables you to build, deploy, and\n",
      "\n",
      "operationalize machine learning models seamlessly within your Fabric experience.\n",
      "\n",
      "It integrates with Azure Machine Learning to provide built-in experiment tracking\n",
      "Source File: Fabric Get Started.pdf\n",
      "\n",
      "Id: Fabric_Get_Started_pdf-2\n",
      "Content: Microsoft Fabric is currently in PREVIEW. This information relates to a prerelease\n",
      "\n",
      "product that may be substantially modified before it's released. Microsoft makes no\n",
      "\n",
      "warranties, expressed or implied, with respect to the information provided here.\n",
      "\n",
      "SaaS foundation\n",
      "\n",
      "Microsoft Fabric brings together new and existing components from Power BI, Azure\n",
      "\n",
      "Synapse, and Azure Data Explorer into a single integrated environment. These\n",
      "\n",
      "components are then presented in various customized user experiences.\n",
      "\n",
      "Fabric brings together experiences such as Data Engineering, Data Factory, Data Science,\n",
      "\n",
      "Data Warehouse, Real-Time Analytics, and Power BI onto a shared SaaS foundation. This\n",
      "\n",
      "integration provides the following advantages:\n",
      "\n",
      "\fAn extensive range of deeply integrated analytics in the industry.\n",
      "\n",
      "Shared experiences across experiences that are familiar and easy to learn.\n",
      "\n",
      "Developers can easily access and reuse all assets.\n",
      "\n",
      "A unified data lake that allows you to retain the data where it is while using your\n",
      "\n",
      "preferred analytics tools.\n",
      "\n",
      "Centralized administration and governance across all experiences.\n",
      "\n",
      "With the Microsoft Fabric SaaS experience, all the data and the services are seamlessly\n",
      "\n",
      "integrated. IT teams can centrally configure core enterprise capabilities and permissions\n",
      "\n",
      "are automatically applied across all the underlying services. Additionally, data sensitivity\n",
      "\n",
      "labels are inherited automatically across the items in the suite.\n",
      "Source File: Fabric Get Started.pdf\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Vector Search with Multi-language support\n",
    "query = \"¿Qué es Microsoft Fabric?\"\n",
    "\n",
    "results = performCogSearch(OpenAiService, OpenAiKey, OpenAiVersion, OpenAiApiKey, SearchService, SearchKey, embeddingModelType, OpenAiEmbedding, query, indexName, 3)\n",
    "  \n",
    "for result in results:  \n",
    "    print(f\"Id: {result['id']}\")  \n",
    "    print(f\"Content: {result['content']}\")  \n",
    "    print(f\"Source File: {result['sourcefile']}\\n\") "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}