Late chunking

README.md

@@ -83,7 +83,8 @@ Develop a Retrieval-Augmented Generation (RAG) application using LanceDB for eff
 | [Instruct-Multitask](./examples/instruct-multitask) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/instruct-multitask/main.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a> [![Python](https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54)](./examples/instruct-multitask/main.py)  [![LLM](https://img.shields.io/badge/local-llm-green)](#)   [![beginner](https://img.shields.io/badge/beginner-B5FF33)](#)|[![Ghost](https://img.shields.io/badge/ghost-000?style=for-the-badge&logo=ghost&logoColor=%23F7DF1E)](https://blog.lancedb.com/multitask-embedding-with-lancedb-be18ec397543)|
 | [Improve RAG with HyDE](/examples/Advance-RAG-with-HyDE/) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Advance-RAG-with-HyDE/main.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>   [![LLM](https://img.shields.io/badge/openai-api-white)](#)  [![intermediate](https://img.shields.io/badge/intermediate-FFDA33)](#)|[![Ghost](https://img.shields.io/badge/ghost-000?style=for-the-badge&logo=ghost&logoColor=%23F7DF1E)](https://blog.lancedb.com/advanced-rag-precise-zero-shot-dense-retrieval-with-hyde-0946c54dfdcb)|
 | [Improve RAG with LOTR ](/examples/Advance_RAG_LOTR/) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Advance_RAG_LOTR/main.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>  [![LLM](https://img.shields.io/badge/openai-api-white)](#)  [![intermediate](https://img.shields.io/badge/intermediate-FFDA33)](#)|[![Ghost](https://img.shields.io/badge/ghost-000?style=for-the-badge&logo=ghost&logoColor=%23F7DF1E)](https://blog.lancedb.com/better-rag-with-lotr-lord-of-retriever-23c8336b9a35)|
-| [Advanced RAG: Context Enrichment Window](./examples/Advanced_RAG_Context_Enrichment_Window/) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Advanced_RAG_Context_Enrichment_Window/Advanced_RAG_Context_Enrichment_Window.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>  [![LLM](https://img.shields.io/badge/openai-api-white)](#)  [![intermediate](https://img.shields.io/badge/intermediate-FFDA33)](#)||
+| [Advanced RAG: Context Enrichment Window](./examples/Advanced_RAG_Context_Enrichment_Window/) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Advanced_RAG_Context_Enrichment_Window/Advanced_RAG_Context_Enrichment_Window.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>  [![LLM](https://img.shields.io/badge/openai-api-white)](#)  [![intermediate](https://img.shields.io/badge/intermediate-FFDA33)](#)|[![Ghost](https://img.shields.io/badge/ghost-000?style=for-the-badge&logo=ghost&logoColor=%23F7DF1E)](https://blog.lancedb.com/https://blog.lancedb.com/advanced-rag-context-enrichment-window/)|
+| [Advanced RAG: Late Chunking](./examples/Advanced_RAG_Late_Chunking/) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Advanced_RAG_Late_Chunking/Late_Chunking_(Chunked_Pooling).ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>  [![LLM](https://img.shields.io/badge/openai-api-white)](#)  [![intermediate](https://img.shields.io/badge/intermediate-FFDA33)](#)|[![Ghost](https://img.shields.io/badge/ghost-000?style=for-the-badge&logo=ghost&logoColor=%23F7DF1E)](https://blog.lancedb.com/https://blog.lancedb.com/advanced-rag-context-enrichment-window/)|
 | [Advanced RAG: Parent Document Retriever](/examples/parent_document_retriever/) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/parent_document_retriever/main.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>  [![LLM](https://img.shields.io/badge/openai-api-white)](#)  [![intermediate](https://img.shields.io/badge/intermediate-FFDA33)](#)|[![Ghost](https://img.shields.io/badge/ghost-000?style=for-the-badge&logo=ghost&logoColor=%23F7DF1E)](https://blog.lancedb.com/modified-rag-parent-document-bigger-chunk-retriever-62b3d1e79bc6)|
 | [Corrective RAG with Langgraph](./tutorials/Corrective-RAG-with_Langgraph/) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/Corrective-RAG-with_Langgraph/CRAG_with_Langgraph.ipynb) [![LLM](https://img.shields.io/badge/openai-api-white)](#) [![intermediate](https://img.shields.io/badge/intermediate-FFDA33)](#)| [![Ghost](https://img.shields.io/badge/ghost-000?style=for-the-badge&logo=ghost&logoColor=%23F7DF1E)](https://blog.lancedb.com/implementing-corrective-rag-in-the-easiest-way-2/)|
 | [Contextual-Compression-with-RAG](/examples/Contextual-Compression-with-RAG/) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/Contextual-Compression-with-RAG/main.ipynb) [![local LLM](https://img.shields.io/badge/local-llm-green)](#)   [![intermediate](https://img.shields.io/badge/intermediate-FFDA33)](#)|[![Ghost](https://img.shields.io/badge/ghost-000?style=for-the-badge&logo=ghost&logoColor=%23F7DF1E)](https://blog.lancedb.com/enhance-rag-integrate-contextual-compression-and-filtering-for-precision-a29d4a810301/) |
@@ -146,6 +147,7 @@ Design an AI agents coordination application with LanceDB for efficient vector-b
 | --------- | -------------------------- | ----------- |
 ||||
 | [AI email assistant with Composio](/examples/AI-Email-Assistant-with-Composio/) |<a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/AI-Email-Assistant-with-Composio/composio-lance.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>  [![LLM](https://img.shields.io/badge/openai-api-white)](#)   [![beginner](https://img.shields.io/badge/beginner-B5FF33)](#)|
+| [Assitant Bot with OpenAI Swarm](./examples/assistance-bot-with-swarm/) |[![Python](https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54)](./examples/assistance-bot-with-swarm/)  [![LLM](https://img.shields.io/badge/openai-api-white)](#)   [![intermediate](https://img.shields.io/badge/intermediate-FFDA33)](#)|
 | [AI Trends Searcher with CrewAI](./examples/AI-Trends-with-CrewAI/) |<a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/AI-Trends-with-CrewAI/CrewAI_AI_Trends.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>  [![LLM](https://img.shields.io/badge/openai-api-white)](#)    [![beginner](https://img.shields.io/badge/beginner-B5FF33)](#)|[![Ghost](https://img.shields.io/badge/ghost-000?style=for-the-badge&logo=ghost&logoColor=%23F7DF1E)](https://blog.lancedb.com/track-ai-trends-crewai-agents-rag/)|
 | [SuperAgent Autogen](/examples/SuperAgent_Autogen) |<a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/SuperAgent_Autogen/main.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a> [![LLM](https://img.shields.io/badge/openai-api-white)](#) [![intermediate](https://img.shields.io/badge/intermediate-FFDA33)](#)||
 | [AI Agents: Reducing Hallucination](/examples/reducing_hallucinations_ai_agents/) | <a href="https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/examples/reducing_hallucinations_ai_agents/main.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a> [![Python](https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54)](./examples/reducing_hallucinations_ai_agents/main.py) [![JS](https://img.shields.io/badge/javascript-%23323330.svg?style=for-the-badge&logo=javascript&logoColor=%23F7DF1E)](./examples/reducing_hallucinations_ai_agents/index.js) [![LLM](https://img.shields.io/badge/openai-api-white)](#) [![advanced](https://img.shields.io/badge/advanced-FF3333)](#) |[![Ghost](https://img.shields.io/badge/ghost-000?style=for-the-badge&logo=ghost&logoColor=%23F7DF1E)](https://blog.lancedb.com/how-to-reduce-hallucinations-from-llm-powered-agents-using-long-term-memory-72f262c3cc1f/)|

examples/assistance-bot-with-swarm/README.md

@@ -0,0 +1,43 @@
+# Assistant bot with OpenAI's Swarm
+
+This example shows a customer service bot that has two parts: one for interacting with users and another for providing help. It includes tools to assist in these tasks. The `run_demo_loop` function helps us create an interactive demo session.
+
+## Overview
+
+The support bot has two main parts:
+
+1. **User Interface Agent**: This part interacts with users at first and directs them to the help center based on what they need.
+2. **Help Center Agent**: This part offers detailed help and support using various tools and is connected to a LanceDB VectorDB to retrieve documents.
+
+## Setup
+
+To start the bot:
+
+set OpenAI key as env variable
+```
+export OPENAI_API_KEY="sk-yourapikey"
+```
+
+1. **Install requirements**
+
+```python3  
+    pip install -r requirements.txt
+    git+ssh://[email protected]/openai/swarm.git
+```
+
+2. **Prepare and Ingest dataset in LanceDB**
+
+We'll prepare dataset from OpenAI in JSON format and ingest it in LanceDB table.
+```python3
+    python3 dataset_prep.py
+```
+
+3. **Ready to RUN**
+
+Now you are ready to run assistant bot with Swarm
+```python3
+python3 main.py
+```
+
+*Note: You can change dataset and ingestion pipeline accordingly for your dataset to build agents around it.*
+

examples/assistance-bot-with-swarm/data/article_6233728.json

@@ -0,0 +1 @@
+{"text": "Introduction\n============\n\n\n\u200bSince releasing the Answers endpoint in beta last year, we\u2019ve developed new methods that achieve better results for this task. As a result, we\u2019ll be removing the Answers endpoint from our documentation and removing access to this endpoint on December 3, 2022 for all organizations. New accounts created after June 3rd will not have access to this endpoint.\n\n\n\nWe strongly encourage developers to switch over to newer techniques which produce better results, outlined below.\n\n\n\nCurrent documentation\n---------------------\n\n\n<https://beta.openai.com/docs/guides/answers> \n\n\n<https://beta.openai.com/docs/api-reference/answers>\n\n\n\nOptions\n=======\n\n\nAs a quick review, here are the high level steps of the current Answers endpoint:\n\n\n\n\n![](https://openai.intercom-attachments-7.com/i/o/524217540/51eda23e171f33f1b9d5acff/rM6ZVI3XZ2CpxcEStmG5mFy6ATBCskmX2g3_GPmeY3FicvrWfJCuFOtzsnbkpMQe-TQ6hi5j1BV9cFo7bCDcsz8VWxFfeOnC1Gb4QNaeVYtJq4Qtg76SBOLLk-jgHUA8mWZ0QgOuV636UgcvMA)All of these options are also outlined [here](https://github.com/openai/openai-cookbook/tree/main/transition_guides_for_deprecated_API_endpoints)\n\n\n\nOption 1: Transition to Embeddings-based search (recommended)\n-------------------------------------------------------------\n\n\nWe believe that most use cases will be better served by moving the underlying search system to use a vector-based embedding search. The major reason for this is that our current system used a bigram filter to narrow down the scope of candidates whereas our embeddings system has much more contextual awareness. Also, in general, using embeddings will be considerably lower cost in the long run. If you\u2019re not familiar with this, you can learn more by visiting our [guide to embeddings](https://beta.openai.com/docs/guides/embeddings/use-cases).\n\n\n\nIf you\u2019re using a small dataset (<10,000 documents), consider using the techniques described in that guide to find the best documents to construct a prompt similar to [this](#h_89196129b2). Then, you can just submit that prompt to our [Completions](https://beta.openai.com/docs/api-reference/completions) endpoint.\n\n\n\nIf you have a larger dataset, consider using a vector search engine like [Pinecone](https://share.streamlit.io/pinecone-io/playground/beyond_search_openai/src/server.py) or [Weaviate](https://weaviate.io/developers/weaviate/current/retriever-vectorizer-modules/text2vec-openai.html) to power that search.\n\n\n\nOption 2: Reimplement existing functionality\n--------------------------------------------\n\n\nIf you\u2019d like to recreate the functionality of the Answers endpoint, here\u2019s how we did it. There is also a [script](https://github.com/openai/openai-cookbook/blob/main/transition_guides_for_deprecated_API_endpoints/answers_functionality_example.py) that replicates most of this functionality.\n\n\n\nAt a high level, there are two main ways you can use the answers endpoint: you can source the data from an uploaded file or send it in with the request.\n\n\n\n**If you\u2019re using the document parameter**\n------------------------------------------\n\n\nThere\u2019s only one step if you provide the documents in the Answers API call.\n\n\n\nHere\u2019s roughly the steps we used: \n\n\n* Construct the prompt [with this format.](#h_89196129b2)\n* Gather all of the provided documents. If they fit in the prompt, just use all of them.\n* Do an [OpenAI search](https://beta.openai.com/docs/api-reference/searches) (note that this is also being deprecated and has a [transition guide](https://app.intercom.com/a/apps/dgkjq2bp/articles/articles/6272952/show)) where the documents are the user provided documents and the query is the query from above. Rank the documents by score.\n* In order of score, attempt to add Elastic search documents until you run out of space in the context.\n* Request a completion with the provided parameters (logit\\_bias, n, stop, etc)\n\n\nThroughout all of this, you\u2019ll need to check that the prompt\u2019s length doesn\u2019t exceed [the model's token limit](https://beta.openai.com/docs/engines/gpt-3). To assess the number of tokens present in a prompt, we recommend <https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2TokenizerFast>. \n\n\n\nIf you're using the file parameter\n----------------------------------\n\n\nStep 1: upload a jsonl file\n\n\n\nBehind the scenes, we upload new files meant for answers to an Elastic search cluster. Each line of the jsonl is then submitted as a document.\n\n\n\nIf you uploaded the file with the purpose \u201canswers,\u201d we additionally split the documents on newlines and upload each of those chunks as separate documents to ensure that we can search across and reference the highest number of relevant text sections in the file.\n\n\n\nEach line requires a \u201ctext\u201d field and an optional \u201cmetadata\u201d field.\n\n\n\nThese are the Elastic search settings and mappings for our index:\n\n\n\n[Elastic searching mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html): \n\n\n\n```\n{  \n    \"properties\": {  \n        \"document\": {\"type\": \"text\", \"analyzer\": \"standard_bigram_analyzer\"}, -> the \u201ctext\u201d field  \n        \"metadata\": {\"type\": \"object\", \"enabled\": False}, -> the \u201cmetadata\u201d field  \n    }  \n}\n```\n\n\n[Elastic search analyzer](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html):\n\n\n\n```\n{  \n    \"analysis\": {  \n        \"analyzer\": {  \n            \"standard_bigram_analyzer\": {  \n                \"type\": \"custom\",  \n                \"tokenizer\": \"standard\",  \n                \"filter\": [\"lowercase\", \"english_stop\", \"shingle\"],  \n            }  \n        },  \n        \"filter\": {\"english_stop\": {\"type\": \"stop\", \"stopwords\": \"_english_\"}},  \n    }  \n}\n```\n\n\nAfter that, we performed [standard Elastic search search calls](https://elasticsearch-py.readthedocs.io/en/v8.2.0/api.html#elasticsearch.Elasticsearch.search) and used `max\\_rerank` to determine the number of documents to return from Elastic search.\n\n\n\nStep 2: Search\n\n\nHere\u2019s roughly the steps we used. Our end goal is to create a [Completions](https://beta.openai.com/docs/api-reference/completions) request [with this format](#h_89196129b2). It will look very similar to [Documents](#h_cb1d8a8d3f)\n\n\n\nFrom there, our steps are: \n\n\n* Start with the `experimental\\_alternative\\_question` or, if that's not provided, what\u2019s in the `question` field. Call that the query.\n* Query Elastic search for `max\\_rerank` documents with query as the search param.\n* Take those documents and do an [OpenAI search](https://beta.openai.com/docs/api-reference/searches) on them where the entries from Elastic search are the docs, and the query is the query that you used above. Use the score from the search to rank the documents.\n* In order of score, attempt to add Elastic search documents until you run out of space in the prompt.\n* Request an OpenAI completion with the provided parameters (logit\\_bias, n, stop, etc). Return that answer to the user.\n\n\nCompletion Prompt\n-----------------\n\n\n\n```\n===  \nContext: #{{ provided examples_context }}  \n===  \nQ: example 1 question  \nA: example 1 answer  \n---  \nQ: example 2 question  \nA: example 2 answer  \n(and so on for all examples provided in the request)   \n===  \nContext: #{{ what we return from Elasticsearch }}  \n===  \nQ: #{{ user provided question }}   \nA:\n```\n", "title": "Answers Transition Guide", "article_id": "6233728", "url": "https://help.openai.com/en/articles/6233728-answers-transition-guide"}