update structured output

tamingllms/_build/html/_sources/notebooks/structured_output.ipynb

@@ -290,12 +290,12 @@
     "\n",
     "Provider-specific APIs can offer ways to handle those challenges. We will explore two approaches here using OpenAI's API:\n",
     "\n",
-    "* **JSON Mode**: Some LLM APIs offer features specifically designed for generating JSON output.\n",
+    "* **JSON Mode**: Most LLM APIs today offer features specifically designed for generating JSON output.\n",
     "* **Structured Outputs**: Some LLM APIs offer features specifically designed for generating structured outputs with type safety.\n",
     "\n",
     "#### JSON Mode\n",
     "\n",
-    "JSON mode is a feature provided by some LLM APIs, such as OpenAI's, that allows the model to generate output in JSON format. This is particularly useful when you need structured data as a result, such as when parsing the output programmatically or integrating it with other systems that require JSON input. As depicted in {numref}`json-mode`, JSON mode is implemented by instructing theLLM model to use JSON as response format and optionally defining a target schema.\n",
+    "JSON mode is a feature provided by most LLM API providers, such as OpenAI, that allows the model to generate output in JSON format. This is particularly useful when you need structured data as a result, such as when parsing the output programmatically or integrating it with other systems that require JSON input. As depicted in {numref}`json-mode`, JSON mode is implemented by instructing theLLM model to use JSON as response format and optionally defining a target schema.\n",
     "\n",
     "\n",
     "```{figure} ../_static/structured_output/json.svg\n",
@@ -610,7 +610,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Outlines supports the OpenAI API. However, one of its key advantages is the ability to ensure structured output for Open Source models, which often lack such guarantees by default."
+    "Outlines can support major proprietary LLM APIs (e.g. OpenAI's via vLLM). However, one of its key advantages is the ability to ensure structured output for Open Source models, which often lack such guarantees by default."
    ]
   },
   {
@@ -674,6 +674,13 @@
     "print(answer)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In this simple example, we use Outlines' `choice` method to constrain the model output to a predefined set of options (\"Positive\" or \"Negative\"). This ensures the model can only return one of these values, avoiding any unexpected or malformed responses.\n"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -735,11 +742,9 @@
     "\n",
     "* **Simplicity vs. Control**: One-shot prompts are simple but offer limited control.  `LangChain`, and Outlines provide greater control but might have a steeper learning curve though quite manageable.\n",
     "\n",
-    "* **Native LLM Support**:  `with_structured_output` in LangChain relies on the underlying LLM having built-in support for structured output APIs, i.e. LangChain is just a wrapper around the LLM's structured output API. Other methods, like parsing or using Outlines, are more broadly applicable with the fomer enabling a wider range of Open Source models.\n",
-    "\n",
-    "* **Flexibility**:  Outlines and LangChain's  `StructuredOutputParser`  offer the most flexibility for defining custom output structures.\n",
+    "* **Native LLM Support**:  `with_structured_output` in LangChain relies on the underlying LLM having built-in support for structured output APIs, i.e. LangChain is a wrapper around the underlying LLM's structured output API. Outlines, on the other hand, is more broadly applicable enabling a wider range of Open Source models.\n",
     "\n",
-    "* **Performance**: Outlines' may offer performance benefits as it introduces no overhead during inference a feature not necessiraly shared by other methods."
+    "* **Flexibility**:  Outlines and LangChain's  `StructuredOutputParser`  offer the most flexibility for defining custom output structures."
    ]
   },
   {
@@ -763,19 +768,19 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Ongoing Debate on Structured Output from LLMs\n",
+    "### Ongoing Debate on LLMs Structured Output\n",
     "\n",
     "The use of structured output, like JSON or XML, for Large Language Models (LLMs) is a developing area. While structured output offers clear benefits in parsing, robustness, and integration, there is growing debate on whether it also potentially comes at the cost of reasoning abilities. \n",
     "\n",
-    "Recent research {cite}`tam2024letspeakfreelystudy` suggests that imposing format restrictions on LLMs might impact their performance, particularly in reasoning-intensive tasks. Further evidence {cite}`aider2024codejson` suggests LLMs may produce lower quality code if they’re asked to return it as part of a structured JSON response:\n",
+    "Recent research \"Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models\" {cite}`tam2024letspeakfreelystudy` suggests that imposing format restrictions on LLMs might impact their performance, particularly in reasoning-intensive tasks. Further evidence {cite}`aider2024codejson` suggests LLMs may produce lower quality code if they’re asked to return it as part of a structured JSON response, in particular:\n",
     "\n",
     "* **Potential performance degradation:**  Enforcing structured output, especially through constrained decoding methods like JSON-mode, can negatively impact an LLM's reasoning abilities. This is particularly evident in tasks that require multi-step reasoning or complex thought processes.\n",
     "\n",
     "* **Overly restrictive schemas:** Imposing strict schemas can limit the expressiveness of LLM outputs and may hinder their ability to generate creative or nuanced responses.  In certain cases, the strictness of the schema might outweigh the benefits of structured output.\n",
     "\n",
     "* **Increased complexity in prompt engineering:** Crafting prompts that effectively guide LLMs to generate structured outputs while maintaining performance can be challenging. It often requires careful consideration of the schema, the task instructions, and the desired level of detail in the response.\n",
     "\n",
-    "On the other hand, those findings are not without criticism. The .txt team challenges the findings of the paper \"Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models\" {cite}`tam2024letspeakfreelystudy`. The paper claims that structured output formats, like JSON, negatively affect the performance of LLMs, particularly when it comes to reasoning. The rebuttal argues that **structured generation, when done correctly, actually *improves* performance**.\n",
+    "On the other hand, those findings are not without criticism. The .txt team challenges the work of {cite}`tam2024letspeakfreelystudy`. The rebuttal argues that **structured generation, when done correctly, actually *improves* performance**.\n",
     "\n",
     "\n",
     "```{figure} ../_static/structured_output/rebuttal.png\n",
@@ -787,6 +792,7 @@
     "---\n",
     "Structured vs Unstructured Results by .txt team.\n",
     "```\n",
+    "\n",
     "The .txt team presents compelling evidence through their reproduction of the paper's experiments. While their unstructured results align with the original paper's findings, their structured results paint a dramatically different picture - demonstrating that structured generation actually improves performance (see {numref}`structured_vs_unstructured`). The team has made their experimental notebooks publicly available on GitHub for independent verification {cite}`dottxt2024demos`.\n",
     "\n",
     "\n",
@@ -797,11 +803,11 @@
     "*   The prompts used for structured generation, particularly in JSON-mode, didn't provide the LLM with sufficient information to properly complete the task. \n",
     "*   The paper conflates \"structured generation\" with \"JSON-mode\", when they are not the same thing. \n",
     "\n",
-    "It is important to note that, as with any rebuttal, .txt presents a specific perspective on the issue of structured output from LLMs. While their findings suggest potential benefits to structured generation, further research and exploration are needed to comprehensively understand the nuances and trade-offs involved in using structured output for various LLM tasks and applications.\n",
+    "It is important to note that while .txt provides a compelling and verifiable argument in favor of (proper) structured output generation in LLMs further research and exploration are needed to comprehensively understand the nuances and trade-offs involved in using structured output for various LLM tasks and applications.\n",
     "\n",
     "In summary, the debate surrounding structured output highlights the ongoing challenges in balancing LLM capabilities with real-world application requirements. While structured outputs offer clear benefits in parsing, robustness, and integration, their potential impact on performance, particularly in reasoning tasks is a topic of ongoing debate. \n",
     "\n",
-    "The ideal approach likely involves a nuanced strategy that considers the specific task, the desired level of structure, and the available LLM capabilities. Further research and development efforts are needed to mitigate the potential drawbacks of structured output and unlock the full potential of LLMs for a wider range of applications. \n"
+    "The ideal approach likely involves a nuanced strategy that considers the specific task, the desired level of structure, and the available LLM capabilities. Further research and development efforts are needed to mitigate potential drawbacks and unlock the full potential of LLMs for a wider range of applications. \n"
    ]
   },
   {
@@ -810,7 +816,11 @@
    "source": [
     "## Conclusion\n",
     "\n",
-    "Extracting structured output from LLMs is crucial for integrating them into real-world applications. By understanding the challenges and employing appropriate strategies and tools, developers can improve the reliability and usability of LLM-powered systems, unlocking their potential to automate complex tasks and generate valuable insights. \n"
+    "Extracting structured output from LLMs is crucial for integrating them into real-world applications. By understanding the challenges and employing appropriate strategies and tools, developers can improve the reliability and usability of LLM-powered systems, unlocking their potential to automate complex tasks and generate valuable insights. \n",
+    "\n",
+    "## Acknowledgements\n",
+    "\n",
+    "We would like to thank Cameron Pfiffer from the .txt team for his insightful review and feedback.\n"
    ]
   },
   {