# setting up the client in Python
import os
@@ -470,7 +488,7 @@ Authentication
Requesting Completions
Most interaction with GPT and other models consist in generating completions for certain tasks (TODO: Link to completions)
To request completions from the OpenAI API, we use Python to send HTTP requests to the designated API endpoint. These requests are structured to include various parameters that guide the generation of text completions. The most fundamental parameter is the prompt text, which sets the context for the completion. Additionally, you can specify the desired model configuration, such as the engine to use (e.g., “gpt-4”), as well as any constraints or preferences for the generated completions, such as the maximum number of tokens or the temperature for controlling creativity (TODO: Link parameterization)
-
+
# creating a completion
chat_completion = client.chat.completions.create(
messages=[
diff --git a/docs/llm/intro.html b/docs/llm/intro.html
index b6a3775..e0c0482 100644
--- a/docs/llm/intro.html
+++ b/docs/llm/intro.html
@@ -285,6 +285,12 @@
+
+
+
diff --git a/docs/llm/parameterization.html b/docs/llm/parameterization.html
index fe59998..73e7807 100644
--- a/docs/llm/parameterization.html
+++ b/docs/llm/parameterization.html
@@ -319,6 +319,12 @@
+
+
+
@@ -450,39 +456,191 @@ Parameterization of GPT
+The GPT models provided by OpenAI provide a variety of parameters that can change the way the language model responds. Below you can find a list of the most important ones.
-Temperature: Temperature is a parameter that controls the randomness of the generated text. Lower temperatures result in more deterministic outputs, where the model tends to choose the most likely tokens at each step. Higher temperatures introduce more randomness, allowing the model to explore less likely tokens and produce more creative outputs. It’s often used to balance between generating safe, conservative responses and more novel, imaginative ones.
Max Tokens: Max Tokens limits the maximum length of the generated text by specifying the maximum number of tokens (words or subwords) allowed in the output. This parameter helps to control the length of the response and prevent the model from generating overly long or verbose outputs, which may not be suitable for certain applications or contexts.
Top P (Nucleus Sampling): Top P, also known as nucleus sampling, dynamically selects a subset of the most likely tokens based on their cumulative probability until the cumulative probability exceeds a certain threshold (specified by the parameter). This approach ensures diversity in the generated text while still prioritizing tokens with higher probabilities. It’s particularly useful for generating diverse and contextually relevant responses.
Frequency Penalty: Frequency Penalty penalizes tokens based on their frequency in the generated text. Tokens that appear more frequently are assigned higher penalties, discouraging the model from repeatedly generating common or redundant tokens. This helps to promote diversity in the generated text and prevent the model from producing overly repetitive outputs.
Presence Penalty: Presence Penalty penalizes tokens that are already present in the input prompt. By discouraging the model from simply echoing or replicating the input text, this parameter encourages the generation of responses that go beyond the provided context. It’s useful for generating more creative and novel outputs that are not directly predictable from the input.
Stop Sequence: Stop Sequence specifies a sequence of tokens that, if generated by the model, signals it to stop generating further text. This parameter is commonly used to indicate the desired ending or conclusion of the generated text. It helps to control the length of the response and ensure that the model generates text that aligns with specific requirements or constraints.
Repetition Penalty: Repetition Penalty penalizes repeated tokens in the generated text by assigning higher penalties to tokens that appear multiple times within a short context window. This encourages the model to produce more varied outputs by avoiding unnecessary repetition of tokens. It’s particularly useful for generating coherent and diverse text without excessive redundancy.
Length Penalty: Length Penalty penalizes the length of the generated text by applying a penalty factor to longer sequences. This helps to balance between generating concise and informative responses while avoiding excessively long or verbose outputs. Length Penalty is often used to control the length of the generated text and ensure that it remains coherent and contextually relevant.
Temperature: Temperature (temperaure) is a parameter that controls the randomness of the generated text. Lower temperatures result in more deterministic outputs, where the model tends to choose the most likely tokens at each step. Higher temperatures introduce more randomness, allowing the model to explore less likely tokens and produce more creative outputs. It’s often used to balance between generating safe, conservative responses and more novel, imaginative ones.
Max Tokens: Max Tokens (max_tokens) limits the maximum length of the generated text by specifying the maximum number of tokens (words or subwords) allowed in the output. This parameter helps to control the length of the response and prevent the model from generating overly long or verbose outputs, which may not be suitable for certain applications or contexts.
Top P (Nucleus Sampling): Top P (top_p), also known as nucleus sampling, dynamically selects a subset of the most likely tokens based on their cumulative probability until the cumulative probability exceeds a certain threshold (specified by the parameter). This approach ensures diversity in the generated text while still prioritizing tokens with higher probabilities. It’s particularly useful for generating diverse and contextually relevant responses.
Frequency Penalty: Frequency Penalty (frequency_penalty) penalizes tokens based on their frequency in the generated text. Tokens that appear more frequently are assigned higher penalties, discouraging the model from repeatedly generating common or redundant tokens. This helps to promote diversity in the generated text and prevent the model from producing overly repetitive outputs.
Presence Penalty: Presence Penalty (presence_penalty) penalizes tokens that are already present in the input prompt. By discouraging the model from simply echoing or replicating the input text, this parameter encourages the generation of responses that go beyond the provided context. It’s useful for generating more creative and novel outputs that are not directly predictable from the input.
Stop Sequence: Stop Sequence (stop) specifies a sequence of tokens that, if generated by the model, signals it to stop generating further text. This parameter is commonly used to indicate the desired ending or conclusion of the generated text. It helps to control the length of the response and ensure that the model generates text that aligns with specific requirements or constraints.
Roles:
-
-
-Code
-from openai import OpenAI
-client = OpenAI()
+In order to cover most tasks you want to perform using a chat format, the OpenAI API let’s you define different roles in the chat. The available roles are system, assistant, user and tools. You should already be familiar with two of them by now: The user role corresponds to the actual user prompting the language model, all answers are given with the assisstant role.
+The system role can now be given to provide some additional general instructions to the language model that are typically not a user input, for example, the style in which the model responds. In this case, an example is better than any explanation.
+
+import os
+from llm_utils.client import get_openai_client
-completion = client.chat.completions.create(
- model="gpt-3.5-turbo",
- messages=[
- {"role": "system", "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."},
- {"role": "user", "content": "Compose a poem that explains the concept of recursion in programming."}
- ]
-)
-
-print(completion.choices[0].message)
-
+MODEL = "gpt4"
+
+client = get_openai_client(
+ model=MODEL,
+ config_path=os.environ.get("CONFIG_PATH")
+)
+
+completion = client.chat.completions.create(
+ model="MODEL",
+ messages=[
+ {"role": "system", "content": "You are an annoyed technician working in a help center for dish washers, who answers in short, unfriendly bursts."},
+ {"role": "user", "content": "My dish washer does not clean the dishes, what could be the reason."}
+ ]
+)
+
+print(completion.choices[0].message.content)
+
+Could be anything. Blocked spray arm. Clogged filter. Faulty pump. Detergent issue. Check all that.
+
-
-Function calling:
-https://platform.openai.com/docs/guides/function-calling
+
+Function calling:
+As we have seen, most interactions with a language model happen in form of a chat with almost “free” question or instructions and answers. While this seems the most natural in most cases, it is not always a practical format if we want to use a language model for very specific purposes. This happens particularly often when we want to employ a language model in business situations, where we require a consistent output of the model.
+As an example, let us try to use GPT for sentiment analysis (see also here). Let’s say we want GPT to classify a text into one of the following four categories:
+
+sentiment_categories = [
+ "positive",
+ "negative",
+ "neutral",
+ "mixed"
+]
+
+We could do the following:
+
+messages = []
+messages.append(
+ {"role": "system", "content": f"Classify the given text into one of the following sentiment categories: {sentiment_categories}."}
+)
+messages.append(
+ {"role": "user", "content": "I really did not like the movie."}
+)
+
+response = client.chat.completions.create(
+ messages=messages,
+ model=MODEL
+)
+
+print(f"Response: '{response.choices[0].message.content}'")
+
+
+
+Response: 'Category: Negative'
+
+
+It is easy to spot the problem: GPT does not necessarily answer in the way we expect or want it to. In this case, instead of simply returning the correct category, it also returns the string Category: alongside it (and capitalized Negative). So if we were to use the answer in a program or data base, we’d now again have to use some NLP techniques to parse it in order eventually retrieve exactly the category we were looking for: negative. What we need instead is a way to constrain GPT to a specific way of answering, and this is where functions or tools come into play (see also Function calling and Function calling (cookbook)).
+This concept allows us to specify the exact output format we expect to receive from GPT (it is called functions since ideally we want to call a function directly on the output of GPT so it has to be in a specific format).
+
+# this looks intimidating but isn't that complicated
+tools = [
+ {
+ "type": "function",
+ "function": {
+ "name": "analyze_sentiment",
+ "description": "Analyze the sentiment in a given text.",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "sentiment": {
+ "type": "string",
+ "enum": sentiment_categories,
+ "description": f"The sentiment of the text."
+ }
+ },
+ "required": ["sentiment"],
+ }
+ }
+ }
+]
+
+
+messages = []
+messages.append(
+ {"role": "system", "content": f"Classify the given text into one of the following sentiment categories: {sentiment_categories}."}
+)
+messages.append(
+ {"role": "user", "content": "I really did not like the movie."}
+)
+
+response = client.chat.completions.create(
+ messages=messages,
+ model=MODEL,
+ tools=tools,
+ tool_choice={
+ "type": "function",
+ "function": {"name": "analyze_sentiment"}}
+)
+
+print(f"Response: '{response.choices[0].message.tool_calls[0].function.arguments}'")
+
+Response: '{
+"sentiment": "negative"
+}'
+
+
+We can now easily extract what we need:
+
+import json
+result = json.loads(response.choices[0].message.tool_calls[0].function.arguments) # remember that the answer is a string
+print(result["sentiment"])
+
+negative
+
+
+We can also include multiple function parameters if our desired output has multiple components. Let’s try to include another parameter which includes the reason for the sentiment.
+
+tools = [
+ {
+ "type": "function",
+ "function": {
+ "name": "analyze_sentiment",
+ "description": "Analyze the sentiment in a given text.",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "sentiment": {
+ "type": "string",
+ "enum": sentiment_categories,
+ "description": f"The sentiment of the text."
+ },
+ "reason": {
+ "type": "string",
+ "description": "The reason for the sentiment in few words. If there is no information, do not make assumptions and leave blank."
+ }
+ },
+ "required": ["sentiment", "reason"],
+ }
+ }
+ }
+]
+
+
+messages = []
+messages.append(
+ {"role": "system", "content": f"Classify the given text into one of the following sentiment categories: {sentiment_categories}. If you can, also extract the reason."}
+)
+messages.append(
+ {"role": "user", "content": "I loved the movie, Johnny Depp is a great actor."}
+)
+
+response = client.chat.completions.create(
+ messages=messages,
+ model=MODEL,
+ tools=tools,
+ tool_choice={
+ "type": "function",
+ "function": {"name": "analyze_sentiment"}}
+)
+
+print(f"Response: '{response.choices[0].message.tool_calls[0].function.arguments}'")
+
+Response: '{
+"sentiment": "positive",
+"reason": "Appreciation for the movie and actor"
+}'
+
+
+Here, again, we could also constrain the possibilities for the reason to a certain set. Hence, functions are great to have more consistent answers of the language model such that we can use it in applications.
diff --git a/docs/llm/prompting.html b/docs/llm/prompting.html
index a9d2d5f..7859158 100644
--- a/docs/llm/prompting.html
+++ b/docs/llm/prompting.html
@@ -285,6 +285,12 @@
+
+
+
@@ -416,9 +422,82 @@ Prompting
-Resources: - https://platform.openai.com/docs/guides/prompt-engineering -
+Learning prompting is a science for itself. The difficulty lies in the probabilistic nature of the language models. That means, small changes to your prompt (that you might even find insignificant) can have a large impact on the result/the answer. In particular, the changes do not have to be “logical”, i.e., depend on your changes in a comprehensible or reproducible way. This can sometimes be frustrating, but can also be avoided in many cases when following the right instructions for prompting. To do so, let’s best follow the creators.
+
+
+
+
+Note
+
+
+
+The following is taken from the OpenAI Guide
+
+
+
+Write clear instructions
+These models can’t read your mind. If outputs are too long, ask for brief replies. If outputs are too simple, ask for expert-level writing. If you dislike the format, demonstrate the format you’d like to see. The less the model has to guess at what you want, the more likely you’ll get it.
+Tactics:
+
+- Include details in your query to get more relevant answers
+- Ask the model to adopt a persona
+- Use delimiters to clearly indicate distinct parts of the input
+- Specify the steps required to complete a task
+- Provide examples
+- Specify the desired length of the output
+
+
+
+Provide reference text
+Language models can confidently invent fake answers, especially when asked about esoteric topics or for citations and URLs. In the same way that a sheet of notes can help a student do better on a test, providing reference text to these models can help in answering with fewer fabrications.
+Tactics:
+
+- Instruct the model to answer using a reference text
+- Instruct the model to answer with citations from a reference text
+
+
+
+Split complex tasks into simpler subtasks
+Just as it is good practice in software engineering to decompose a complex system into a set of modular components, the same is true of tasks - submitted to a language model. Complex tasks tend to have higher error rates than simpler tasks. Furthermore, complex tasks can often be re-defined as a workflow of simpler tasks in which the outputs of earlier tasks are used to construct the inputs to later tasks.
+Tactics:
+
+- Use intent classification to identify the most relevant instructions for a user query
+- For dialogue applications that require very long conversations, summarize or filter previous dialogue
+- Summarize long documents piecewise and construct a full summary recursively
+
+
+
+Give the model time to “think”
+If asked to multiply 17 by 28, you might not know it instantly, but can still work it out with time. Similarly, models make more reasoning errors when trying to answer right away, rather than taking time to work out an answer. Asking for a “chain of thought” before an answer can help the model reason its way toward correct answers more reliably.
+Tactics:
+
+- Instruct the model to work out its own solution before rushing to a conclusion
+- Use inner monologue or a sequence of queries to hide the model’s reasoning process
+- Ask the model if it missed anything on previous passes
+
+
+
+Use external tools
+Compensate for the weaknesses of the model by feeding it the outputs of other tools. For example, a text retrieval system (sometimes called RAG or retrieval augmented generation) can tell the model about relevant documents. A code execution engine like OpenAI’s Code Interpreter can help the model do math and run code. If a task can be done more reliably or efficiently by a tool rather than by a language model, offload it to get the best of both.
+Tactics:
+
+- Use embeddings-based search to implement efficient knowledge retrieval
+- Use code execution to perform more accurate calculations or call external APIs
+- Give the model access to specific functions
+
+
+
+Test changes systematically
+Improving performance is easier if you can measure it. In some cases a modification to a prompt will achieve better performance on a few isolated examples but lead to worse overall performance on a more representative set of examples. Therefore to be sure that a change is net positive to performance it may be necessary to define a comprehensive test suite (also known an as an “eval”).
+Tactic:
+
+- Evaluate model outputs with reference to gold-standard answers
+
+
Back to top