-
Notifications
You must be signed in to change notification settings - Fork 15.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parsing a dictionary of lists problem #27453
Comments
Just a quick glance -- but this does not appear to be a bug in langchain, but an issue with the chat model failing to produce the correct output I'd suggest adding reference examples into the prompt to help the model output the correct thing |
For some reason, it seems that every time the LLM is prompted with generating some type of dictionary, it is not included in the response. Consider this simple code and some variations: from typing import Dict, List
from pydantic import BaseModel
from langchain_openai import ChatOpenAI
model = ChatOpenAI(model="gpt-4o-mini-2024-07-18", temperature=0.0)
class TestModel(BaseModel):
# variations here
structured_llm = model.with_structured_output(TestModel)
response = structured_llm.invoke('prompt')
print(response) When we define output: int Here is the output (I am also outputting the openai tool that gets bound to the model):
Even if we define output: List We get this:
But as soon as output: Dict Then all of the sudden the model does not respond with anything!
If there is a output: Dict
output_2: int Gives:
Why are these fields getting ignored? Is this an issue with the model or what? |
Actually I found a format that works more suitable for me. Anyway I can turn from on format to Dict with one line of Python code. @ethanglide does it work with examples and not just 'prompt' string ? I don't have too much experience providing examples for the case. |
Unfortunately I am not able to get it to work with examples either, assuming I did those correctly. from typing import Dict, List
from pydantic import BaseModel
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate, FewShotChatMessagePromptTemplate
model = ChatOpenAI(model="gpt-4o-mini-2024-07-18", temperature=0.0)
class TestModel(BaseModel):
output: Dict[str, str]
structured_llm = model.with_structured_output(TestModel)
examples = [
{
"input": "What is the capital of France?",
"output": '{"output": "Paris"}'
},
{
"input": "What is the capital of Germany?",
"output": '{"output": "Berlin"}'
},
{
"input": "What is the capital of Italy?",
"output": '{"output": "Rome"}'
}
]
example_prompt = ChatPromptTemplate.from_messages(
[
("human", "{input}"),
("ai", "{output}"),
]
)
few_shot_prompt = FewShotChatMessagePromptTemplate(
example_prompt=example_prompt,
examples=examples,
)
final_prompt = ChatPromptTemplate.from_messages(
[
("system", "You are a geography expert."),
few_shot_prompt,
("human", "{input}"),
]
)
chain = final_prompt | structured_llm
response = chain.invoke({"input": "What is the capital of Lithuania?"})
print(response) This gives:
Which stays consistent with the issues I had above. Of course, there are ways around this, and we should be asking ourselves whether or not having the model respond with arbitrarily structured dictionaries with arbitrary amounts of keys is something that should be done. But it really is strange that it works with |
The examples should be ai messages with tool calls not just content since you're using the tool calling API. Check the how to guides for tool calling (apologies on 📱 right now) Should look like System, human, ai, tool, human, ai, tool Or else squeeze the examples into the system prompt |
Thank you for the guidance, I haven't quite been able to make it work but I'm sure it is possible. Program: from typing import Dict
from pydantic import BaseModel
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.messages import HumanMessage, AIMessage, ToolMessage
from langchain_core.runnables import RunnablePassthrough
model = ChatOpenAI(model="gpt-4o-mini-2024-07-18", temperature=0.0)
class TestModel(BaseModel):
output: Dict
structured_llm = model.with_structured_output(TestModel)
examples = [
HumanMessage("What is the capital of France?"),
AIMessage(
'',
name='geography_assistant',
tool_calls=[
{
'name': 'TestModel',
'args': {"capital": "Paris"},
'id': '1'
},
],
),
ToolMessage({"output": {"capital": "Paris"}}, tool_call_id='1'),
HumanMessage("What is the capital of Germany?"),
AIMessage(
'',
name='geography_assistant',
tool_calls=[
{
'name': 'TestModel',
'args': {"capital": "Berlin"},
'id': '2'
},
],
),
ToolMessage({"output": {"capital": "Berlin"}}, tool_call_id='2'),
HumanMessage("What is the capital of Italy?"),
AIMessage(
'',
name='geography_assistant',
tool_calls=[
{
'name': 'TestModel',
'args': {"capital": "Rome"},
'id': '3'
},
],
),
ToolMessage({"output": {"capital": "Rome"}}, tool_call_id='3'),
]
final_prompt = ChatPromptTemplate.from_messages(
[
("system", "You are a geography expert."),
*examples,
("human", "{input}"),
]
)
chain = {'input': RunnablePassthrough()} | final_prompt | structured_llm
response = chain.invoke('What is the capital of Lithuania?')
print(response) Output:
As you can see |
Checked other resources
Example Code
I want to get "a" as a key in ppp , but code (using Dict) below fails:
Error Message and Stack Trace (if applicable)
ValidationError: 1 validation error for C
ppp
Field required [type=missing, input_value={'ccc': [{'a_1': 'Price',...tant to Battery Life'}]}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
Description
I have a code that works:
import os
from pydantic import BaseModel, Field
from langchain_openai import ChatOpenAI
model = ChatOpenAI(model="gpt-4o-mini-2024-07-18", temperature=0.0)
class A(BaseModel):
a_1: str
a_2: str
r: str
class B(BaseModel):
a: str
b_1: str
b_2: str
r: str
class C(BaseModel):
ccc:List[A]
ppp: List[B]
structured_llm = model.with_structured_output(C)
response = structured_llm.invoke(prompt)
System Info
System Information
Package Information
Optional packages not installed
Other Dependencies
The text was updated successfully, but these errors were encountered: