Better default chat formatting

[antonpibm]

Hi everyone,

I want to discuss how message trajectories are formatted in inputs to an LLM. The root of what I'll discuss doesn't originate from LangGraph itself, the code is in LangChain, but it becomes apparent when working with LangGraph.

When a list of messages is added in a graph and is formatted in the invocation to an LLM it's formatted with the _convert_input method of the model, there the list is formatted using the ChatPromptValue class.
What happens is that a prefix of "System:/AI:"/"Human:" is added before each message based on the message type.
Example trajectory:
[SystemMessage(content="You are a helpful bot"), HumanMessage(content="Tell me a joke")]
would result in the following:

System: You are a helpful bot
Human: Tell me a joke

So far so good, but there are 2 problems:

1. The "name" field from the message is not used, this field is common in LangGraph and I belive having a "name:" prefix as input to the LLM is better, this is a minor issue, the more relevant one is
2. The generation "AI:" prefix is not added to the string sent to the LLM
   So for a graph with multiple steps, e.g. when we add an offensive output filtering after the generation, the output of the LLM looks something like this for the following trajectory:
   [SystemMessage(content="You are a helpful bot"), HumanMessage(content="Tell me a joke"), AIMessage(content="Why does 6 afraid of 7? because 7 8 9")]
   Output would be:
   AI: There is no offensive language in the text

This cause the next trajectory string to be:
Trajectory:
[SystemMessage(content="You are a helpful bot"), HumanMessage(content="Tell me a joke"), AIMessage(content="Why does 6 afraid of 7? because 7 8 9"), AIMessage(content="AI: There is no offensive language in the text")]
String output:

System: You are a helpful bot
Human: Tell me a joke
AI: Why does 6 afraid of 7? because 7 8 9
AI: AI: There is no offensive language in the text

This causes a lot of problems with generation.

Instead, the role can be supplied in the invocation, which could in tern add it as a prefix upon invocation, i.e. for
[SystemMessage(content="You are a helpful bot"), HumanMessage(content="Tell me a joke"), AIMessage(content="Why does 6 afraid of 7? because 7 8 9")]
The invocation would receive this string as input:

System: You are a helpful bot
Human: Tell me a joke
AI: Why does 6 afraid of 7? because 7 8 9
AI:

[hinthornw]

What LLM are you using?

Most people either use a) a chat model (where the model provider decides what to do with the parameters) or b) their own formatting (since they want to control the strings)

[antonpibm]

@hinthornw thank you for your response. I'm using models from the WatsonX (IBM) family, the non-chat class produces the output I've shown above. A follow up questions -

1. Are queries in chat model sent as lists? I wasn't able to see the final query neither in the monitoring tool nor with LangChain debug mode turned on, only see the list of messages
2. Is there a standard function for the formatting or developers are writing their own custom formatting functions?
3. The formatting in LangGraph for the non-chat model still suffer from the issue I've described, aren't people raising this as an issue? adding the generation prefix is quite straight forward and I believe is the right way to query, wouldn't you agree?