ChatAnthropic doesn't stream tool_call tokens as tool_call_chunks but normal chunk content

[leSullivan]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

const llm = new ChatAnthropic({
        model: XXX,
        apiKey: XXX,
      });

const tool = someTool()

const chain = llm.bindTools([tool])

const stream = chatChain.streamEvents(XXX, {
      ...config,
      version: 'v2',
    });

for await (const { event, data, name, metadata } of stream) {
     if (
          event === 'on_chat_model_stream' &&
          metadata.langgraph_node === '...'
        ) {
          const { content } = data.chunk;
          if (!content) {
            continue;
          }

          response.content = content;
          aiResponse += content;

          chatStream.next({ data: response });
        }
}

Description

Im using langgraph for a agent orchestration.

As a node i have defined an agent that has some tools but can also respond without using tools.
When this agent decides to use a tool he still responds with normal chunk.content, which i also don't want but i haven't found a smart way disable that. Afaik the tool call is always streamed after the normal content so i don't know beforehand if the llm uses a tool or not in my stream_events loop. But now comes the super confusing part:

The function call chunks are also streamed via the normal chunk.content event. This means that the generated json is streamed to the frontend which is definitely not intended from our side. Shouldn't the fn call json be streamed via the tool_call_chunks ? This happens when im using the ChatAnthropic LLM. If someone can help me how to check in stream_events weather a tool_call was called or not i would also highly appreaciate it!

System Info

MacBook Pro M2
MacOS 15
"@langchain/anthropic": "^0.3.11",
"@langchain/community": "^0.3.22",
"@langchain/core": "^0.3.27",
"@langchain/langgraph": "^0.2.39",
"@langchain/openai": "^0.3.16",

[dosubot[bot]]

I found some relevant discussions and an issue related to your question:

1. How to filter out tool input from streamEvents when using ChatAnthropic? - This is an unsolved discussion that might provide some insights into handling tool inputs in stream events ^[1].

2. Feature Request: Streaming Anthropic Tool Usage - Another unsolved discussion that could be relevant to your situation ^[2].

3. ChatAnthropic streaming with tools not working and instead forces non-stream response - This issue was closed with a pull request that handled non-text LLM tokens, which might be related to your problem ^[3].

To address your specific issue, you can use the extractToolUseContent function to check whether a tool_call was made in the stream_events loop. This function processes AIMessageChunk objects and identifies if a tool_use type is present in the content. Here's a brief example of how you can implement this:

import { AIMessageChunk } from "@langchain/core/messages";
import { extractToolUseContent } from "path/to/anthropic";

// Assume `chunk` is an AIMessageChunk received in the stream_events loop
let concatenatedChunks: AIMessageChunk | undefined;

const { toolUseContent, concatenatedChunks: updatedConcatenatedChunks } = extractToolUseContent(chunk, concatenatedChunks);

if (toolUseContent) {
  console.log("Tool call was made:", toolUseContent);
  // Process the tool call as needed
}

concatenatedChunks = updatedConcatenatedChunks;

This code checks each AIMessageChunk for a tool_use type and processes it if found, allowing you to handle tool calls appropriately ^[4].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[leSullivan]

@dosu Thanks for the answer. The first issue you referenced adresses my Problem but doesn't solve it unfortunately. Does extractToolUseContent work during the streamEvents API ? From my experience the toolcall always gets appended after the normal content, which wouldn't help me

[dosubot[bot]]

The extractToolUseContent function can be used during the streamEvents API to identify tool calls within the streamed content. However, it has limitations that might affect its effectiveness in your scenario:

1. Order of Content: As you mentioned, if the tool call is appended after the normal content, the function might not help in real-time detection since it processes chunks sequentially. This means you might only detect the tool call after processing the normal content.

2. Chunk Concatenation: The function relies on concatenating chunks to form a complete message. If chunks are missing or out of order, it may not correctly reconstruct the tool use content ^[1].

3. Silent Failures: The function uses a try-catch block without handling errors, which means it silently fails if an error occurs during JSON parsing, potentially leading to incomplete or missing tool use data ^[1].

Given these limitations, you might need to implement additional logic to handle the order of events or explore other ways to differentiate between normal content and tool calls in real-time.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[leSullivan]

@dosu is there a way to limit my LLM to either return a tool_call OR respond normally, but never do both ?

[dosubot[bot]]

Yes, you can limit your LLM to either return a tool call or respond normally, but not both, by using the tool_choice parameter in the ChatAnthropicToolsCallOptions interface. You can set tool_choice to a specific function name to enforce tool calling, or leave it as "auto" to let the model decide. By specifying a function, you ensure that the model will only use that tool and not provide a normal response ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[leSullivan]

@dosu I have it on auto. Sometimes it responds with a tool_call and also produces normal text content along with that. I don't want that.

[dosubot[bot]]

To ensure your LLM only returns a tool_call without producing any normal text content when set to "auto", you should use the ChatAnthropicTools class and configure it to handle tool calls exclusively. This involves setting the tool_choice option to "auto" and ensuring that the _prepareAndParseToolCall method processes the tool calls correctly. This method is designed to parse tool calls and return them without additional text content ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}