How to use streaming

[k11q]

Hi I am pretty new in coding, and I have no idea how streaming works, right now this will output/stream the logs on the server, so how would I make it way to the client? I am using nuxtjs btw. Can someone enlighten me + example code?

[linonetwo]

Do something like this https://github.com/cocktailpeanut/dalai/blob/de8351215b476cbc25af4263bda53a4aa771cc72/index.js#L444

io.emit in the handleNewToken callback

[k11q]

thank you!

[nfcampos]

Hi,

You can find two examples here

• using Server-Sent Events https://github.com/sullivan-sean/chat-langchainjs/blob/main/pages/api/chat.ts
• using Websockets https://github.com/sullivan-sean/chat-langchainjs/blob/main/pages/api/chat-stream.ts

[k11q]

thank you! got it!

[peps212]

Hi, im using the chat-langchainjs repository to implement the streaming function, but in doing so i encountered the error "TypeError: stream.getReader is not a function"

I opened an issue too, do you know what might cause this?

Here's the link to the issue
#779

[lsan75]

Hi,
Is there any way to know the stream is complete ? Openai returns a '[DONE']' token that doesn't seem to be returned by the handleLLMNewToken function.

[nfcampos]

There are 2 ways:

• The promise returned by .call() method will resolve when the stream is complete
• We also have a handleLLMEnd method which will be called when the stream ends

import { CallbackManager } from "langchain/callbacks";
import { ChatOpenAI } from "langchain/chat_models";
import { HumanChatMessage } from "langchain/schema";

export const run = async () => {
  const chat = new ChatOpenAI({
    maxTokens: 25,
    streaming: true,
    callbackManager: CallbackManager.fromHandlers({
      async handleLLMNewToken(token: string) {
        console.log({ token });
      },
      async handleLLMEnd(output) {
        console.log("End of stream.", output);
      },
    }),
  });

  const response = await chat.call([new HumanChatMessage("Tell me a joke.")]);
};

[lsan75]

thanks a lot, I tried both solutions before. The promise never resolves actually. Maybe because I use the OpenAI llm through azure openai. Can't use the chat models, don't have the rights for now from here.

[gkgourav45]

• error node_modules\langchain\dist\util\stream.js (41:44) @ ReadableStream
• error Error [ReferenceError]: ReadableStream is not defined
  at eval (webpack-internal:///(rsc)/./node_modules/langchain/dist/util/stream.js:44:38)