Fixed streaming support using callback handler for Google genai

[hasansustcse13]

• Description: Fixed streaming support using callback handler for Google genai. Callback handler not working with the ChatGoogleGenerativeAI which has been fixed in this PR.
• Issue: CallbackHandler on_llm_new_token method not fire with ChatGoogleGenerativeAI(Gemini) but works fine with ChatOpenAI when streaming true langchain-google#120 ConversationalRetrievalChain not working with stream callback handler for ChatGoogleGenerativeAI #15006

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Jan 4, 2024 4:26am

[hasansustcse13]

@efriis Could you please look into this PR? Our current project has a large dependency on it.

[efriis]

Hey @hasansustcse13 ! This isn't the interface to follow anymore. Instead, you should stream using .stream() instead!

Let me know if you have any other questions.

Erick

[hasansustcse13]

@efriis Streaming doesn't work if I don't use LangChain Expression Language (LCEL). In our project, we have used streaming using a Callback handler(on_llm_new_token) and different kinds of built-in chains. As Gemini is not implanted the same as OpenAI, it does not work with a callback handler (on_llm_new_token). To need to support streaming in Gemini along with other LLM we have to convert all the code to LCEL. I believe this will be faced by other users also. That's why I have made the Change in the PR. To understand the depth please see the Attached bug in the description. Is there any other way to support streaming without LCEL and with minimal changes?

from langchain.chains import LLMChain
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import PromptTemplate
from langchain_google_genai import ChatGoogleGenerativeAI

prompt_template = "Tell me details about the Company {name} with 4 bullet point?"


def gemini_stream():
    llm = ChatGoogleGenerativeAI(model="gemini-pro", google_api_key='AIzaSyliRCeTRontPQ')
    llm_chain = LLMChain(llm=llm, prompt=PromptTemplate.from_template(prompt_template), verbose=True)
    for content in llm_chain.stream({'name': "Google"}):
        print(content)


def gemini_stream_with_lang_expression():
    llm = ChatGoogleGenerativeAI(model="gemini-pro", google_api_key='AIzaSyCuQ')
    prompt = PromptTemplate.from_template(prompt_template)
    chain = prompt | llm | StrOutputParser()
    for content in chain.stream({'name': "Google"}):
        print(content)


if __name__ == '__main__':
    # This is not working though I have use .stream() method. It gives the output after finising the completetion completely.
    gemini_stream()

    # This is working
    # gemini_stream_with_lang_expression()

[pseudotensor]

@efriis I also agree with @hasansustcse13 that you are confused.

One does not just invoke llm and a .stream() directly, but one can pass an llm with callback into something like load_qa_chain, then streaming is broken for gemini but not other models.

Gemini wasn't always broken, it started with the recent changes that improved exception handling.

[atuncer]

Is there a version that I can import via pip

[RushikeshKothawade07]

how to use map reduce chain type with gemini models using chatgooglegenerativeai and also get streaming responses at the same time using custom callback handler where I want token level streaming by appending in a queue

[efriis]

try ChatGoogleGenerativeAI(...).bind(stream=True)