[Question]: Groq: client.chat.completions.create function location.

[fnavarro94]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

Hi, I am trying to get a full understanding on how function a function call message si done, in particular when using Groq with a FunctionCalling agent created in the following way. I want to understand what is the structure of the message passed to the most basic llm agent in the chain.
where the message is passed to

from llama_index.llms.groq import Groq
llm = Groq(
    model="llama3-groq-70b-8192-tool-use-preview",
    api_key=GROQ_API_KEY,
    temperature=0
)

agent_worker = FunctionCallingAgentWorker.from_tools(
    [multiply_tool, add_tool, subtract_tool, query_tool],
    llm=llm,
    verbose=True,
    allow_parallel_tool_calls=False,
    system_prompt=(
       ""
    ),
)

I was able to trace the user input up until the file
https://github.com/run-llama/llama_index/blob/67c7e50e782f9ce12e1fd76b4ac3a131a505f19b/llama-index-integrations/llms/llama-index-llms-openai/llama_index/llms/openai/base.py

Up to the function call on line 429

response = client.chat.completions.create(
                messages=message_dicts,
                stream=False,
                **self._get_model_kwargs(**kwargs),
            )

however I am unable to find where the definition of client.chat.completions.create is when I am using a groq model. If I try to print in python the location of the function through

def _get_client(self) -> SyncOpenAI:
        print("_get_client is running")
        if not self.reuse_client:
            print("not reuse client")
            return SyncOpenAI(**self._get_credential_kwargs())

I get sent to an openai module. This one ( on my local compyter it is not a link)
lib/python3.11/site-packages/openai/_utils/_utils.py

which is not part of llamaindex and it is also not Groq.

Can you please point me to the correct source code that handles the user message with function call for through tool use with Groq please?

[dosubot]

The source code that handles user messages with function calls through tool use when using a Groq model, specifically in relation to the client.chat.completions.create function, can be found in the following snippet:

is_function = False
for response in client.chat.completions.create(
    messages=message_dicts,
    **self._get_model_kwargs(stream=True, **kwargs),
):
    response = cast(ChatCompletionChunk, response)
    if len(response.choices) > 0:
        delta = response.choices[0].delta
    else:
        if self._is_azure_client():
            continue
        else:
            delta = ChoiceDelta()

    # check if this chunk is the start of a function call
    if delta.tool_calls:
        is_function = True

    # update using deltas
    role = delta.role or MessageRole.ASSISTANT
    content_delta = delta.content or ""
    content += content_delta

    additional_kwargs = {}
    if is_function:
        tool_calls = update_tool_calls(tool_calls, delta.tool_calls)
        if tool_calls:
            additional_kwargs["tool_calls"] = tool_calls

    yield ChatResponse(
        message=ChatMessage(
            role=role,
            content=content,
            additional_kwargs=additional_kwargs,
        ),
        delta=content_delta,
        raw=response,
        additional_kwargs=self._get_response_token_counts(response),
    )

This code is responsible for processing the responses from the client.chat.completions.create function, checking for function calls, and updating the message content and additional keyword arguments accordingly ^[1].

To continue talking to Dosu, mention @dosu.

[fnavarro94]

That is not the information I am looking for. I want the source code of client.chat.completions.create for the particular case of using Groq API through llamaindex

[dosubot]

The source code for client.chat.completions.create when using the Groq API through LlamaIndex can be found in the OpenAIMultiModal class within the _chat method. Here is the relevant part of the code:

def _chat(self, messages: Sequence[ChatMessage], **kwargs: Any) -> ChatResponse:
    all_kwargs = self._get_model_kwargs(**kwargs)
    message_dicts = to_openai_message_dicts(messages)
    response = self._client.chat.completions.create(
        messages=message_dicts,
        stream=False,
        **all_kwargs,
    )
    openai_message = response.choices[0].message
    message = from_openai_message(openai_message)

    return ChatResponse(
        message=message,
        raw=response,
        additional_kwargs=self._get_response_token_counts(response),
    )

This method uses the self._client.chat.completions.create function to create chat completions. The OpenAIMultiModal class is part of the LlamaIndex framework, which integrates with OpenAI's API to handle chat completions ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[logan-markewich]

The FunctionCallingAgentWorker uses the lower level llm.chat_with_tools() and llm.get_tool_calls_from_response() functions

For groq, we can see that it its just a light wrapper around the openai llm

llama_index/llama-index-integrations/llms/llama-index-llms-groq/llama_index/llms/groq/base.py

Line 7 in be7f61a

class Groq(OpenAILike):

Looking at the openai llm class, we can see that the tools are prepared here

llama_index/llama-index-integrations/llms/llama-index-llms-openai/llama_index/llms/openai/base.py

Line 823 in be7f61a

def _prepare_chat_with_tools(

Which that method is called from the base class here

llama_index/llama-index-core/llama_index/core/llms/function_calling.py

Line 40 in be7f61a

chat_kwargs = self._prepare_chat_with_tools(

And then tool calls are parsed from the response here

llama_index/llama-index-integrations/llms/llama-index-llms-openai/llama_index/llms/openai/base.py

Line 877 in be7f61a

def get_tool_calls_from_response(

[fnavarro94]

Thank you for the quick response. I was actually able to find those files.
I was trying to dig deeper into the function call

response = client.chat.completions.create(
                messages=message_dicts,
                stream=False,
                **self._get_model_kwargs(**kwargs),
            )

found in file (429)
llama_index/llama-index-integrations/llms/llama-index-llms-openai/llama_index/llms/openai/base.py

I am interested in looking at the prompt that ultimately gets sent to the groq API, when it is no longer formated for llamaindex structure and classes. I am interested in that piece of text. But I am having trouble tracing back that function `client.chat.completions.create. If I try to print its location with

 import inspect
            print(inspect.getfile(client.chat.completions.create))

I get sent to https://github.com/openai/openai-python/blob/main/src/openai/_utils/_utils.py. Which clearly is not the place for the Groq API call which i am looking for.

I am particularly interested in looking at the instruction sent to grow API which details how to structure the output so that it can properly do function calling.

Any help with this would be greatly appreciated.

Cheers