Function calling support of openai style api for Qwen1.5 and Qwen2 model #662

Stephen-SMJ · 2024-06-17T09:57:25Z

Hi everyone. This is my implementation of function calling in the Openai-style API of the Qwen1.5/Qwen2 model.
The question I mentioned in #252. Since many people want the code, I share it today.

Note that some details may not be perfect, but you can continue to refine them.

Stephen-SMJ

In line 471:
response, _ = model.chat(
tokenizer,
query,
history=history,
system=system,
stop_words_ids=stop_words_ids,
**gen_kwargs,
)

The chat interface supports function calling now.

Stephen-SMJ · 2024-08-08T09:52:24Z

openai_api/openai_api.py

+from starlette.responses import Response
+from transformers import AutoModelForCausalLM, AutoTokenizer, Qwen2ForCausalLM, Qwen2Tokenizer
+from transformers.generation import GenerationConfig
+from qwen2chat import Qwen2ForChatLM


In line 471:
response, _ = model.chat(
tokenizer,
query,
history=history,
system=system,
stop_words_ids=stop_words_ids,
**gen_kwargs,
)

The chat interface supports function calling now.

lonngxiang · 2024-08-12T01:16:08Z

qwen2运行报错：
openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': "[{'type': 'extra_forbidden', 'loc': ('body', 'tool_choice'), 'msg': 'Extra inputs are not permitted', 'input': 'auto'}, {'type': 'extra_forbidden', 'loc': ('body', 'tools'), 'msg': 'Extra inputs are not permitted', 'input': [{'type': 'function', 'function': {'name': 'get_weather', 'description': '获取某个地点的天气，用户应首先提供一个地点', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': '城市，例如：北京'}}, 'required': ['location']}}}, {'type': 'function', 'function': {'name': 'get_time', 'description': '获取当前时间，给一个地方点', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': '城市，例如：上海'}}, 'required': ['location']}}}]}]", 'type': 'BadRequestError', 'param': None, 'code': 400}

Stephen-SMJ · 2024-08-12T03:58:50Z

qwen2运行报错： openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': "[{'type': 'extra_forbidden', 'loc': ('body', 'tool_choice'), 'msg': 'Extra inputs are not permitted', 'input': 'auto'}, {'type': 'extra_forbidden', 'loc': ('body', 'tools'), 'msg': 'Extra inputs are not permitted', 'input': [{'type': 'function', 'function': {'name': 'get_weather', 'description': '获取某个地点的天气，用户应首先提供一个地点', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': '城市，例如：北京'}}, 'required': ['location']}}}, {'type': 'function', 'function': {'name': 'get_time', 'description': '获取当前时间，给一个地方点', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': '城市，例如：上海'}}, 'required': ['location']}}}]}]", 'type': 'BadRequestError', 'param': None, 'code': 400}

应该是你json file的问题，你可以把json file发出来我有空看看。

lonngxiang · 2024-08-12T04:49:11Z

qwen2运行报错： openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': "[{'type': 'extra_forbidden', 'loc': ('body', 'tool_choice'), 'msg': 'Extra inputs are not permitted', 'input': 'auto'}, {'type': 'extra_forbidden', 'loc': ('body', 'tools'), 'msg': 'Extra inputs are not permitted', 'input': [{'type': 'function', 'function': {'name': 'get_weather', 'description': '获取某个地点的天气，用户应首先提供一个地点', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': '城市，例如：北京'}}, 'required': ['location']}}}, {'type': 'function', 'function': {'name': 'get_time', 'description': '获取当前时间，给一个地方点', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': '城市，例如：上海'}}, 'required': ['location']}}}]}]", 'type': 'BadRequestError', 'param': None, 'code': 400}

应该是你json file的问题，你可以把json file发出来我有空看看。

def send_messages(messages):
    response = client.chat.completions.create(
        model="/qwen-7b",
        messages=messages,
        tools=tools,
        tool_choice="auto",
    )
    return response.choices[0].message

def get_weather(location):
    weather_data = {
        "杭州": "24℃",
        "北京": "18℃",
    }
    return weather_data.get(location, "天气数据不可用")

def get_time(location):
    return datetime.now().strftime("%Y-%m-%d %H:%M:%S")

client = OpenAI(base_url="http://192.1****:10860/v1", api_key="EMPTY")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "获取某个地点的天气，用户应首先提供一个地点",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "城市，例如：北京",
                    }
                },
                "required": ["location"]
            },
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_time",
            "description": "获取当前时间，给一个地方点",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "城市，例如：上海",
                    },
                },
                "required": ["location"],
            },
        }
    }
]

messages = [{"role": "user", "content": "杭州现在时间，天气怎么样？"}]

lonngxiang · 2024-08-12T05:55:47Z

qwen2运行报错： openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': "[{'type': 'extra_forbidden', 'loc': ('body', 'tool_choice'), 'msg': 'Extra inputs are not permitted', 'input': 'auto'}, {'type': 'extra_forbidden', 'loc': ('body', 'tools'), 'msg': 'Extra inputs are not permitted', 'input': [{'type': 'function', 'function': {'name': 'get_weather', 'description': '获取某个地点的天气，用户应首先提供一个地点', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': '城市，例如：北京'}}, 'required': ['location']}}}, {'type': 'function', 'function': {'name': 'get_time', 'description': '获取当前时间，给一个地方点', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': '城市，例如：上海'}}, 'required': ['location']}}}]}]", 'type': 'BadRequestError', 'param': None, 'code': 400}

应该是你json file的问题，你可以把json file发出来我有空看看。
def send_messages(messages):
    response = client.chat.completions.create(
        model="/qwen-7b",
        messages=messages,
        tools=tools,
        tool_choice="auto",
    )
    return response.choices[0].message

def get_weather(location):
    weather_data = {
        "杭州": "24℃",
        "北京": "18℃",
    }
    return weather_data.get(location, "天气数据不可用")

def get_time(location):
    return datetime.now().strftime("%Y-%m-%d %H:%M:%S")

client = OpenAI(base_url="http://192.1****:10860/v1", api_key="EMPTY")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "获取某个地点的天气，用户应首先提供一个地点",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "城市，例如：北京",
                    }
                },
                "required": ["location"]
            },
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_time",
            "description": "获取当前时间，给一个地方点",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "城市，例如：上海",
                    },
                },
                "required": ["location"],
            },
        }
    }
]

messages = [{"role": "user", "content": "杭州现在时间，天气怎么样？"}]

deploy by vllm

Stephen-SMJ · 2024-08-12T07:31:33Z

qwen2运行报错： openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': "[{'type': 'extra_forbidden', 'loc': ('body', 'tool_choice'), 'msg': 'Extra inputs are not permitted', 'input': 'auto'}, {'type': 'extra_forbidden', 'loc': ('body', 'tools'), 'msg': 'Extra inputs are not permitted', 'input': [{'type': 'function', 'function': {'name': 'get_weather', 'description': '获取某个地点的天气，用户应首先提供一个地点', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': '城市，例如：北京'}}, 'required': ['location']}}}, {'type': 'function', 'function': {'name': 'get_time', 'description': '获取当前时间，给一个地方点', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': '城市，例如：上海'}}, 'required': ['location']}}}]}]", 'type': 'BadRequestError', 'param': None, 'code': 400}

应该是你json file的问题，你可以把json file发出来我有空看看。
def send_messages(messages):
    response = client.chat.completions.create(
        model="/qwen-7b",
        messages=messages,
        tools=tools,
        tool_choice="auto",
    )
    return response.choices[0].message

def get_weather(location):
    weather_data = {
        "杭州": "24℃",
        "北京": "18℃",
    }
    return weather_data.get(location, "天气数据不可用")

def get_time(location):
    return datetime.now().strftime("%Y-%m-%d %H:%M:%S")

client = OpenAI(base_url="http://192.1****:10860/v1", api_key="EMPTY")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "获取某个地点的天气，用户应首先提供一个地点",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "城市，例如：北京",
                    }
                },
                "required": ["location"]
            },
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_time",
            "description": "获取当前时间，给一个地方点",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "城市，例如：上海",
                    },
                },
                "required": ["location"],
            },
        }
    }
]

messages = [{"role": "user", "content": "杭州现在时间，天气怎么样？"}]
deploy by vllm

The version of openai is different. You can try the older function calling method, like：
response = client.chat.completions.create(
model="/qwen-7b",
messages=messages,
function_call = "auto"
)
Your json file：
{
"name": "get_weather",
"description": "获取某个地点的天气，用户应首先提供一个地点",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "城市，例如：北京",
}
},
"required": ["location"]
},
}

lonngxiang · 2024-08-12T08:45:28Z

tools

tools 多个agent函数需要怎么修改呢

只修改下面这里还是报错：openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': "[{'type': 'extra_forbidden', 'loc': ('body', 'function_call'), 'msg': 'Extra inputs are not permitted', 'input': 'auto'}, {'type': 'extra_forbidden', 'loc': ('body', 'functions'), 'msg': 'Extra inputs are not permitted', 'input': [{'type': 'function', 'function': {'name': 'get_weather', 'description': '获取某个地点的天气，用户应首先提供一个地点', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': '城市，例如：北京'}}, 'required': ['location']}}}, {'type': 'function', 'function': {'name': 'get_time', 'description': '获取当前时间，给一个地方点', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': '城市，例如：上海'}}, 'required': ['location']}}}]}]", 'type': 'BadRequestError', 'param': None, 'code': 400}

def send_messages(messages):
    response = client.chat.completions.create(
        model="/qwen-7b",
        messages=messages,
        functions=tools,
        function_call="auto",
    )
    return response.choices[0].message

lonngxiang · 2024-08-12T08:48:48Z

tools改成下面这样还是报错
openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': "[{'type': 'extra_forbidden', 'loc': ('body', 'function_call'), 'msg': 'Extra inputs are not permitted', 'input': 'auto'}, {'type': 'extra_forbidden', 'loc': ('body', 'functions'), 'msg': 'Extra inputs are not permitted', 'input': [{'name': 'get_weather', 'description': '获取某个地点的天气，用户应首先提供一个地点', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': '城市，例如：北京'}}, 'required': ['location']}}, {'name': 'get_time', 'description': '获取当前时间，给一个地方点', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': ' 城市，例如：上海'}}, 'required': ['location']}}]}]", 'type': 'BadRequestError', 'param': None, 'code': 400}

tools = [
    {
        
            "name": "get_weather",
            "description": "获取某个地点的天气，用户应首先提供一个地点",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "城市，例如：北京",
                    }
                },
                "required": ["location"]
            },
        
    },
    {
            "name": "get_time",
            "description": "获取当前时间，给一个地方点",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "城市，例如：上海",
                    },
                },
                "required": ["location"],
            },
        
    }
]

function calling support of openai style api for qwen1.5 and qwen2 model

71214d6

This was referenced Jun 17, 2024

how to make Qwen1.5 supporty function calling like openai_api.py in Qwen #252

Closed

Qwen1.5适配OpenAI API接口的 openai_api.py #279

Open

Stephen-SMJ commented Aug 8, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Function calling support of openai style api for Qwen1.5 and Qwen2 model #662

Function calling support of openai style api for Qwen1.5 and Qwen2 model #662

Stephen-SMJ commented Jun 17, 2024

Stephen-SMJ left a comment

Stephen-SMJ Aug 8, 2024

lonngxiang commented Aug 12, 2024

Stephen-SMJ commented Aug 12, 2024

lonngxiang commented Aug 12, 2024

lonngxiang commented Aug 12, 2024

Stephen-SMJ commented Aug 12, 2024

lonngxiang commented Aug 12, 2024

lonngxiang commented Aug 12, 2024

Function calling support of openai style api for Qwen1.5 and Qwen2 model #662

Are you sure you want to change the base?

Function calling support of openai style api for Qwen1.5 and Qwen2 model #662

Conversation

Stephen-SMJ commented Jun 17, 2024

Stephen-SMJ left a comment

Choose a reason for hiding this comment

Stephen-SMJ Aug 8, 2024

Choose a reason for hiding this comment

lonngxiang commented Aug 12, 2024

Stephen-SMJ commented Aug 12, 2024

lonngxiang commented Aug 12, 2024

lonngxiang commented Aug 12, 2024

Stephen-SMJ commented Aug 12, 2024

lonngxiang commented Aug 12, 2024

lonngxiang commented Aug 12, 2024