[Feature]: Support tool calls for DeepSeek.

[tingjun-cs]

🚀 The feature, motivation and pitch

I saw from the official documentation (https://docs.vllm.ai/en/latest/features/tool_calling.html) that sglang supports tool calls, but I can't seem to find the tool parse for deepseekv3/r1. Does this mean that the deepseek model does not support tool calls?

From the DeepSeek official website, it seems that function call support has been implemented on the model side, although it may still be unstable. https://api-docs.deepseek.com/zh-cn/guides/function_calling

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[gaocegege]

DeepSeek R1 does not support tool calling. But QwQ-32B supports it. Ref #14490

[tingjun-cs]

DeepSeek R1 does not support tool calling. But QwQ-32B supports it. Ref #14490

What about DeepSeek-V3? From the DeepSeek official website, it appears that DeepSeek-Chat (also known as DeepSeek-V3) already supports function calls, although it may still be unstable. https://api-docs.deepseek.com/zh-cn/guides/function_calling

[gaocegege]

Yes, v3 supports it.

/cc @WangErXiao

[tingjun-cs]

Yes, v3 supports it.

/cc @WangErXiao

So, what tool-call-parser should be used for DeepSeek-V3?

[gaocegege]

I do not think we support it now.

[tingjun-cs]

I do not think we support it now.

Is there a roadmap or schedule for supporting the tool call feature for DeepSeek-V3? If the official team supports this feature, it seems that it could be implemented?

[gaocegege]

Yes, it is possible technically.

/cc @aarnphm Not sure if you are interested in this

[gaocegege]

@tingjun-cs And, welcome contributions if you are interested in it 😄

[tingjun-cs]

@tingjun-cs And, welcome contributions if you are interested in it 😄

Can you briefly introduce the implementation principles and ideas in this area? I'm currently just starting to get acquainted with this field.

[aarnphm]

Probably want to work on supporting structural tag first, but can check if deepseek has their own function calling format.

You can try pythonic, hermes, or llama3_json

[WangErXiao]

I found DeepSeek V3 tool_calling requires additional handling of the logic within the messages, where the tool_calls information is placed inside the messages. @gaocegege @aarnphm @tingjun-cs
Its chat template contains
{%- for tool in message['tool_calls']%}

[aarnphm]

It seems like they have a special format for this

<｜tool▁calls▁begin｜>

we can write a parser for this.

[WangErXiao]

Yes, it also need to handle logic of serving_chat.py. Currently tools info are in request.tools field. But DeepSeek V3 chat_template only handle tools info of message. @aarnphm

[tingjun-cs]

Yes, it also need to handle logic of serving_chat.py. Currently tools info are in request.tools field. But DeepSeek V3 chat_template only handle tools info of message. @aarnphm

@WangErXiao @aarnphm In the chat-template for DeepSeek-V3, I did not find any logic in DeepSeek-V3 that handles tools, as is done in QWQ-32B. So, how does DeepSeek-V3 implement the parsing of natural language into function calls?

QWQ-32B:

{%- if tools %}
    {{- '<|im_start|>system\n' }}
    {%- if messages[0]['role'] == 'system' %}
        {{- messages[0]['content'] }}
    {%- else %}
        {{- '' }}
    {%- endif %}
    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query."
        "\n\nYou are provided with function signatures within <tools></tools> XML tags:\n"
        "<tools>"
    }}
    {%- for tool in tools %}
        {{- "\n" }}
        {{- tool | tojson }}
    {%- endfor %}
    {{- "\n</tools>\n\nFor each function call, return a json object with function name and "
        "arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n"
        "{\"name\": <function-name>, \"arguments\": <args-json-object>}\n"
        "</tool_call><|im_end|>\n"
    }}
{%- else %}
    {%- if messages[0]['role'] == 'system' %}
        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
    {%- endif %}
{%- endif %}

DeepSeek v3:

{% if not add_generation_prompt is defined %}
    {% set add_generation_prompt = false %}
{% endif %}

{% set ns = namespace(
    is_first=false,
    is_tool=false,
    is_output_first=true,
    system_prompt='',
    is_first_sp=true
) %}

{%- for message in messages %}
    {%- if message['role'] == 'system' %}
        {%- if ns.is_first_sp %}
            {% set ns.system_prompt = ns.system_prompt + message['content'] %}
            {% set ns.is_first_sp = false %}
        {%- else %}
            {% set ns.system_prompt = ns.system_prompt + '\n\n' + message['content'] %}
        {%- endif %}
    {%- endif %}
{%- endfor %}

{{ bos_token }}

{{ ns.system_prompt }}

{%- for message in messages %}
    {%- if message['role'] == 'user' %}
        {%- set ns.is_tool = false -%}
        {{ '' + message['content'] }}
    
    {%- elif message['role'] == 'assistant' and message['content'] is none %}
        {%- set ns.is_tool = false -%}
        {%- for tool in message['tool_calls'] %}
            {%- if not ns.is_first %}
                {{ '' + tool['type'] + '' + tool['function']['name'] 
                + '\n' + '```json' + '\n' 
                + tool['function']['arguments'] + '\n' + '```' + '' }}
                {%- set ns.is_first = true -%}
            {%- else %}
                {{ '\n' + tool['type'] + '' + tool['function']['name'] 
                + '\n' + '```json' + '\n' 
                + tool['function']['arguments'] + '\n' + '```' + '' }}
                {{ '' }}
            {%- endif %}
        {%- endfor %}
    
    {%- elif message['role'] == 'assistant' and message['content'] is not none %}
        {%- if ns.is_tool %}
            {{ '' + message['content'] + '' }}
            {%- set ns.is_tool = false -%}
        {%- else %}
            {{ '' + message['content'] + '' }}
        {%- endif %}
    
    {%- elif message['role'] == 'tool' %}
        {%- set ns.is_tool = true -%}
        {%- if ns.is_output_first %}
            {{ '' + message['content'] + '' }}
            {%- set ns.is_output_first = false %}
        {%- else %}
            {{ '\n' + message['content'] + '' }}
        {%- endif %}
    {%- endif %}
{%- endfor -%}

{% if ns.is_tool %}
    {{ '' }}
{% endif %}
{% if add_generation_prompt and not ns.is_tool %}
    {{ '' }}
{% endif %}