[Bug]: Tools usage with mistralai/Devstral-Small-2505 fails #18628
Description
Your current environment
The output of python collect_env.py
INFO 05-23 13:44:00 [__init__.py:239] Automatically detected platform cuda.
Collecting environment information...
==============================
System Info
==============================
OS : Ubuntu 24.04.2 LTS (x86_64)
GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
Clang version : Could not collect
CMake version : version 4.0.2
Libc version : glibc-2.39
==============================
PyTorch Info
==============================
PyTorch version : 2.6.0+cu124
Is debug build : False
CUDA used to build PyTorch : 12.4
ROCM used to build PyTorch : N/A
==============================
Python Environment
==============================
Python version : 3.12.3 (main, Feb 4 2025, 14:48:35) [GCC 13.3.0] (64-bit runtime)
Python platform : Linux-6.8.0-59-generic-x86_64-with-glibc2.39
==============================
CUDA / GPU Info
==============================
Is CUDA available : True
CUDA runtime version : 12.8.93
CUDA_MODULE_LOADING set to : LAZY
GPU models and configuration : GPU 0: NVIDIA H200
Nvidia driver version : 570.133.20
cuDNN version : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.9.0
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : True
==============================
CPU Info
==============================
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 44
On-line CPU(s) list: 0-43
Vendor ID: AuthenticAMD
BIOS Vendor ID: QEMU
Model name: AMD EPYC 9654 96-Core Processor
BIOS Model name: pc-q35-8.2 CPU @ 2.0GHz
BIOS CPU family: 1
CPU family: 25
Model: 17
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 44
Stepping: 1
BogoMIPS: 4800.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512_bf16 clzero xsaveerptr wbnoinvd arat npt lbrv nrip_save tsc_scale vmcb_clean pausefilter pfthreshold v_vmsave_vmload vgif avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid fsrm arch_capabilities
Virtualization: AMD-V
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 2.8 MiB (44 instances)
L1i cache: 2.8 MiB (44 instances)
L2 cache: 22 MiB (44 instances)
L3 cache: 704 MiB (44 instances)
NUMA node(s): 1
NUMA node0 CPU(s): 0-43
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-cufile-cu12==1.11.1.6
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.4.0
[pip3] torch==2.6.0
[pip3] torchaudio==2.6.0
[pip3] torchvision==0.21.0
[pip3] transformers==4.52.3
[pip3] triton==3.2.0
[conda] Could not collect
==============================
vLLM Info
==============================
ROCM Version : Could not collect
Neuron SDK Version : N/A
vLLM Version : 0.8.5
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X 0-43 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
==============================
Environment Variables
==============================
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY
🐛 Describe the bug
When calling vLLM /v1/chat/completions with tools, using the local MCP integration, the HTTP 400 is returned.
The error is returned after the example JSON payload is sent to the vLLM.
Here is a curl call example:
curl http://localhost:8000/v1/chat/completions -H "content-type: application/json" -d '{
"model": "mistralai/Devstral-Small-2505",
"stream": false,
"tools": [
{
"type": "function",
"function": {
"name": "getWeather_mcp_spring-mcp-server",
"description": "Get weather information by city name",
"parameters": {
"type": "object",
"properties": {
"cityName": {
"type": "string"
}
},
"required": [
"cityName"
],
"$schema": "http://json-schema.org/draft-07/schema#"
}
}
}
],
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "How is the weather in Roma?"
}
]
},
{
"role": "assistant",
"reasoning_content": null,
"content": null,
"tool_calls": [
{
"id": "5Wsrymo6g",
"type": "function",
"function": {
"name": "getWeather_mcp_spring-mcp-server",
"arguments": "{\"cityName\": \"Roma\"}"
}
}
]
},
{
"role": "tool",
"content": [
{
"type": "text",
"text": "Sunny"
}
],
"name": "getWeather_mcp_spring-mcp-server",
"tool_call_id": "5Wsrymo6g"
}
]
}'Here is an error from the vLLM log:
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] An error occurred in `mistral_common` while applying chat template
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] Traceback (most recent call last):
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] File "/root/.cache/.venv/vllm/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 1238, in apply_mistral_chat_template
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] return tokenizer.apply_chat_template(
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] File "/root/.cache/.venv/vllm/lib/python3.12/site-packages/vllm/transformers_utils/tokenizers/mistral.py", line 378, in apply_chat_template
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] request = make_mistral_chat_completion_request(messages, tools)
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] File "/root/.cache/.venv/vllm/lib/python3.12/site-packages/vllm/transformers_utils/tokenizers/mistral.py", line 176, in make_mistral_chat_completion_request
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] return ChatCompletionRequest(messages=messages,
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] File "/root/.cache/.venv/vllm/lib/python3.12/site-packages/pydantic/main.py", line 253, in __init__
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] pydantic_core._pydantic_core.ValidationError: 2 validation errors for ChatCompletionRequest
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] messages.1.assistant.reasoning_content
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] Extra inputs are not permitted [type=extra_forbidden, input_value=None, input_type=NoneType]
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] For further information visit https://errors.pydantic.dev/2.11/v/extra_forbidden
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] messages.2.tool.content
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] Input should be a valid string [type=string_type, input_value=[{'text': 'Sunny', 'type': 'text'}], input_type=list]
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [chat_utils.py:1256] For further information visit https://errors.pydantic.dev/2.11/v/string_type
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] Error in preprocessing prompt inputs
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] Traceback (most recent call last):
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] File "/root/.cache/.venv/vllm/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 1238, in apply_mistral_chat_template
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] return tokenizer.apply_chat_template(
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] File "/root/.cache/.venv/vllm/lib/python3.12/site-packages/vllm/transformers_utils/tokenizers/mistral.py", line 378, in apply_chat_template
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] request = make_mistral_chat_completion_request(messages, tools)
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] File "/root/.cache/.venv/vllm/lib/python3.12/site-packages/vllm/transformers_utils/tokenizers/mistral.py", line 176, in make_mistral_chat_completion_request
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] return ChatCompletionRequest(messages=messages,
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] File "/root/.cache/.venv/vllm/lib/python3.12/site-packages/pydantic/main.py", line 253, in __init__
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] pydantic_core._pydantic_core.ValidationError: 2 validation errors for ChatCompletionRequest
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] messages.1.assistant.reasoning_content
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] Extra inputs are not permitted [type=extra_forbidden, input_value=None, input_type=NoneType]
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] For further information visit https://errors.pydantic.dev/2.11/v/extra_forbidden
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] messages.2.tool.content
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] Input should be a valid string [type=string_type, input_value=[{'text': 'Sunny', 'type': 'text'}], input_type=list]
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] For further information visit https://errors.pydantic.dev/2.11/v/string_type
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200]
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] The above exception was the direct cause of the following exception:
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200]
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] Traceback (most recent call last):
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] File "/root/.cache/.venv/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 183, in create_chat_completion
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] ) = await self._preprocess_chat(
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] File "/root/.cache/.venv/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_engine.py", line 421, in _preprocess_chat
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] request_prompt = apply_mistral_chat_template(
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] File "/root/.cache/.venv/vllm/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 1259, in apply_mistral_chat_template
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] raise ValueError from e
May 23 14:01:16 test vllm.sh[2533]: ERROR 05-23 14:01:16 [serving_chat.py:200] ValueError
May 23 14:01:16 test vllm.sh[2533]: /root/.cache/.venv/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py:201: RuntimeWarning: coroutine 'AsyncMultiModalItemTracker.all_mm_data' was never awaited
May 23 14:01:16 test vllm.sh[2533]: return self.create_error_response(str(e))
May 23 14:01:16 test vllm.sh[2533]: RuntimeWarning: Enable tracemalloc to get the object allocation traceback
May 23 14:01:16 test vllm.sh[2533]: INFO: 127.0.0.1:36778 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.