[Bug]: tool parsers can block the async event loop #18304
Description
Your current environment
I hit this in some testing of the berkeley function calling leaderboard with pythonic_tool_parser.py (ie pythonic) as my tool call parser with a Llama 4 Scout model on vLLM 0.8.5.post1.
🐛 Describe the bug
Tool call parsers run within an async event loop. If a tool call parser takes a long time servicing a single request for whatever reason, it can block that event loop until the parser is finished executing. This ends up blocking the processing of other requests until the tool call parser finishes. This is fine if the tool call parsers are all written in such a way that they have a lower upper-bound of possible execution time.
However, in my case, the pythonic tool parser had a very long (30+ minutes) execution time for a single request, due to a complex regular expression match that appears to trigger some catastrophic backtracking behavior on certain inputs. Here's a very contrived example that takes the regex from pythonic_tool_parser.py to demonstrate this:
import re
test_string = """[foo(testingsomethingagain=True, a=, b=, c=, d=, e=
[reschedule_event(event_identifier="your_event_id", new_datetime="next_friday_date_in_iso_format")]
[reschedule_event(event_identifier="12345", new_datetime="2024-09-27T00:00:00Z")]
[reschedule_event(event_identifier="12345", new_datetime="2024-09-27T00:00:00Z")]
[reschedule_event(event_identifier="12345", new_datetime="2024-09-27T00:00:00Z")]
[reschedule_event(event_identifier="12345", new_datetime="2024-09-27T00:00:00Z")]
[reschedule_event(event_identifier="12345", new_datetime="2024-09-27T00:00:00Z")]
[reschedule_event(event_identifier="12345", new_datetime="2024-09-27T00:00:00Z")]
]"""
TOOL_CALL_REGEX = re.compile(
r"\[([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s)?\),\s*)*([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s*)?\)\s*)+\]",
re.DOTALL)
if TOOL_CALL_REGEX.match(test_string):
print("match")
else:
print("no match")
On my laptop, the example above takes over 15 seconds to execute.
$ time python test_regex.py
match
python test_regex.py 15.55s user 0.06s system 99% cpu 15.726 total
There are two types of fixes possible here.
The first is to audit all the tool call parsers and replace any regular expressions that are susceptible to catastrophic backtracking with something else, as well as a general once-over to identify any other potentially long-running operations in the parsers.
The second would be to prevent tool call parsers from blocking the event loop by executing them in a background thread. This wouldn't fix the issue of some individual responses being slow when encountering a misbehaving tool parser, but it would keep that misbehaving parser from dramatically impacting the handling of other requests.
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.