[gpt-oss] disable tool server initialization if no tool in request #25790
Conversation
qandrew
changed the title
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a targeted fix to prevent a RuntimeError by avoiding tool server initialization when a request does not specify any tools. This is achieved by refactoring the initialization logic into a new _initialize_tool_sessions method, which now includes a guard clause to check for the presence of tools in the request. The refactoring is cleanly applied to both streaming and non-streaming response generation paths. The change is correct and improves the robustness of the tool handling logic. I have one suggestion to remove some potentially confusing commented-out code to improve maintainability.
Summary: First part of fixing vllm-project#25697 we can first fix this error ``` RuntimeError: Attempted to exit cancel scope in a different task than it was entered in ``` by not initializing the tool servers if the request doesn't need tools. Test Plan: - non stream request with tools still gets tools - stream without tools no longer outputs the error - stream with tools still works (but it still outputs the error) Differential Revision: D83387320 Signed-off-by: Andrew Xia <[email protected]>
qandrew
force-pushed
the
export-D83387320
branch
from
778f5f7 to
484f054
Compare
| context: ConversationContext, | ||
| exit_stack: AsyncExitStack): | ||
| # we should only initialize the tool session if the request needs tools | ||
| if len(request.tools) == 0: |
There was a problem hiding this comment.
i believe @Hanchenli also mentioned this issue. can we also cover this in unit test?
There was a problem hiding this comment.
let me know if you have any thoughts @Hanchenli ?
I added a UT, should be ready for review
There was a problem hiding this comment.
This change looks good to me. The issue I mentioned was that the model might generate function_call requests even if we do not provide tools to them.
Signed-off-by: Andrew Xia <[email protected]>
qandrew
requested review from
DarkLight1337,
robertgshaw2-redhat,
simon-mo and
NickLucche
as code owners
Summary:
First part of fixing #25697
we can first fix this error
by not initializing the tool servers if the request doesn't need tools.
Test Plan:
pytest -sv tests/entrypoints/openai/test_serving_responses.pyDifferential Revision: D83387320