feat: Support JSON Schema in OpenAI-Compatible API #6321
Conversation
📝 Walkthrough""" WalkthroughThe changes introduce support for a new Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant OpenAIProtocol
participant Model
Client->>OpenAIProtocol: CompletionRequest(response_format={"type": "json", "schema": ...})
OpenAIProtocol->>OpenAIProtocol: Validate schema presence
OpenAIProtocol->>Model: GuidedDecodingParams(json=schema)
Model-->>OpenAIProtocol: Generated JSON response
OpenAIProtocol-->>Client: JSON response (validated against schema)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~15–20 minutes Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. 📜 Recent review detailsConfiguration used: .coderabbit.yaml 📒 Files selected for processing (4)
🚧 Files skipped from review as they are similar to previous changes (4)
✨ Finishing Touches
🧪 Generate unit tests
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
|
/bot run |
|
PR_Github #12807 [ run ] triggered by Bot |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
tensorrt_llm/serve/openai_protocol.py(3 hunks)tests/integration/defs/test_e2e.py(1 hunks)tests/integration/test_lists/test-db/l0_a10.yml(1 hunks)tests/unittest/llmapi/apps/_test_openai_chat_json.py(1 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (2)
tests/integration/defs/test_e2e.py (1)
tests/integration/defs/conftest.py (3)
llm_venv(707-723)test_root(2185-2186)unittest_path(90-91)
tensorrt_llm/serve/openai_protocol.py (1)
tensorrt_llm/sampling_params.py (1)
GuidedDecodingParams(14-36)
🪛 Ruff (0.12.2)
tests/unittest/llmapi/apps/_test_openai_chat_json.py
134-134: Undefined name first_message
(F821)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
🔇 Additional comments (7)
tests/integration/test_lists/test-db/l0_a10.yml (1)
25-25: LGTM! Test addition follows established conventions.The new test
test_openai_chat_json_exampleis properly placed among other OpenAI chat-related tests and follows the consistent naming pattern used throughout the test suite.tests/integration/defs/test_e2e.py (1)
1446-1452: LGTM! Function implementation follows established patterns perfectly.The new test function correctly:
- Uses the standard
llm_venvfixture parameter- Constructs the test path using
unittest_path()helper function- Runs pytest on the appropriate test file
_test_openai_chat_json.py- Follows the exact same structure as other similar functions like
test_openai_chat_structural_tag_exampleThe placement is logical and consistent with the surrounding code organization.
tensorrt_llm/serve/openai_protocol.py (3)
55-57: LGTM! Clean addition of new JSON schema support.The addition of
"json"to the response format types and the newschemafield are well-structured and follow the existing patterns in the class.
146-151: Well-implemented validation and integration.The new JSON response format handling correctly validates that a schema is provided when required and properly integrates with the
GuidedDecodingParamsclass. The error message is clear and actionable.
215-216: Documentation correctly updated.The docstring properly reflects the new
'json'response format option alongside the existing supported types.tests/unittest/llmapi/apps/_test_openai_chat_json.py (2)
19-81: Well-structured test fixtures with proper resource management.The fixtures follow pytest best practices with appropriate scoping, proper temporary file cleanup, and a well-defined JSON schema for testing. The guided decoding backend configuration correctly uses "xgrammar" and disables the overlap scheduler as required.
84-148: Comprehensive test design for JSON schema validation.The test effectively validates the new JSON response format feature through multi-turn conversation, proper schema validation using jsonschema, and verification that the model generates distinct responses. The helper function provides good separation of concerns for response creation and validation.
|
PR_Github #12807 [ run ] completed with state |
nv-guomingz
force-pushed
the
user/guomingz/cherry-pick-5957
branch
2 times, most recently
from
182f8c9 to
4dc7dff
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #12875 [ run ] triggered by Bot |
|
PR_Github #12875 [ run ] completed with state |
nv-guomingz
force-pushed
the
user/guomingz/cherry-pick-5957
branch
from
4dc7dff to
c940301
Compare
Signed-off-by: noiji <[email protected]> Signed-off-by: nv-guomingz <[email protected]>
nv-guomingz
force-pushed
the
user/guomingz/cherry-pick-5957
branch
from
c940301 to
cec0cc4
Compare
|
/bot run |
|
PR_Github #13009 [ run ] triggered by Bot |
|
PR_Github #13009 [ run ] completed with state |
Signed-off-by: noiji <[email protected]> Signed-off-by: Shreyas Misra <[email protected]>
Signed-off-by: noiji <[email protected]> Signed-off-by: Ransiki Zhang <[email protected]>
Signed-off-by: noiji <[email protected]> Signed-off-by: Lanyu Liao <[email protected]>
It's a clean version of #5957 and #6197
Summary by CodeRabbit
New Features
Tests