fix: support mixture of text & multimodal prompts #6345
Conversation
📝 WalkthroughWalkthroughThe changes relax strict checks and assertions on the presence and count of multimodal inputs in several model forward methods and input processors. Early returns and conditional logic are introduced to handle cases where multimodal data is absent, and variable names are clarified throughout. No public interfaces or method signatures are modified. Additionally, new multimodal modalities are added to example scripts and test suites, and input loading logic is extended to support these new modalities. Changes
Sequence Diagram(s)sequenceDiagram
participant InputProcessor
participant Model
InputProcessor->>InputProcessor: Receive text & multimodal data
alt Multimodal data absent
InputProcessor-->>Caller: Return tokenized IDs, empty multimodal dict
else Multimodal data present
InputProcessor->>InputProcessor: Preprocess multimodal data
InputProcessor-->>Caller: Return tokenized IDs, multimodal dict
end
Caller->>Model: Call forward with input IDs and multimodal params
alt Multimodal params present
Model->>Model: Extract multimodal embeddings
Model->>Model: Fuse embeddings with input tokens
else Multimodal params absent
Model->>Model: Proceed without multimodal embeddings
end
Model-->>Caller: Return model output
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Suggested reviewers
Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. ✨ Finishing Touches
🧪 Generate unit tests
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
coderabbitai
bot
requested review from
FrankD412,
hyukn,
nv-yilinf and
tijyojwad
|
Are there any unit tests we could be adding? |
Signed-off-by: yechank <[email protected]>
Signed-off-by: yechank <[email protected]>
yechank-nvidia
force-pushed
the
fix_mixture_prompts
branch
from
51c8045 to
0d2ee11
Compare
|
@2ez4bz added test for Mistral case. Other models can reference from it. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
tests/integration/test_lists/qa/llm_sanity_test.txt (1)
105-105: Add…-mixture_text_image-Falsevariant for symmetryAll other multimodal test-list entries come in
<modality>-Trueand<modality>-Falsepairs to exercise both code paths. Adding only theTruevariant leaves the “no mixture” branch untested and breaks the implicit pattern.tests/integration/test_lists/qa/examples_test_list.txt (1)
539-539: Mirror the new test with theFalseflag and verify naming consistency
- Include
…-mixture_text_image-Falseto exercise the code path where mixed images & text are disabled (keeps parity with the existingimage-False/Trueandvideo-False/Truecases).- Double-check that the modality string
mixture_text_imagematches what the loader expects (no camel-case or dash variations).tensorrt_llm/inputs/utils.py (1)
473-585: Consider adding unit tests for the new modalities.The implementation of
"multiple_image"and"mixture_text_image"modalities is solid and well-integrated. Given that a commenter on the PR inquired about unit tests, consider adding test cases to verify:
"multiple_image"modality processes multiple images correctly"mixture_text_image"modality handles mixed content with empty media slots- Conditional
"multi_modal_data"inclusion works for both text-only and multimodal promptsWould you like me to generate unit test cases for these new modalities to ensure comprehensive coverage of the mixed text/multimodal prompt functionality?
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (13)
examples/llm-api/quickstart_multimodal.py(5 hunks)tensorrt_llm/_torch/models/modeling_gemma3vl.py(0 hunks)tensorrt_llm/_torch/models/modeling_hyperclovax.py(1 hunks)tensorrt_llm/_torch/models/modeling_llama.py(1 hunks)tensorrt_llm/_torch/models/modeling_llava_next.py(2 hunks)tensorrt_llm/_torch/models/modeling_mistral.py(2 hunks)tensorrt_llm/_torch/models/modeling_phi4mm.py(1 hunks)tensorrt_llm/_torch/models/modeling_vila.py(2 hunks)tensorrt_llm/inputs/utils.py(3 hunks)tests/integration/defs/test_e2e.py(3 hunks)tests/integration/test_lists/qa/examples_test_list.txt(1 hunks)tests/integration/test_lists/qa/llm_sanity_test.txt(1 hunks)tests/integration/test_lists/test-db/l0_h100.yml(1 hunks)
💤 Files with no reviewable changes (1)
- tensorrt_llm/_torch/models/modeling_gemma3vl.py
✅ Files skipped from review due to trivial changes (1)
- tests/integration/test_lists/test-db/l0_h100.yml
🚧 Files skipped from review as they are similar to previous changes (6)
- tensorrt_llm/_torch/models/modeling_hyperclovax.py
- tensorrt_llm/_torch/models/modeling_llama.py
- tensorrt_llm/_torch/models/modeling_mistral.py
- tensorrt_llm/_torch/models/modeling_llava_next.py
- tensorrt_llm/_torch/models/modeling_phi4mm.py
- tensorrt_llm/_torch/models/modeling_vila.py
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit Inference Engine (CODING_GUIDELINES.md)
**/*.py: Python code should conform to Python 3.8+.
Indent Python code with 4 spaces. Do not use tabs.
Always maintain the namespace when importing in Python, even if only one class or function from a module is used.
Python filenames should use snake_case (e.g., some_file.py).
Python classes should use PascalCase (e.g., class SomeClass).
Python functions and methods should use snake_case (e.g., def my_awesome_function():).
Python local variables should use snake_case, and prefix k for variable names that start with a number (e.g., k_99th_percentile).
Python global variables should use upper snake_case and prefix G (e.g., G_MY_GLOBAL).
Python constants should use upper snake_case (e.g., MY_CONSTANT).
Avoid shadowing variables declared in an outer scope in Python.
Initialize all externally visible members of a Python class in the constructor.
For interfaces that may be used outside a Python file, prefer docstrings over comments.
Comments in Python should be reserved for code within a function, or interfaces that are local to a file.
Use Google style docstrings for Python classes and functions, which can be parsed by Sphinx.
Attributes and variables in Python can be documented inline; attribute docstrings will be rendered under the docstring for the class.
Avoid using reflection in Python when functionality can be easily achieved without it.
When using try-except blocks in Python, limit the except to the smallest set of errors possible.
When using try-except blocks to handle multiple possible variable types in Python, keep the body of the try as small as possible, using the else block to implement the logic.
Files:
tests/integration/defs/test_e2e.pyexamples/llm-api/quickstart_multimodal.pytensorrt_llm/inputs/utils.py
**/*.{cpp,h,hpp,cc,cxx,cu,py}
📄 CodeRabbit Inference Engine (CODING_GUIDELINES.md)
All TensorRT-LLM Open Source Software code should contain an NVIDIA copyright header that includes the current year. This includes .cpp, .h, .cu, .py, and any other source files which are compiled or interpreted.
Files:
tests/integration/defs/test_e2e.pyexamples/llm-api/quickstart_multimodal.pytensorrt_llm/inputs/utils.py
🧠 Learnings (5)
📓 Common learnings
Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
tests/integration/test_lists/qa/llm_sanity_test.txt (1)
Learnt from: moraxu
PR: #6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
tests/integration/test_lists/qa/examples_test_list.txt (1)
Learnt from: moraxu
PR: #6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
tests/integration/defs/test_e2e.py (1)
Learnt from: moraxu
PR: #6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
tensorrt_llm/inputs/utils.py (1)
Learnt from: yechank-nvidia
PR: #6254
File: tensorrt_llm/_torch/pyexecutor/model_engine.py:1201-1204
Timestamp: 2025-07-22T09:22:14.726Z
Learning: In TensorRT-LLM's multimodal processing pipeline, shared tensor recovery using from_shared_tensor() is only needed during the context phase. Generation requests reuse the already-recovered tensor data and only need to call strip_for_generation() to remove unnecessary multimodal data while preserving the recovered tensors. This avoids redundant tensor recovery operations during generation.
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
🔇 Additional comments (11)
tests/integration/defs/test_e2e.py (3)
1942-1942: LGTM! Good addition of the new modality to test coverage.The addition of
"mixture_text_image"to the parametrized test extends coverage to the new modality, following the established pattern for multimodal testing.
1990-1999: Well-structured test data for the new mixture modality.The test data effectively covers both text-only and image-based prompts within the same modality, with corresponding empty and populated media lists. This design properly validates the mixture functionality.
2055-2057: Appropriate expected keywords for the mixture modality test.The keywords align well with the test prompts - internet invention-related terms for the text prompt and landscape descriptors for the image prompt.
examples/llm-api/quickstart_multimodal.py (5)
58-77: Well-designed example data for new multimodal modalities.The addition of
"multiple_image"and"mixture_text_image"modalities follows the established pattern and provides appropriate test cases. The mixture modality correctly demonstrates both text-only and image-based prompts with corresponding media structures.
88-91: LGTM! Proper addition of new modalities to CLI choices.The new modality options are correctly added to the argument parser choices, making them available for command-line usage.
107-110: Useful addition of device configuration option.The new
--deviceargument provides helpful flexibility for controlling input tensor device placement while maintaining backward compatibility with the "cpu" default.
162-166: Good repositioning of default data assignment logic.Moving the default prompt and media assignment after model type determination is a logical improvement that ensures the model type is available before processing example data.
175-175: LGTM! Proper use of the configurable device parameter.Replacing the hardcoded "cpu" with
args.devicecorrectly implements the device configuration feature introduced earlier.tensorrt_llm/inputs/utils.py (3)
490-490: LGTM! Clean addition of multiple_image support.The extension to support "multiple_image" modality alongside "image" is well-implemented. Both modalities are processed identically, which maintains consistency in the downstream pipeline while providing a more descriptive option for users working with multiple images.
533-541: LGTM! Appropriate handling for mixture_text_image modality.The implementation correctly filters out empty media items while processing valid images. This design supports the mixed text/image use case where some positions in the media array may be empty (representing text-only segments). The conditional check
if m:prevents potential errors from attempting to load empty or None media items.
573-583: LGTM! Improved formatting and conditional multimodal data inclusion.The refactoring enhances code readability and correctly implements conditional inclusion of multimodal data. The key improvement is only adding
"multi_modal_data"to the input dictionary when multimodal placeholders are actually present (mm_placeholder_countsis truthy). This prevents unnecessary inclusion of empty multimodal data in text-only prompts, which aligns perfectly with the PR objective of supporting mixed text and multimodal prompts.
|
/bot run |
|
PR_Github #13352 [ run ] triggered by Bot |
|
PR_Github #13352 [ run ] completed with state |
| choices=["image", "video", "audio", "image_audio"], | ||
| choices=[ | ||
| "image", "video", "audio", "image_audio", | ||
| "multiple_image", "mixture_text_image" |
There was a problem hiding this comment.
Sorry to comment again on a closed PR, but quick question — do we actually need to create/define a new modality (other than image, video etc) here when there are multiple images or videos?
Can we update default loader to accommodate various combinations? [pure_txt, multiple_image with txt, image with txt, etc.]
Signed-off-by: yechank <[email protected]> Signed-off-by: Lanyu Liao <[email protected]>
Signed-off-by: yechank <[email protected]>
Summary by CodeRabbit
Bug Fixes
New Features
Refactor