Skip to content

[TRTLLM-4618][feat] Fix cutlass MoE GEMM fallback failure on FP8 + add e2e test for Mixtral 8x7B FP8 on RTX6000 Pro (SM120) #4335

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
May 19, 2025
Merged

[TRTLLM-4618][feat] Fix cutlass MoE GEMM fallback failure on FP8 + add e2e test for Mixtral 8x7B FP8 on RTX6000 Pro (SM120) #4335

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
e086d37
add mixtral7x8b fp8 test with fixed cutlass fp8 moe gemm
farazkh80 May 15, 2025
436fcbd
update cutlass versions
farazkh80 May 16, 2025
4d246a1
added internal cutlass with fix and docker update
farazkh80 May 16, 2025
e7ced10
Merge branch 'main' into moe-fp8-fix
farazkh80 May 16, 2025
4c8a3b2
added mixtral to pro 6000
farazkh80 May 16, 2025
11a47c6
Merge branch 'main' into moe-fp8-fix
farazkh80 May 18, 2025
2c12776
Merge branch 'main' into moe-fp8-fix
farazkh80 May 18, 2025
9865d64
Merge branch 'main' into moe-fp8-fix
farazkh80 May 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions ...nal_cutlass_kernels/aarch64-linux-gnu/tensorrt_llm_internal_cutlass_kernels_static.tar.xz
View file
Open in desktop
Git LFS file not shown
4 changes: 2 additions & 2 deletions cpp/tensorrt_llm/kernels/internal_cutlass_kernels/aarch64-linux-gnu/version.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
557bea7fa3fcd0b9230edecf3a8dbb10eeb1b8d78928861b1c6da1043f973c35 libtensorrt_llm_internal_cutlass_kernels_static.a
commit 2e68c0113ad486c0026619711e5974f5ae0dc626
aff0f8e617f6ca2f95d121ab9cf0ab17c4e8077cf9e8896bf153d3942a4a50df libtensorrt_llm_internal_cutlass_kernels_static.a
commit d61e7684bc095c8ff5ec540363949bd1f491c960
4 changes: 2 additions & 2 deletions ...rnal_cutlass_kernels/x86_64-linux-gnu/tensorrt_llm_internal_cutlass_kernels_static.tar.xz
View file
Open in desktop
Git LFS file not shown
4 changes: 2 additions & 2 deletions cpp/tensorrt_llm/kernels/internal_cutlass_kernels/x86_64-linux-gnu/version.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
a956018ada7a223b2f9e4aa3331d7bdc4be8f69bdef9e3828f5f1c65beb11699 libtensorrt_llm_internal_cutlass_kernels_static.a
commit 2e68c0113ad486c0026619711e5974f5ae0dc626
9f0a29070b95a7db62f70cc45ef151e27c2a58697a2d50cbb002ff339035fb8e libtensorrt_llm_internal_cutlass_kernels_static.a
commit d61e7684bc095c8ff5ec540363949bd1f491c960
3 changes: 3 additions & 0 deletions tests/integration/defs/test_e2e.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1275,6 +1275,9 @@ def test_ptp_quickstart(llm_root, llm_venv):
pytest.param('Mixtral-8x7B-NVFP4',
'nvfp4-quantized/Mixtral-8x7B-Instruct-v0.1',
marks=skip_pre_blackwell),
pytest.param('Mixtral-8x7B-FP8',
'Mixtral-8x7B-Instruct-v0.1-fp8',
marks=skip_pre_blackwell),
])
def test_ptp_quickstart_advanced(llm_root, llm_venv, model_name, model_path):
print(f"Testing {model_name}.")
Expand Down
1 change: 1 addition & 0 deletions tests/integration/test_lists/qa/llm_release_rtx_pro_6000.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ test_e2e.py::test_ptp_quickstart_advanced[Llama3.1-70B-NVFP4-nvfp4-quantized/Met
test_e2e.py::test_ptp_quickstart_advanced[Llama3.1-70B-FP8-llama-3.1-model/Llama-3.1-70B-Instruct-FP8]
test_e2e.py::test_ptp_quickstart_advanced[Nemotron-Super-49B-v1-FP8-nemotron-nas/Llama-3_3-Nemotron-Super-49B-v1-FP8]
test_e2e.py::test_ptp_quickstart_advanced[Mixtral-8x7B-NVFP4-nvfp4-quantized/Mixtral-8x7B-Instruct-v0.1]
test_e2e.py::test_ptp_quickstart_advanced[Mixtral-8x7B-FP8-Mixtral-8x7B-Instruct-v0.1-fp8]
test_e2e.py::test_ptp_quickstart_advanced_2gpus_sm120[Llama3.1-70B-BF16-llama-3.1-model/Meta-Llama-3.1-70B]
test_e2e.py::test_ptp_quickstart_advanced_2gpus_sm120[Nemotron-Super-49B-v1-BF16-nemotron-nas/Llama-3_3-Nemotron-Super-49B-v1]
test_e2e.py::test_ptp_quickstart_advanced_2gpus_sm120[Mixtral-8x7B-BF16-Mixtral-8x7B-Instruct-v0.1]
1 change: 1 addition & 0 deletions tests/integration/test_lists/test-db/l0_rtx_pro_6000.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,4 @@ l0_rtx_pro_6000:
- test_e2e.py::test_ptp_quickstart_advanced[Llama3.1-70B-FP8-llama-3.1-model/Llama-3.1-70B-Instruct-FP8]
- test_e2e.py::test_ptp_quickstart_advanced[Nemotron-Super-49B-v1-FP8-nemotron-nas/Llama-3_3-Nemotron-Super-49B-v1-FP8]
- test_e2e.py::test_ptp_quickstart_advanced[Mixtral-8x7B-NVFP4-nvfp4-quantized/Mixtral-8x7B-Instruct-v0.1]
- test_e2e.py::test_ptp_quickstart_advanced[Mixtral-8x7B-FP8-Mixtral-8x7B-Instruct-v0.1-fp8]