hopper-style context MLA #5713

zhou-yuxin · 2025-07-03T12:12:20Z

Hopper-style Context MLA

PerkzZheng

the changes on the fmha_v2 source codes have already been reviewed internall, so I will just skip them.

I think you still need to update the fmhaRunner.cpp about how the TMA descriptors are set. We still use that path even though most of the kernels are compiled from the source codes instead of cubins.

cpp/kernels/fmha_v2/setup.py

...tention/cubin/fmha_v2_flash_attention_bf16_64_128_S_q_paged_kv_192x128_tma_ws_sm90.cubin.cpp

Signed-off-by: Yuxin <[email protected]>

Signed-off-by: Tomer Asida <[email protected]> Signed-off-by: Yuxin <[email protected]>

…IDIA#5694) Signed-off-by: Yiqing Yan <[email protected]> Signed-off-by: Yuxin <[email protected]>

Signed-off-by: qqiao <[email protected]> Signed-off-by: Yuxin <[email protected]>

Signed-off-by: Fred Wei <[email protected]> Signed-off-by: Yuxin <[email protected]>

Signed-off-by: Omer Ullman Argov <[email protected]> Signed-off-by: Yuxin <[email protected]>

Signed-off-by: Yuxin <[email protected]>

…MLA; 2. fix setup.py that add cubin headers; 3. remove 192x128 cubins Signed-off-by: Yuxin <[email protected]>

2. only generate SEPARATE_Q_K_V for sm90 192x128; 3. delete cubins which is not in (sm90 && head_size=128); Signed-off-by: Yuxin <[email protected]>

Signed-off-by: Netanel Haber <[email protected]> Signed-off-by: Yuxin <[email protected]>

Signed-off-by: Robin Kobus <[email protected]> Signed-off-by: Yuxin <[email protected]>

NVIDIA#5695) Signed-off-by: nv-guomingz <[email protected]> Signed-off-by: Yuxin <[email protected]>

Signed-off-by: Rashid K <[email protected]> Co-authored-by: Zhihan Jiang <[email protected]> Signed-off-by: Yuxin <[email protected]>

…n trt backend LLM (NVIDIA#5705) Signed-off-by: Zhenhuan Chen <[email protected]> Signed-off-by: Yuxin <[email protected]>

…every hour (NVIDIA#5658) Signed-off-by: Po-Wei Wang (Vincent) <[email protected]> Signed-off-by: Yuxin <[email protected]>

…ate (NVIDIA#5675) This PR moves MTPSampler and derivatives to use the universal seq_slot indexing for sampling. This is the last piece of the puzzle: After this, all of the samplers will use this format. See: 6ee94c7 Signed-off-by: Netanel Haber <[email protected]> Signed-off-by: Yuxin <[email protected]>

Signed-off-by: Lucas Liebenwein <[email protected]> Signed-off-by: Frida Hou <[email protected]> Co-authored-by: Neta Zmora <[email protected]> Co-authored-by: Fridah-nv <[email protected]> Signed-off-by: Yuxin <[email protected]>

…#5115) Signed-off-by: Clay <[email protected]> Signed-off-by: Yuxin <[email protected]>

…st_openai_chat_structural_tag_example (NVIDIA#5740) Signed-off-by: Venky <[email protected]> Signed-off-by: Yuxin <[email protected]>

Signed-off-by: Xin He (SW-GPU) <[email protected]> Signed-off-by: Yuxin <[email protected]>

Signed-off-by: Superjomn <[email protected]> Signed-off-by: Yuxin <[email protected]>

Signed-off-by: nv-guomingz <[email protected]> Signed-off-by: Yuxin <[email protected]>

) Signed-off-by: zhengd-nv <[email protected]> Signed-off-by: Yuxin <[email protected]>

…A#5397) Signed-off-by: Yi Zhang <[email protected]> Signed-off-by: Yuxin <[email protected]>

) Signed-off-by: Kaiyu Xie <[email protected]> Signed-off-by: Yuxin <[email protected]>

…lculation. (NVIDIA#5625) Signed-off-by: Frank Di Natale <[email protected]> Signed-off-by: Yuxin <[email protected]>

PerkzZheng · 2025-07-17T01:25:33Z

/bot run

tensorrt-cicd · 2025-07-17T01:30:49Z

PR_Github #12132 [ run ] triggered by Bot

tensorrt-cicd · 2025-07-17T06:04:06Z

PR_Github #12132 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #9015 completed with status: 'FAILURE'

coderabbitai · 2025-07-17T06:08:03Z

Important

Review skipped

More than 25% of the files skipped due to max files limit. The review is being skipped to prevent a low-quality review.

116 files out of 223 files are above the max files limit of 100. Please upgrade to Pro plan to get higher limits.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing Touches

📝 Generate Docstrings

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Explain this complex logic.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai explain this code block.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and explain its main purpose.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

zhou-yuxin · 2025-07-17T09:46:50Z

/bot run

qsang-nv · 2025-07-18T02:18:47Z

/bot run --disable-fail-fast

tensorrt-cicd · 2025-07-18T02:23:57Z

PR_Github #12259 [ run ] triggered by Bot

tensorrt-cicd · 2025-07-18T09:25:37Z

PR_Github #12259 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #9102 completed with status: 'FAILURE'

zhou-yuxin · 2025-07-21T00:42:59Z

/bot run

zhou-yuxin · 2025-07-21T00:48:13Z

/bot run --disable-fail-fast

tensorrt-cicd · 2025-07-21T00:48:36Z

PR_Github #12392 [ run ] triggered by Bot

tensorrt-cicd · 2025-07-21T00:53:41Z

PR_Github #12393 [ run ] triggered by Bot

tensorrt-cicd · 2025-07-21T00:53:44Z

PR_Github #12392 [ run ] completed with state ABORTED

tensorrt-cicd · 2025-07-21T06:52:49Z

PR_Github #12393 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #9212 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

zhou-yuxin · 2025-07-21T07:20:29Z

/bot run

tensorrt-cicd · 2025-07-21T07:42:38Z

PR_Github #12426 [ run ] triggered by Bot

tensorrt-cicd · 2025-07-21T13:37:33Z

PR_Github #12426 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #9242 completed with status: 'SUCCESS'

Signed-off-by: Yuxin <[email protected]> Signed-off-by: Tomer Asida <[email protected]> Signed-off-by: Yiqing Yan <[email protected]> Signed-off-by: qqiao <[email protected]> Signed-off-by: Fred Wei <[email protected]> Signed-off-by: Omer Ullman Argov <[email protected]> Signed-off-by: Netanel Haber <[email protected]> Signed-off-by: Robin Kobus <[email protected]> Signed-off-by: nv-guomingz <[email protected]> Signed-off-by: Rashid K <[email protected]> Signed-off-by: Zhenhuan Chen <[email protected]> Signed-off-by: Po-Wei Wang (Vincent) <[email protected]> Signed-off-by: Netanel Haber <[email protected]> Signed-off-by: Lucas Liebenwein <[email protected]> Signed-off-by: Frida Hou <[email protected]> Signed-off-by: Clay <[email protected]> Signed-off-by: Venky <[email protected]> Signed-off-by: Xin He (SW-GPU) <[email protected]> Signed-off-by: Superjomn <[email protected]> Signed-off-by: zhengd-nv <[email protected]> Signed-off-by: Yi Zhang <[email protected]> Signed-off-by: Kaiyu Xie <[email protected]> Signed-off-by: Frank Di Natale <[email protected]> Signed-off-by: Balaram Buddharaju <[email protected]> Signed-off-by: Linda-Stadter <[email protected]> Signed-off-by: Shunkang <[email protected]> Signed-off-by: Yuan Tong <[email protected]> Signed-off-by: Tailing Yuan <[email protected]> Signed-off-by: Faraz Khoubsirat <[email protected]> Signed-off-by: peaceh <[email protected]> Signed-off-by: ixlmar <[email protected]> Signed-off-by: Hui Gao <[email protected]> Signed-off-by: ShiXiaowei02 <[email protected]> Signed-off-by: Chuang Zhu <[email protected]> Signed-off-by: Stefan Niebler <[email protected]> Signed-off-by: jthomson04 <[email protected]> Signed-off-by: Xianjie <[email protected]> Signed-off-by: Xianjie Qiao <[email protected]> Signed-off-by: Julien Debache <[email protected]> Signed-off-by: Yanchao Lu <[email protected]> Signed-off-by: Yiteng Niu <[email protected]> Signed-off-by: Daniel Stokes <[email protected]> Signed-off-by: bhsueh <[email protected]> Signed-off-by: Bo Li <[email protected]> Signed-off-by: Christina Zhang <[email protected]> Signed-off-by: xinhe-nv <[email protected]> Signed-off-by: Dylan Chen <[email protected]> Signed-off-by: Daniel Campora <[email protected]> Signed-off-by: David Clark <[email protected]> Signed-off-by: yechank <[email protected]> Signed-off-by: Jin Li <[email protected]> Signed-off-by: JieXin Liang <[email protected]> Signed-off-by: Venky Ganesh <[email protected]> Signed-off-by: Enwei Zhu <[email protected]> Signed-off-by: Xiwen Yu <[email protected]> Signed-off-by: Yegor <[email protected]> Signed-off-by: Yegor Yershov <[email protected]> Signed-off-by: Yukun He <[email protected]> Signed-off-by: raayandhar <[email protected]> Signed-off-by: Dom Brown <[email protected]> Signed-off-by: Iman Tabrizian <[email protected]> Signed-off-by: xsimmons <[email protected]> Signed-off-by: jiahanc <[email protected]> Signed-off-by: Jhao-Ting Chen <[email protected]> Signed-off-by: Wanli Jiang <[email protected]> Signed-off-by: Erin Ho <[email protected]> Signed-off-by: Chenfei Zhang <[email protected]> Signed-off-by: Dongxu Yang <[email protected]> Signed-off-by: Hao Lu <[email protected]> Signed-off-by: William Zhang <[email protected]> Signed-off-by: Ubuntu <[email protected]> Signed-off-by: Hanjun Cho <[email protected]> Signed-off-by: junq <[email protected]> Signed-off-by: Aurelien Chartier <[email protected]> Signed-off-by: Anthony Chang <[email protected]> Signed-off-by: CarstyYou <[email protected]> Signed-off-by: Jinyang Yuan <[email protected]> Signed-off-by: narutolhy <[email protected]> Signed-off-by: ZhanruiSunCh <[email protected]> Signed-off-by: wili-65535 <[email protected]> Signed-off-by: Frank <[email protected]> Signed-off-by: Yilin Zhang <[email protected]> Signed-off-by: William Tambellini <[email protected]> Co-authored-by: tomeras91 <[email protected]> Co-authored-by: Yiqing Yan <[email protected]> Co-authored-by: Emma Qiao <[email protected]> Co-authored-by: WeiHaocheng <[email protected]> Co-authored-by: Omer Ullman Argov <[email protected]> Co-authored-by: Netanel Haber <[email protected]> Co-authored-by: Robin Kobus <[email protected]> Co-authored-by: nv-guomingz <[email protected]> Co-authored-by: Rashid Kaleem <[email protected]> Co-authored-by: Zhihan Jiang <[email protected]> Co-authored-by: Zhenhuan Chen <[email protected]> Co-authored-by: Po-Wei (Vincent) <[email protected]> Co-authored-by: Lucas Liebenwein <[email protected]> Co-authored-by: Neta Zmora <[email protected]> Co-authored-by: Fridah-nv <[email protected]> Co-authored-by: Clay <[email protected]> Co-authored-by: Venky <[email protected]> Co-authored-by: xinhe-nv <[email protected]> Co-authored-by: Yan Chunwei <[email protected]> Co-authored-by: Zheng Duan <[email protected]> Co-authored-by: Yi Zhang <[email protected]> Co-authored-by: Kaiyu Xie <[email protected]> Co-authored-by: Frank <[email protected]> Co-authored-by: brb-nv <[email protected]> Co-authored-by: Linda <[email protected]> Co-authored-by: Shunkangz <[email protected]> Co-authored-by: Yuan Tong <[email protected]> Co-authored-by: Tailing Yuan <[email protected]> Co-authored-by: Faraz <[email protected]> Co-authored-by: peaceh-nv <[email protected]> Co-authored-by: ixlmar <[email protected]> Co-authored-by: HuiGao-NV <[email protected]> Co-authored-by: Chuang Zhu <[email protected]> Co-authored-by: ShiXiaowei02 <[email protected]> Co-authored-by: Stefan Niebler <[email protected]> Co-authored-by: jthomson04 <[email protected]> Co-authored-by: Xianjie Qiao <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Julien Debache <[email protected]> Co-authored-by: Yanchao Lu <[email protected]> Co-authored-by: Yiteng Niu <[email protected]> Co-authored-by: Daniel Stokes <[email protected]> Co-authored-by: bhsueh_NV <[email protected]> Co-authored-by: Bo Li <[email protected]> Co-authored-by: ChristinaZ <[email protected]> Co-authored-by: Larry <[email protected]> Co-authored-by: DylanChen-NV <[email protected]> Co-authored-by: Daniel Cámpora <[email protected]> Co-authored-by: davidclark-nv <[email protected]> Co-authored-by: Nikita Korobov <[email protected]> Co-authored-by: Yechan Kim <[email protected]> Co-authored-by: liji-nv <[email protected]> Co-authored-by: JieXin Liang <[email protected]> Co-authored-by: Enwei Zhu <[email protected]> Co-authored-by: xiweny <[email protected]> Co-authored-by: Yegor <[email protected]> Co-authored-by: Yukun He <[email protected]> Co-authored-by: Raayan Dhar <[email protected]> Co-authored-by: Dom Brown <[email protected]> Co-authored-by: Chang Liu <[email protected]> Co-authored-by: Pamela Peng <[email protected]> Co-authored-by: Iman Tabrizian <[email protected]> Co-authored-by: xavier-nvidia <[email protected]> Co-authored-by: jiahanc <[email protected]> Co-authored-by: Jhao-Ting Chen <[email protected]> Co-authored-by: Wanli Jiang <[email protected]> Co-authored-by: Erin <[email protected]> Co-authored-by: chenfeiz0326 <[email protected]> Co-authored-by: dongxuy04 <[email protected]> Co-authored-by: 2ez4bz <[email protected]> Co-authored-by: Hanjun Cho <[email protected]> Co-authored-by: Ubuntu <[email protected]> Co-authored-by: QI JUN <[email protected]> Co-authored-by: Aurelien Chartier <[email protected]> Co-authored-by: Anthony Chang <[email protected]> Co-authored-by: CarstyYou <[email protected]> Co-authored-by: Jinyang Yuan <[email protected]> Co-authored-by: narutolhy <[email protected]> Co-authored-by: Zhanrui Sun <[email protected]> Co-authored-by: wili <[email protected]> Co-authored-by: wili-65535 <[email protected]> Co-authored-by: Void <[email protected]> Co-authored-by: William Tambellini <[email protected]> Signed-off-by: Shreyas Misra <[email protected]>

Signed-off-by: Yuxin <[email protected]> Signed-off-by: Tomer Asida <[email protected]> Signed-off-by: Yiqing Yan <[email protected]> Signed-off-by: qqiao <[email protected]> Signed-off-by: Fred Wei <[email protected]> Signed-off-by: Omer Ullman Argov <[email protected]> Signed-off-by: Netanel Haber <[email protected]> Signed-off-by: Robin Kobus <[email protected]> Signed-off-by: nv-guomingz <[email protected]> Signed-off-by: Rashid K <[email protected]> Signed-off-by: Zhenhuan Chen <[email protected]> Signed-off-by: Po-Wei Wang (Vincent) <[email protected]> Signed-off-by: Netanel Haber <[email protected]> Signed-off-by: Lucas Liebenwein <[email protected]> Signed-off-by: Frida Hou <[email protected]> Signed-off-by: Clay <[email protected]> Signed-off-by: Venky <[email protected]> Signed-off-by: Xin He (SW-GPU) <[email protected]> Signed-off-by: Superjomn <[email protected]> Signed-off-by: zhengd-nv <[email protected]> Signed-off-by: Yi Zhang <[email protected]> Signed-off-by: Kaiyu Xie <[email protected]> Signed-off-by: Frank Di Natale <[email protected]> Signed-off-by: Balaram Buddharaju <[email protected]> Signed-off-by: Linda-Stadter <[email protected]> Signed-off-by: Shunkang <[email protected]> Signed-off-by: Yuan Tong <[email protected]> Signed-off-by: Tailing Yuan <[email protected]> Signed-off-by: Faraz Khoubsirat <[email protected]> Signed-off-by: peaceh <[email protected]> Signed-off-by: ixlmar <[email protected]> Signed-off-by: Hui Gao <[email protected]> Signed-off-by: ShiXiaowei02 <[email protected]> Signed-off-by: Chuang Zhu <[email protected]> Signed-off-by: Stefan Niebler <[email protected]> Signed-off-by: jthomson04 <[email protected]> Signed-off-by: Xianjie <[email protected]> Signed-off-by: Xianjie Qiao <[email protected]> Signed-off-by: Julien Debache <[email protected]> Signed-off-by: Yanchao Lu <[email protected]> Signed-off-by: Yiteng Niu <[email protected]> Signed-off-by: Daniel Stokes <[email protected]> Signed-off-by: bhsueh <[email protected]> Signed-off-by: Bo Li <[email protected]> Signed-off-by: Christina Zhang <[email protected]> Signed-off-by: xinhe-nv <[email protected]> Signed-off-by: Dylan Chen <[email protected]> Signed-off-by: Daniel Campora <[email protected]> Signed-off-by: David Clark <[email protected]> Signed-off-by: yechank <[email protected]> Signed-off-by: Jin Li <[email protected]> Signed-off-by: JieXin Liang <[email protected]> Signed-off-by: Venky Ganesh <[email protected]> Signed-off-by: Enwei Zhu <[email protected]> Signed-off-by: Xiwen Yu <[email protected]> Signed-off-by: Yegor <[email protected]> Signed-off-by: Yegor Yershov <[email protected]> Signed-off-by: Yukun He <[email protected]> Signed-off-by: raayandhar <[email protected]> Signed-off-by: Dom Brown <[email protected]> Signed-off-by: Iman Tabrizian <[email protected]> Signed-off-by: xsimmons <[email protected]> Signed-off-by: jiahanc <[email protected]> Signed-off-by: Jhao-Ting Chen <[email protected]> Signed-off-by: Wanli Jiang <[email protected]> Signed-off-by: Erin Ho <[email protected]> Signed-off-by: Chenfei Zhang <[email protected]> Signed-off-by: Dongxu Yang <[email protected]> Signed-off-by: Hao Lu <[email protected]> Signed-off-by: William Zhang <[email protected]> Signed-off-by: Ubuntu <[email protected]> Signed-off-by: Hanjun Cho <[email protected]> Signed-off-by: junq <[email protected]> Signed-off-by: Aurelien Chartier <[email protected]> Signed-off-by: Anthony Chang <[email protected]> Signed-off-by: CarstyYou <[email protected]> Signed-off-by: Jinyang Yuan <[email protected]> Signed-off-by: narutolhy <[email protected]> Signed-off-by: ZhanruiSunCh <[email protected]> Signed-off-by: wili-65535 <[email protected]> Signed-off-by: Frank <[email protected]> Signed-off-by: Yilin Zhang <[email protected]> Signed-off-by: William Tambellini <[email protected]> Co-authored-by: tomeras91 <[email protected]> Co-authored-by: Yiqing Yan <[email protected]> Co-authored-by: Emma Qiao <[email protected]> Co-authored-by: WeiHaocheng <[email protected]> Co-authored-by: Omer Ullman Argov <[email protected]> Co-authored-by: Netanel Haber <[email protected]> Co-authored-by: Robin Kobus <[email protected]> Co-authored-by: nv-guomingz <[email protected]> Co-authored-by: Rashid Kaleem <[email protected]> Co-authored-by: Zhihan Jiang <[email protected]> Co-authored-by: Zhenhuan Chen <[email protected]> Co-authored-by: Po-Wei (Vincent) <[email protected]> Co-authored-by: Lucas Liebenwein <[email protected]> Co-authored-by: Neta Zmora <[email protected]> Co-authored-by: Fridah-nv <[email protected]> Co-authored-by: Clay <[email protected]> Co-authored-by: Venky <[email protected]> Co-authored-by: xinhe-nv <[email protected]> Co-authored-by: Yan Chunwei <[email protected]> Co-authored-by: Zheng Duan <[email protected]> Co-authored-by: Yi Zhang <[email protected]> Co-authored-by: Kaiyu Xie <[email protected]> Co-authored-by: Frank <[email protected]> Co-authored-by: brb-nv <[email protected]> Co-authored-by: Linda <[email protected]> Co-authored-by: Shunkangz <[email protected]> Co-authored-by: Yuan Tong <[email protected]> Co-authored-by: Tailing Yuan <[email protected]> Co-authored-by: Faraz <[email protected]> Co-authored-by: peaceh-nv <[email protected]> Co-authored-by: ixlmar <[email protected]> Co-authored-by: HuiGao-NV <[email protected]> Co-authored-by: Chuang Zhu <[email protected]> Co-authored-by: ShiXiaowei02 <[email protected]> Co-authored-by: Stefan Niebler <[email protected]> Co-authored-by: jthomson04 <[email protected]> Co-authored-by: Xianjie Qiao <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Julien Debache <[email protected]> Co-authored-by: Yanchao Lu <[email protected]> Co-authored-by: Yiteng Niu <[email protected]> Co-authored-by: Daniel Stokes <[email protected]> Co-authored-by: bhsueh_NV <[email protected]> Co-authored-by: Bo Li <[email protected]> Co-authored-by: ChristinaZ <[email protected]> Co-authored-by: Larry <[email protected]> Co-authored-by: DylanChen-NV <[email protected]> Co-authored-by: Daniel Cámpora <[email protected]> Co-authored-by: davidclark-nv <[email protected]> Co-authored-by: Nikita Korobov <[email protected]> Co-authored-by: Yechan Kim <[email protected]> Co-authored-by: liji-nv <[email protected]> Co-authored-by: JieXin Liang <[email protected]> Co-authored-by: Enwei Zhu <[email protected]> Co-authored-by: xiweny <[email protected]> Co-authored-by: Yegor <[email protected]> Co-authored-by: Yukun He <[email protected]> Co-authored-by: Raayan Dhar <[email protected]> Co-authored-by: Dom Brown <[email protected]> Co-authored-by: Chang Liu <[email protected]> Co-authored-by: Pamela Peng <[email protected]> Co-authored-by: Iman Tabrizian <[email protected]> Co-authored-by: xavier-nvidia <[email protected]> Co-authored-by: jiahanc <[email protected]> Co-authored-by: Jhao-Ting Chen <[email protected]> Co-authored-by: Wanli Jiang <[email protected]> Co-authored-by: Erin <[email protected]> Co-authored-by: chenfeiz0326 <[email protected]> Co-authored-by: dongxuy04 <[email protected]> Co-authored-by: 2ez4bz <[email protected]> Co-authored-by: Hanjun Cho <[email protected]> Co-authored-by: Ubuntu <[email protected]> Co-authored-by: QI JUN <[email protected]> Co-authored-by: Aurelien Chartier <[email protected]> Co-authored-by: Anthony Chang <[email protected]> Co-authored-by: CarstyYou <[email protected]> Co-authored-by: Jinyang Yuan <[email protected]> Co-authored-by: narutolhy <[email protected]> Co-authored-by: Zhanrui Sun <[email protected]> Co-authored-by: wili <[email protected]> Co-authored-by: wili-65535 <[email protected]> Co-authored-by: Void <[email protected]> Co-authored-by: William Tambellini <[email protected]> Signed-off-by: Ransiki Zhang <[email protected]>

Signed-off-by: Yuxin <[email protected]> Signed-off-by: Tomer Asida <[email protected]> Signed-off-by: Yiqing Yan <[email protected]> Signed-off-by: qqiao <[email protected]> Signed-off-by: Fred Wei <[email protected]> Signed-off-by: Omer Ullman Argov <[email protected]> Signed-off-by: Netanel Haber <[email protected]> Signed-off-by: Robin Kobus <[email protected]> Signed-off-by: nv-guomingz <[email protected]> Signed-off-by: Rashid K <[email protected]> Signed-off-by: Zhenhuan Chen <[email protected]> Signed-off-by: Po-Wei Wang (Vincent) <[email protected]> Signed-off-by: Netanel Haber <[email protected]> Signed-off-by: Lucas Liebenwein <[email protected]> Signed-off-by: Frida Hou <[email protected]> Signed-off-by: Clay <[email protected]> Signed-off-by: Venky <[email protected]> Signed-off-by: Xin He (SW-GPU) <[email protected]> Signed-off-by: Superjomn <[email protected]> Signed-off-by: zhengd-nv <[email protected]> Signed-off-by: Yi Zhang <[email protected]> Signed-off-by: Kaiyu Xie <[email protected]> Signed-off-by: Frank Di Natale <[email protected]> Signed-off-by: Balaram Buddharaju <[email protected]> Signed-off-by: Linda-Stadter <[email protected]> Signed-off-by: Shunkang <[email protected]> Signed-off-by: Yuan Tong <[email protected]> Signed-off-by: Tailing Yuan <[email protected]> Signed-off-by: Faraz Khoubsirat <[email protected]> Signed-off-by: peaceh <[email protected]> Signed-off-by: ixlmar <[email protected]> Signed-off-by: Hui Gao <[email protected]> Signed-off-by: ShiXiaowei02 <[email protected]> Signed-off-by: Chuang Zhu <[email protected]> Signed-off-by: Stefan Niebler <[email protected]> Signed-off-by: jthomson04 <[email protected]> Signed-off-by: Xianjie <[email protected]> Signed-off-by: Xianjie Qiao <[email protected]> Signed-off-by: Julien Debache <[email protected]> Signed-off-by: Yanchao Lu <[email protected]> Signed-off-by: Yiteng Niu <[email protected]> Signed-off-by: Daniel Stokes <[email protected]> Signed-off-by: bhsueh <[email protected]> Signed-off-by: Bo Li <[email protected]> Signed-off-by: Christina Zhang <[email protected]> Signed-off-by: xinhe-nv <[email protected]> Signed-off-by: Dylan Chen <[email protected]> Signed-off-by: Daniel Campora <[email protected]> Signed-off-by: David Clark <[email protected]> Signed-off-by: yechank <[email protected]> Signed-off-by: Jin Li <[email protected]> Signed-off-by: JieXin Liang <[email protected]> Signed-off-by: Venky Ganesh <[email protected]> Signed-off-by: Enwei Zhu <[email protected]> Signed-off-by: Xiwen Yu <[email protected]> Signed-off-by: Yegor <[email protected]> Signed-off-by: Yegor Yershov <[email protected]> Signed-off-by: Yukun He <[email protected]> Signed-off-by: raayandhar <[email protected]> Signed-off-by: Dom Brown <[email protected]> Signed-off-by: Iman Tabrizian <[email protected]> Signed-off-by: xsimmons <[email protected]> Signed-off-by: jiahanc <[email protected]> Signed-off-by: Jhao-Ting Chen <[email protected]> Signed-off-by: Wanli Jiang <[email protected]> Signed-off-by: Erin Ho <[email protected]> Signed-off-by: Chenfei Zhang <[email protected]> Signed-off-by: Dongxu Yang <[email protected]> Signed-off-by: Hao Lu <[email protected]> Signed-off-by: William Zhang <[email protected]> Signed-off-by: Ubuntu <[email protected]> Signed-off-by: Hanjun Cho <[email protected]> Signed-off-by: junq <[email protected]> Signed-off-by: Aurelien Chartier <[email protected]> Signed-off-by: Anthony Chang <[email protected]> Signed-off-by: CarstyYou <[email protected]> Signed-off-by: Jinyang Yuan <[email protected]> Signed-off-by: narutolhy <[email protected]> Signed-off-by: ZhanruiSunCh <[email protected]> Signed-off-by: wili-65535 <[email protected]> Signed-off-by: Frank <[email protected]> Signed-off-by: Yilin Zhang <[email protected]> Signed-off-by: William Tambellini <[email protected]> Co-authored-by: tomeras91 <[email protected]> Co-authored-by: Yiqing Yan <[email protected]> Co-authored-by: Emma Qiao <[email protected]> Co-authored-by: WeiHaocheng <[email protected]> Co-authored-by: Omer Ullman Argov <[email protected]> Co-authored-by: Netanel Haber <[email protected]> Co-authored-by: Robin Kobus <[email protected]> Co-authored-by: nv-guomingz <[email protected]> Co-authored-by: Rashid Kaleem <[email protected]> Co-authored-by: Zhihan Jiang <[email protected]> Co-authored-by: Zhenhuan Chen <[email protected]> Co-authored-by: Po-Wei (Vincent) <[email protected]> Co-authored-by: Lucas Liebenwein <[email protected]> Co-authored-by: Neta Zmora <[email protected]> Co-authored-by: Fridah-nv <[email protected]> Co-authored-by: Clay <[email protected]> Co-authored-by: Venky <[email protected]> Co-authored-by: xinhe-nv <[email protected]> Co-authored-by: Yan Chunwei <[email protected]> Co-authored-by: Zheng Duan <[email protected]> Co-authored-by: Yi Zhang <[email protected]> Co-authored-by: Kaiyu Xie <[email protected]> Co-authored-by: Frank <[email protected]> Co-authored-by: brb-nv <[email protected]> Co-authored-by: Linda <[email protected]> Co-authored-by: Shunkangz <[email protected]> Co-authored-by: Yuan Tong <[email protected]> Co-authored-by: Tailing Yuan <[email protected]> Co-authored-by: Faraz <[email protected]> Co-authored-by: peaceh-nv <[email protected]> Co-authored-by: ixlmar <[email protected]> Co-authored-by: HuiGao-NV <[email protected]> Co-authored-by: Chuang Zhu <[email protected]> Co-authored-by: ShiXiaowei02 <[email protected]> Co-authored-by: Stefan Niebler <[email protected]> Co-authored-by: jthomson04 <[email protected]> Co-authored-by: Xianjie Qiao <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Julien Debache <[email protected]> Co-authored-by: Yanchao Lu <[email protected]> Co-authored-by: Yiteng Niu <[email protected]> Co-authored-by: Daniel Stokes <[email protected]> Co-authored-by: bhsueh_NV <[email protected]> Co-authored-by: Bo Li <[email protected]> Co-authored-by: ChristinaZ <[email protected]> Co-authored-by: Larry <[email protected]> Co-authored-by: DylanChen-NV <[email protected]> Co-authored-by: Daniel Cámpora <[email protected]> Co-authored-by: davidclark-nv <[email protected]> Co-authored-by: Nikita Korobov <[email protected]> Co-authored-by: Yechan Kim <[email protected]> Co-authored-by: liji-nv <[email protected]> Co-authored-by: JieXin Liang <[email protected]> Co-authored-by: Enwei Zhu <[email protected]> Co-authored-by: xiweny <[email protected]> Co-authored-by: Yegor <[email protected]> Co-authored-by: Yukun He <[email protected]> Co-authored-by: Raayan Dhar <[email protected]> Co-authored-by: Dom Brown <[email protected]> Co-authored-by: Chang Liu <[email protected]> Co-authored-by: Pamela Peng <[email protected]> Co-authored-by: Iman Tabrizian <[email protected]> Co-authored-by: xavier-nvidia <[email protected]> Co-authored-by: jiahanc <[email protected]> Co-authored-by: Jhao-Ting Chen <[email protected]> Co-authored-by: Wanli Jiang <[email protected]> Co-authored-by: Erin <[email protected]> Co-authored-by: chenfeiz0326 <[email protected]> Co-authored-by: dongxuy04 <[email protected]> Co-authored-by: 2ez4bz <[email protected]> Co-authored-by: Hanjun Cho <[email protected]> Co-authored-by: Ubuntu <[email protected]> Co-authored-by: QI JUN <[email protected]> Co-authored-by: Aurelien Chartier <[email protected]> Co-authored-by: Anthony Chang <[email protected]> Co-authored-by: CarstyYou <[email protected]> Co-authored-by: Jinyang Yuan <[email protected]> Co-authored-by: narutolhy <[email protected]> Co-authored-by: Zhanrui Sun <[email protected]> Co-authored-by: wili <[email protected]> Co-authored-by: wili-65535 <[email protected]> Co-authored-by: Void <[email protected]> Co-authored-by: William Tambellini <[email protected]> Signed-off-by: Lanyu Liao <[email protected]>

svc-trtllm-gh-bot added the Community want to contribute PRs initiated from Community label Jul 3, 2025

qsang-nv approved these changes Jul 8, 2025

View reviewed changes

PerkzZheng self-requested a review July 9, 2025 01:33

PerkzZheng reviewed Jul 9, 2025

View reviewed changes

cpp/kernels/fmha_v2/setup.py Show resolved Hide resolved

...tention/cubin/fmha_v2_flash_attention_bf16_64_128_S_q_paged_kv_192x128_tma_ws_sm90.cubin.cpp Show resolved Hide resolved

zhou-yuxin and others added 26 commits July 14, 2025 20:20

hopper-style context MLA

5e4ad8d

Signed-off-by: Yuxin <[email protected]>

[TRTLLM-4923][feat] Enable CUDA graphs for Nemotron-H (NVIDIA#5646)

82b4c72

Signed-off-by: Tomer Asida <[email protected]> Signed-off-by: Yuxin <[email protected]>

[Infra] - Fix test stage check for the package sanity check stage (NV…

6858c6f

…IDIA#5694) Signed-off-by: Yiqing Yan <[email protected]> Signed-off-by: Yuxin <[email protected]>

[Infra] - Waive a failed case on main (NVIDIA#5702)

538d431

Signed-off-by: qqiao <[email protected]> Signed-off-by: Yuxin <[email protected]>

fix: Set init value for moe expert id (NVIDIA#5660)

cccdae5

Signed-off-by: Fred Wei <[email protected]> Signed-off-by: Yuxin <[email protected]>

[ci] small multigpu speedups (NVIDIA#5643)

d21fffa

Signed-off-by: Omer Ullman Argov <[email protected]> Signed-off-by: Yuxin <[email protected]>

precommit

9a7a9b6

Signed-off-by: Yuxin <[email protected]>

1. fix bug in fmha_test.py that skips tests for hopper-style context …

727d937

…MLA; 2. fix setup.py that add cubin headers; 3. remove 192x128 cubins Signed-off-by: Yuxin <[email protected]>

1. update fmhaRunner;

3a19a9f

2. only generate SEPARATE_Q_K_V for sm90 192x128; 3. delete cubins which is not in (sm90 && head_size=128); Signed-off-by: Yuxin <[email protected]>

delete duplicate eagle3 and ngram tests (NVIDIA#5711)

d883fb2

Signed-off-by: Netanel Haber <[email protected]> Signed-off-by: Yuxin <[email protected]>

chore: Remove unused isFullContextRequest method (NVIDIA#5666)

86d538b

Signed-off-by: Robin Kobus <[email protected]> Signed-off-by: Yuxin <[email protected]>

chore: refine the default value by using pydantic default instead of … (

0f18869

NVIDIA#5695) Signed-off-by: nv-guomingz <[email protected]> Signed-off-by: Yuxin <[email protected]>

[ModelLoad] Concurrent load model (NVIDIA#5291)

e60fbfd

Signed-off-by: Rashid K <[email protected]> Co-authored-by: Zhihan Jiang <[email protected]> Signed-off-by: Yuxin <[email protected]>

[https://nvbugs/5365714] fix(scaffolding): use default LLM rather tha…

cca9f1f

…n trt backend LLM (NVIDIA#5705) Signed-off-by: Zhenhuan Chen <[email protected]> Signed-off-by: Yuxin <[email protected]>

[None][infra] Update the auto-community label action to be triggered …

0dad702

…every hour (NVIDIA#5658) Signed-off-by: Po-Wei Wang (Vincent) <[email protected]> Signed-off-by: Yuxin <[email protected]>

[AutoDeploy] merge feat/ad-2025-06-29 (NVIDIA#5737)

c8494b0

Signed-off-by: Lucas Liebenwein <[email protected]> Signed-off-by: Frida Hou <[email protected]> Co-authored-by: Neta Zmora <[email protected]> Co-authored-by: Fridah-nv <[email protected]> Signed-off-by: Yuxin <[email protected]>

feat: support more parameters in openai worker of scaffolding (NVIDIA…

c41a271

…#5115) Signed-off-by: Clay <[email protected]> Signed-off-by: Yuxin <[email protected]>

Waive tests : test_openai_lora, test_trtllm_serve_lora_example and te…

d401a66

…st_openai_chat_structural_tag_example (NVIDIA#5740) Signed-off-by: Venky <[email protected]> Signed-off-by: Yuxin <[email protected]>

tests: waive failures on main (NVIDIA#5704)

54d600b

Signed-off-by: Xin He (SW-GPU) <[email protected]> Signed-off-by: Yuxin <[email protected]>

fix [nvbug5351244]: test_mpi_session submit sync/async (NVIDIA#5608)

d5b05bf

Signed-off-by: Superjomn <[email protected]> Signed-off-by: Yuxin <[email protected]>

fix:https://nvbugs/5362398 (NVIDIA#5609)

642961f

Signed-off-by: nv-guomingz <[email protected]> Signed-off-by: Yuxin <[email protected]>

[nvbug 5300551] test: increase block count in eviction test (NVIDIA#5465

8db5fa5

) Signed-off-by: zhengd-nv <[email protected]> Signed-off-by: Yuxin <[email protected]>

test: add more tests for GB200 with 8 GPUs/2 nodes in L0 tests (NVIDI…

91cd65f

…A#5397) Signed-off-by: Yi Zhang <[email protected]> Signed-off-by: Yuxin <[email protected]>

doc: Fix outdated config in DeepSeek best perf practice doc (NVIDIA#5638

a6794d8

) Signed-off-by: Kaiyu Xie <[email protected]> Signed-off-by: Yuxin <[email protected]>

[https://nvbugspro.nvidia.com/bug/5351333][fix] Update to chunking ca…

ce0ca66

…lculation. (NVIDIA#5625) Signed-off-by: Frank Di Natale <[email protected]> Signed-off-by: Yuxin <[email protected]>

PerkzZheng removed the Community want to contribute PRs initiated from Community label Jul 17, 2025

Merge branch 'main' into context_MLA

b4f45a0

Merge branch 'main' into context_MLA

faebfb8

Merge branch 'main' into context_MLA

d41ad22

Merge branch 'main' into context_MLA

e558b70

Merge branch 'main' into context_MLA

1a77111

zhou-yuxin added 2 commits July 21, 2025 14:57

Merge branch 'main' into context_MLA

b163771

Merge branch 'main' into context_MLA

2c8cccf

PerkzZheng approved these changes Jul 23, 2025

View reviewed changes

PerkzZheng merged commit fca13b8 into NVIDIA:main Jul 23, 2025
3 checks passed

hopper-style context MLA #5713

hopper-style context MLA #5713

Uh oh!

Conversation

zhou-yuxin commented Jul 3, 2025

Uh oh!

PerkzZheng left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

PerkzZheng commented Jul 17, 2025

Uh oh!

tensorrt-cicd commented Jul 17, 2025

Uh oh!

tensorrt-cicd commented Jul 17, 2025

Uh oh!

coderabbitai bot commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Chat

Support

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

Documentation and Community

Uh oh!

zhou-yuxin commented Jul 17, 2025

Uh oh!

qsang-nv commented Jul 18, 2025

Uh oh!

tensorrt-cicd commented Jul 18, 2025

Uh oh!

tensorrt-cicd commented Jul 18, 2025

Uh oh!

zhou-yuxin commented Jul 21, 2025

Uh oh!

zhou-yuxin commented Jul 21, 2025

Uh oh!

tensorrt-cicd commented Jul 21, 2025

Uh oh!

tensorrt-cicd commented Jul 21, 2025

Uh oh!

tensorrt-cicd commented Jul 21, 2025

Uh oh!

tensorrt-cicd commented Jul 21, 2025

Uh oh!

zhou-yuxin commented Jul 21, 2025

Uh oh!

tensorrt-cicd commented Jul 21, 2025

Uh oh!

tensorrt-cicd commented Jul 21, 2025

Uh oh!

Uh oh!

Uh oh!

coderabbitai bot commented Jul 17, 2025 •

edited

Loading