[Kernels][DP/EP] Optimize Silu Kernel for R1

[elvircrn]

Purpose

The purpose of this PR is to replace the triton silu implementation with a faster cuda version.

Here's a benchmark slice:

This was achieved by launching additional cuda blocks and parallelizing over T dimension. This means that we end up launching NOOP threads and that the parallelization factor is now an additional tunable parameter.

To understand the impact of the parallelization factor, see the following graphs for E <=9 :

For E=32, we have:

16X seems like a parallelization that works well for most configuration - this was chosen as the default.

Test Plan

Given y of shape (E, T, 2H) as input the function is expected to work for all GROUP_SIZE=128 and H divisible by 128.

Test Result

VLLM_ALL2ALL_BACKEND="deepep_low_latency" VLLM_USE_DEEP_GEMM=1 g2 lm_eval --model vllm --model_args pretrained=Qwen/Qwen3-30B-A3B-Instruct-2507-FP8,data_parallel_size=2,enable_expert_parallel=True --trust_remote_code --tasks gsm8k --num_fewshot 5 --batch_size auto

vllm (pretrained=Qwen/Qwen3-30B-A3B-Instruct-2507-FP8,data_parallel_size=2,enable_expert_parallel=True,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8848|±  |0.0088|
|     |       |strict-match    |     5|exact_match|↑  |0.8711|±  |0.0092|

From main:

vllm (pretrained=Qwen/Qwen3-30B-A3B-Instruct-2507-FP8,data_parallel_size=2,enable_expert_parallel=True,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8848|±  |0.0088|
|     |       |strict-match    |     5|exact_match|↑  |0.8711|±  |0.0092|

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces a new CUDA kernel for silu_mul_fp8_quant to replace the existing Triton implementation, aiming for better performance. The changes include the new CUDA kernel, its C++ bindings, and updates to tests and benchmarks to compare against the old implementation, which is preserved as a baseline. While the overall approach is sound, the review identified several critical correctness issues in the new CUDA kernel related to parameter handling, as well as high-severity maintainability problems such as dead code, code duplication, and style violations. These issues should be addressed to ensure the correctness and long-term health of the codebase.

[gemini-code-assist]

The kernel uses a hardcoded EPS value, ignoring the eps parameter passed to the host function silu_mul_fp8_quant_deep_gemm_cuda. This is a correctness bug. The eps value should be passed to this kernel as an argument and used instead of the hardcoded constant.

This will require changes in:

1. The kernel signature to accept eps.
2. The kernel body to use the eps argument (e.g., float y_max = eps; on line 265).
3. The host launcher silu_mul_fp8_quant_deep_gemm_cuda to pass eps to the kernel.

[elvircrn]

@LucasWilkinson Is this OK?

[elvircrn]

Maybe we should remove eps/fp8_max/fp8_min from the python API?

[robertgshaw2-redhat]

• I got the following:

(APIServer pid=1) (EngineCore_0 pid=283) RuntimeError: Worker failed with error ''Keyword argument NUM_WARPS was specified but unrecognised'', please check the stack trace above for the root cause

• root cause higher up in the stack trace

(APIServer pid=1) (EngineCore_7 pid=304) (VllmWorker pid=368) ERROR 09-03 01:11:27 [multiproc_executor.py:611]     a2q, a2q_scale = silu_mul_fp8_quant_deep_gemm(workspace1,
(APIServer pid=1) (EngineCore_7 pid=304) (VllmWorker pid=368) ERROR 09-03 01:11:27 [multiproc_executor.py:611]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) (EngineCore_7 pid=304) (VllmWorker pid=368) ERROR 09-03 01:11:27 [multiproc_executor.py:611]   File "/opt/vllm-source/vllm/model_executor/layers/fused_moe/batched_deep_gemm_moe.py", line 206, in silu_mul_fp8_quant_deep_gemm
(APIServer pid=1) (EngineCore_7 pid=304) (VllmWorker pid=368) ERROR 09-03 01:11:27 [multiproc_executor.py:611]     _silu_mul_fp8_quant_deep_gemm[grid](
(APIServer pid=1) (EngineCore_7 pid=304) (VllmWorker pid=368) ERROR 09-03 01:11:27 [multiproc_executor.py:611]   File "/opt/vllm/lib64/python3.12/site-packages/triton/runtime/jit.py", line 390, in <lambda>
(APIServer pid=1) (EngineCore_7 pid=304) (VllmWorker pid=368) ERROR 09-03 01:11:27 [multiproc_executor.py:611]     return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
(APIServer pid=1) (EngineCore_7 pid=304) (VllmWorker pid=368) ERROR 09-03 01:11:27 [multiproc_executor.py:611]                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

It looks like the triton kernel is still hooked up to the actual layer.

[robertgshaw2-redhat]

made branch on nm repo that hooks up the kernel: 376129e

I still have an IMA:

RuntimeError: CUDA driver error: an illegal memory access was encountered

Im trying to build it locally

[LucasWilkinson]

made branch on nm repo that hooks up the kernel: 376129e

I still have an IMA:

RuntimeError: CUDA driver error: an illegal memory access was encountered

Im trying to build it locally

@elvircrn would a coredump help? https://blog.vllm.ai/2025/08/11/cuda-debugging.html @robertgshaw2-redhat would you be able to enable this environment variables?

[elvircrn]

@LucasWilkinson Yes, a coredump would help, thanks!

[elvircrn]

@robertgshaw2-redhat @LucasWilkinson Can H ever not be divisible by 128?

[elvircrn]

/gemini-review

[gemini-code-assist]

I've received your request for a new review. Please note that this is a command for the system to initiate a new review, and I will not be providing a conversational response here. The review process will begin shortly.

[elvircrn]

@LucasWilkinson @tlrmchlsmth

1. Blackwell failing test seems to be resolved with: [CI Failure] Fix test_flashinfer_cutlass_mxfp4_mxfp8_fused_moe #24750 but fails on main
2. Neuron soft failing with: bash: .buildkite/scripts/hardware_ci/run-neuron-test.sh: No such file or directory also on main.
3. TPU tests also failing on main

Used this for reference a5b84f1

[tlrmchlsmth]

@LucasWilkinson @tlrmchlsmth

1. Blackwell failing test seems to be resolved with: [CI Failure] Fix test_flashinfer_cutlass_mxfp4_mxfp8_fused_moe #24750 but fails on main
2. Neuron soft failing with: bash: .buildkite/scripts/hardware_ci/run-neuron-test.sh: No such file or directory also on main.
3. TPU tests also failing on main

Used this for reference a5b84f1

sounds good - let's wait for distributed-tests-2-gpus and entrypoints-integration-test-api-server to finish and then we can request a force merge

[elvircrn]

@tlrmchlsmth @LucasWilkinson CI is done.