[Performance]: Diagnose speed difference using spec decode branch

[tomasruizt]

Report of performance regression

@litone01 reported a performance regression on the branch of spec decode (#24322), where models run slower than on main even without using spec decode.

His branch: https://github.com/litone01/vllm/tree/origin/feature/spec-decode-draft-model-debug

This issue is for tracking progress.

[tomasruizt]

Edit: I reran the benchmarks after setting output_len to 576 manually, and updated the .zip file accordingly.

• I used Qwen3-4B to run on a local GPU
• I found all three branches to have almost the same runtime. The small difference of ~0.05s is likely unrelated to the issue you reported @litone01.

branch	runtime [s]
main 15.11.25 (be263f7)	6.83s
litone1:feature/spec-decode-draft-model-debug	6.89s
tomasruizt:feature/spec-decode-draft-model	6.83s

All logs and results can be downloaded here: berkeley-ppl-sd.zip

[litone01]

Hi @tomasruizt, thanks for the update. I noticed that for main and tomasruizt:feature/spec-decode-draft-model, the "results-run1.txt" says "Total generated tokens: 200 ", which is a fixed limit set by the benchmark datasets. In theory, the request should stop at EOS, but using vllm bench serve, it is not working as intended for now.

A temporary fix is to set it to 576 mannually, as done on my branch L1327 at vllm/benchmarks/serve.py. 576 is the output length for that request based on my previous profiled result.

Can you try running the request again and allow it to generate up to 576 tokens? Thanks!

[tomasruizt]

You are absolutely right @litone01. I reran the benchmarks and updated the .zip file. Now I don't find any difference in runtime (or very small) between all 3 branches 🤔. Could you confirm if you observe speed differences using Qwen3-4b?

Note: After switching branches, I often have to reinstall vLLM. I use the command below, where the --prerelease flag can often be skipped.

VLLM_USE_PRECOMPILED=1 uv pip install -e ".[bench]" --prerelease=allow

[litone01]

I think there is some issue with my enviroment built, rebuilding it seems to solve the problem.

[tomasruizt]

@litone01 can you confirm if the issue is resolved?

[litone01]

Hi @tomasruizt yep, thanks for your help!