[Performance]: EAGLE-3: Discrepancy Between Throughput and Acceptance Length Improvements

[singh-git10]

Proposal to improve performance

We performed the benchmarking using both the official LLaMA 3 70B EAGLE-2 head (yuhuili/EAGLE-LLaMA3-Instruct-70B) and the LLaMA 3.3 70B EAGLE-3 head (yuhuili/EAGLE3-LLaMA3.3-Instruct-70B). In the case of the 70B EAGLE-3 head, despite a ~50% reduction in total model parameters, the OTPS (Output Tokens Per Second) gain is roughly proportional to the acceptance length gain. I believe there are still gaps in the current EAGLE-3 implementation that, if addressed, could lead to further performance improvements.

Report of performance regression

No response

Misc discussion on performance

No response

Your current environment (if you think it is necessary)

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ekagra-ranjan]

Oh, this is fine. Running the draft model is not the biggest overhead, its the verification step: which is going to be same. Earlier it was said that there is no improvement on tops of eagle 3 over eagle 2 which was concering:

#19033 (comment)

[Neo9061]

Thanks @ekagra-ranjan regarding on my comment, I have trained a EAGLE-3 which has ~7% tau improvement over EAGLE-2. But when benchmarking, the OTPS gain becomes worse, on HumanEval dataset with llmperf. The EAGLE-3 I trained does not have reduced vocabulary size and hidden dimension size, as the official released one.

Do you see any potential issue on high drafting latency making the perf gain diminished? For example, will non-parallel linear layer of fusing hidden states from three layers can create a very high drafting latency? (line. Many thanks!

[ekagra-ranjan]

Sorry i dont get you. You two are working together? One comment says there is 26% gain and other says there is degradation. What's the diff bw these 2 setups? Also, what's the exact command used to measure speedup?

I see some data on 1st comment. Could you pls share as much detail as possible as what model, what dataset, what AL is seen on the datasets for which method and their otps fir which vllm version n llmperf version?

This would help in resolving the confusion.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!