[Benchmark][V1][Spec Decode][EAGLE] Tracking benchmark for V1 EAGLE

[ekagra-ranjan]

We have been doing perf bench on MTBench so that e2e speedup and AL are comparable with other setups and academic papers. Thanks to @luyuzhe111 and others for the discussion and helping with measuring the gaps!

llama 3 8b

During model wt loading

• [V1][Spec Decode] Eagle Model loading #16035 (comment)
  • SGL correction: [V1][Spec Decode] Eagle Model loading #16035 (comment)
  • SGL setup: https://docs.google.com/document/d/18ETJLsnxR88Qq3VDk5Mq-Hb7vuE9o3VNZ-hhz-OqAXk/edit?usp=sharing
• [V1][Spec Decode] Eagle Model loading #16035 (comment)

During KV Cache slot

• [V1][Spec Decode] KV cache slots for eagle heads #16370 (comment)

llama 3.1 8b

• [V1][Spec Decode] KV cache slots for eagle heads #16370 (comment)
• EAGLE - 1/3
  • offline serving:
    • [V1][Spec Decode] EAGLE-3 Support #16937 (comment)
    • E1 Spec Bench AL: [Spec Decode][Benchmark] Add Spec Bench Dataset for benchmarking #23563
  • online serving:
    • [Benchmark] Add single turn MTBench to Serving Bench #17202 (comment)
    • EAGLE 3: Fix preamble so that measured speedup over Eagle 1 becomes 32% instead of 5% on MTBench #25916 - using chat endpoint correctly
• ngram-eagle: [Spec Decode][Hybrid] Add ngram-eagle SD method #24344

torch compile & CUDA graph:

• EAGLE 1
  • [V1][Spec Decode] Apply torch.compile & cudagraph to EAGLE #17211 (comment)
  • [V1][Spec Decode] Apply torch.compile & cudagraph to EAGLE #17211 (comment)
• EAGLE 3
  • [V1][Spec Decode] Apply torch.compile & cudagraph to EAGLE3 #17504 (comment)

llama 4

• EAGLE 1
  • LLM: [Meta] Llama4 EAGLE Support #20591
  • MM: [Meta] Official Eagle mm support, first enablement on llama4 #20788

[wwl2755]

Great job and thanks on collecting these!

Do you have a unified table/doc that keeps all the up-to-date benchmarking (and also the gaps with ideal condition), which I believe would be much helpful?

[wwl2755]

And also, making the metrics-related PRs (#16367 and #17010) finialized and merged would be great.

[ekagra-ranjan]

Do you have a unified table/doc that keeps all the up-to-date benchmarking (and also the gaps with ideal condition), which I believe would be much helpful?

I didnt get the time to do it. I believe the most recent comment would be the one to refer.

[ekagra-ranjan]

I did an investigation on how much AL we can reproduce for EAGLE-1. Sharing here for posterity the results of llama 2 7b Eagle 1 and comparison with published numbers on MTBench

Summary:

On MTBench with K=5 for llama 2 7b chat

• vLLM without preamble: 2.6
• vLLM with preamble: 2.82
• Official paper: 3.00

This means that we can reproduce 94% of the reported AL of by the authors, i.e., (2.82/3.0)

Details:

target model: meta-llama/Llama-2-7b-chat-hf
eagle draft: yuhuili/EAGLE-llama2-chat-7B

replace the draft and target model name in examples/offline_inference/spec_decode.py and run following
time VLLM_USE_HYBRID_MEM=0 VLLM_USE_V1=1 python3 examples/offline_inference/spec_decode.py --method eagle --num_spec_tokens 5 --max_num_seqs 64 --tp 1 --draft_tp 1 --dataset-name hf --dataset-path philschmid/mt-bench --num-prompts 80

E1 llama 2 7b chat - no preamble
--------------------------------------------------           
mean acceptance length: 2.60                 
--------------------------------------------------
acceptance at token 0:0.67               
acceptance at token 1:0.41             
acceptance at token 2:0.25             
acceptance at token 3:0.16             
acceptance at token 4:0.11  

E1 llama 2 7b chat - llama 2 preamble
--------------------------------------------------
mean acceptance length: 2.82
--------------------------------------------------
acceptance at token 0:0.72
acceptance at token 1:0.48
acceptance at token 2:0.30
acceptance at token 3:0.20
acceptance at token 4:0.13

For reproducing the numbers with preamble, add the below here
The prompt was obtained from a previous commit of the official EAGLE repo here

[
                    {
                        "role": "system", 
                        "content": "You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\n\nIf a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information."
                    },
                    {
                        "role": "user",
                        "content": prompt
                    }
                ],