[Bug]: Performance Regression in Acceptance length for EAGLE3

[tomasruizt]

Your current environment

The output of python collect_env.py

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.4 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version                : Could not collect
CMake version                : version 3.26.4
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.8.0+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0] (64-bit runtime)
Python platform              : Linux-5.15.0-153-generic-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.4.131
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration : GPU 0: NVIDIA H100
Nvidia driver version        : 535.261.03
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           52 bits physical, 57 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  96
On-line CPU(s) list:                     0-95
Vendor ID:                               AuthenticAMD
Model name:                              AMD EPYC 9254 24-Core Processor
CPU family:                              25
Model:                                   17
Thread(s) per core:                      2
Core(s) per socket:                      24
Socket(s):                               2
Stepping:                                1
Frequency boost:                         enabled
CPU max MHz:                             2900.0000
CPU min MHz:                             1500.0000
BogoMIPS:                                5800.48
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d
Virtualization:                          AMD-V
L1d cache:                               1.5 MiB (48 instances)
L1i cache:                               1.5 MiB (48 instances)
L2 cache:                                48 MiB (48 instances)
L3 cache:                                256 MiB (8 instances)
NUMA node(s):                            2
NUMA node0 CPU(s):                       0-23,48-71
NUMA node1 CPU(s):                       24-47,72-95
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB disabled; STIBP disabled; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsx async abort:           Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.2.8
[pip3] galore-torch==1.0
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.13.0
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-ml-py==12.570.86
[pip3] nvidia-nccl-cu12==2.27.3
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvshmem-cu12==3.3.9
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pynvml==12.0.0
[pip3] pytorch-triton==3.4.0+git11ec6354
[pip3] pyzmq==26.4.0
[pip3] torch==2.8.0
[pip3] torchaudio==2.8.0
[pip3] torchvision==0.23.0
[pip3] transformers==4.56.1
[pip3] triton==3.4.0+git663e04e8
[pip3] triton_kernels==1.0.0
[pip3] vector-quantize-pytorch==1.22.2
[conda] blas                      1.0                         mkl  
[conda] cuda-cudart               12.4.127                      0    nvidia
[conda] cuda-cupti                12.4.127                      0    nvidia
[conda] cuda-libraries            12.4.0                        0    nvidia
[conda] cuda-nvrtc                12.4.127                      0    nvidia
[conda] cuda-nvtx                 12.4.127                      0    nvidia
[conda] cuda-opencl               12.5.39                       0    nvidia
[conda] cuda-runtime              12.4.0                        0    nvidia
[conda] cuda-version              12.5                          3    nvidia
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] libcublas                 12.4.2.65                     0    nvidia
[conda] libcufft                  11.2.0.44                     0    nvidia
[conda] libcufile                 1.10.1.7                      0    nvidia
[conda] libcurand                 10.3.6.82                     0    nvidia
[conda] libcusolver               11.6.0.99                     0    nvidia
[conda] libcusparse               12.3.0.142                    0    nvidia
[conda] libjpeg-turbo             2.0.0                h9bf148f_0    pytorch
[conda] libnpp                    12.2.5.2                      0    nvidia
[conda] libnvfatbin               12.5.82                       0    nvidia
[conda] libnvjitlink              12.4.99                       0    nvidia
[conda] libnvjpeg                 12.3.1.89                     0    nvidia
[conda] mkl                       2023.1.0         h213fc3f_46344  
[conda] mkl-service               2.4.0           py311h5eee18b_1  
[conda] mkl_fft                   1.3.8           py311h5eee18b_0  
[conda] mkl_random                1.2.4           py311hdb19cb5_0  
[conda] numpy                     1.26.4          py311h08b1b3b_0  
[conda] numpy-base                1.26.4          py311hf175353_0  
[conda] optree                    0.12.1                   pypi_0    pypi
[conda] pytorch                   2.4.0           py3.11_cuda12.4_cudnn9.1.0_0    pytorch
[conda] pytorch-cuda              12.4                 hc786d27_6    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                2.4.0               py311_cu124    pytorch
[conda] torchelastic              0.2.2                    pypi_0    pypi
[conda] torchtriton               3.0.0                     py311    pytorch
[conda] torchvision               0.19.0              py311_cu124    pytorch
[conda] transformers              4.47.0                   pypi_0    pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.1.dev9651+gdaee8ec1e (git sha: daee8ec1e)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
        GPU0    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      SYS     SYS     SYS     SYS     PXB     NODE    26-35,74-83     1               N/A
NIC0    SYS      X      PIX     NODE    NODE    SYS     SYS
NIC1    SYS     PIX      X      NODE    NODE    SYS     SYS
NIC2    SYS     NODE    NODE     X      NODE    SYS     SYS
NIC3    SYS     NODE    NODE    NODE     X      SYS     SYS
NIC4    PXB     SYS     SYS     SYS     SYS      X      NODE
NIC5    NODE    SYS     SYS     SYS     SYS     NODE     X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
NCCL_VERSION=2.20.5-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_WORKER_MULTIPROC_METHOD=spawn
NVIDIA_PRODUCT_NAME=CUDA
CUDA_VERSION=12.4.0
PYTORCH_VERSION=2.4.0
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
VLLM_USE_V1=1
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

The acceptance length of EAGLE3 on main is lower than it used to be.

python examples/offline_inference/spec_decode.py \
        --model-dir meta-llama/Llama-3.1-8B-Instruct \
        --eagle-dir yuhuili/EAGLE3-LLaMA3.1-Instruct-8B \
        --method eagle3 \
        --num_spec_tokens 7 \
        --dataset-name hf \
        --dataset-path philschmid/mt-bench \
        --num_prompts 80 \
        --temp 0.0

According to previous measurements, like #17504 (comment), confirmed here: #24322

Previous AL=3.53

total_num_output_tokens: 16990
num_drafts: 4816
num_draft_tokens: 33712
num_accepted_tokens: 12208
mean acceptance length: 3.53
--------------------------------------------------
acceptance at token 0: 0.74
acceptance at token 1: 0.54
acceptance at token 2: 0.41
acceptance at token 3: 0.31
acceptance at token 4: 0.23
acceptance at token 5: 0.17
acceptance at token 6: 0.13

Current AL=3.40

--------------------------------------------------
total_num_output_tokens: 17028
num_drafts: 5010
num_draft_tokens: 35070
num_accepted_tokens: 12067
mean acceptance length: 3.41
--------------------------------------------------
acceptance at token 0: 0.74
acceptance at token 1: 0.53
acceptance at token 2: 0.38
acceptance at token 3: 0.28
acceptance at token 4: 0.21
acceptance at token 5: 0.15
acceptance at token 6: 0.12

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[tomasruizt]

A git bisection analysis showed two things:

1. the AL reduction to ~3.44 first appears in [Spec Decode] Efficient padded speculation #24539. The commit directly before it has higher AL (3.54) ([V0 Deprecation] Remove more V0 tests #25117).
2. the AL reduction is dependent on the flag disable_padded_drafter_batch
   • When disable_padded_drafter_batch = False then AL=3.44. This is the flag's default value.
   • When disable_padded_drafter_batch = True, then AL=3.55 (no regression).

@benchislett The different behavior based on the flag is interesting. Do you have an idea what could be causing it?

The git bisection information is here: git-bisection.md

Edit: The AL difference with the flag can be reproduced directly on main, too. It requires passing the flag in examples/offline_inference/spec_decode.py, like this:

speculative_config = {
    "method": args.method,
    "model": eagle_dir,
    "disable_padded_drafter_batch": True,  # or False
    "num_speculative_tokens": args.num_spec_tokens,
}

[tomasruizt]

I found potentially related bugfixes #26231 and #26198

[tomasruizt]

@benchislett The AL still differs on main (d9836d4) after the bugfix in #26231

Here are the respective log files main-True.log main-False.log

[benchislett]

Draft patch here: #26498
Still not sure why it happens, though