[Bug]: Can't run vllm model because of the FlashAttention.

[qscqesze]

Your current environment

The output of `python collect_env.py`

Collecting environment information...
PyTorch version: 2.6.0+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.31.4
Libc version: glibc-2.35

Python version: 3.12.8 (main, Dec  4 2024, 08:54:12) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-162-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA H20
GPU 1: NVIDIA H20
GPU 2: NVIDIA H20
GPU 3: NVIDIA H20
GPU 4: NVIDIA H20
GPU 5: NVIDIA H20
GPU 6: NVIDIA H20
GPU 7: NVIDIA H20

Nvidia driver version: 535.161.08
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      52 bits physical, 57 bits virtual
Byte Order:                         Little Endian
CPU(s):                             180
On-line CPU(s) list:                0-179
Vendor ID:                          GenuineIntel
Model name:                         Intel(R) Xeon(R) Platinum 8457C
CPU family:                         6
Model:                              143
Thread(s) per core:                 2
Core(s) per socket:                 45
Socket(s):                          2
Stepping:                           8
BogoMIPS:                           5200.00
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512_bf16 wbnoinvd arat avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b md_clear arch_capabilities
Hypervisor vendor:                  KVM
Virtualization type:                full
L1d cache:                          4.2 MiB (90 instances)
L1i cache:                          2.8 MiB (90 instances)
L2 cache:                           180 MiB (90 instances)
L3 cache:                           195 MiB (2 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0-89
NUMA node1 CPU(s):                  90-179
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Unknown: No mitigations
Vulnerability Retbleed:             Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Mitigation; TSX disabled

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-ml-py==12.570.86
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.1
[pip3] torch==2.6.0
[pip3] torchaudio==2.6.0
[pip3] torchvision==0.21.0
[pip3] transformers==4.48.2
[pip3] triton==3.2.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.1.dev5307+g65c8274 (git sha: 65c8274
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    SYS     PIX     NODE    SYS     SYS     0-89    0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    SYS     PIX     NODE    SYS     SYS     0-89    0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    SYS     NODE    PIX     SYS     SYS     0-89    0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    SYS     NODE    PIX     SYS     SYS     0-89    0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     PIX     NODE    90-179  1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     PIX     NODE    90-179  1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     NODE    PIX     90-179  1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     NODE    PIX     90-179  1               N/A
NIC0    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS
NIC1    PIX     PIX     NODE    NODE    SYS     SYS     SYS     SYS     SYS      X      NODE    SYS     SYS
NIC2    NODE    NODE    PIX     PIX     SYS     SYS     SYS     SYS     SYS     NODE     X      SYS     SYS
NIC3    SYS     SYS     SYS     SYS     PIX     PIX     NODE    NODE    SYS     SYS     SYS      X      NODE
NIC4    SYS     SYS     SYS     SYS     NODE    NODE    PIX     PIX     SYS     SYS     SYS     NODE     X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4

NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
NCCL_VERSION=2.17.1-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
VLLM_USAGE_SOURCE=production-docker-image
NVIDIA_CUDA_END_OF_LIFE=1
CUDA_VERSION=12.1.0
VLLM_FLASH_ATTN_VERSION=3
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

Description

The code in #13454 hasn't been merged for nearly a week. As a result, the code that could run previously can no longer work properly. However, if I roll back the code to the state it was in a week ago, everything runs fine again.

Error Logs

Here are some of the error messages I'm getting:

ERROR 03-20 09:51:26 [core.py:340]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
ERROR 03-20 09:51:26 [core.py:340]     return self._call_impl(*args, **kwargs)
ERROR 03-20 09:51:26 [core.py:340]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-20 09:51:26 [core.py:340]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
ERROR 03-20 09:51:26 [core.py:340]     return forward_call(*args, **kwargs)
ERROR 03-20 09:51:26 [core.py:340]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-20 09:51:26 [core.py:340]   File "/data4/qinggangying/qingjun/vllm/vllm/model_executor/models/minimax_text_01.py", line 1031, in forward
ERROR 03-20 09:51:26 [core.py:340]     hidden_states = self.model(input_ids, positions, self.kv_cache,
ERROR 03-20 09:51:26 [core.py:340]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-20 09:51:26 [core.py:340]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
ERROR 03-20 09:51:26 [core.py:340]     return self._call_impl(*args, **kwargs)
ERROR 03-20 09:51:26 [core.py:340]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-20 09:51:26 [core.py:340]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
ERROR 03-20 09:51:26 [core.py:340]     return forward_call(*args, **kwargs)
ERROR 03-20 09:51:26 [core.py:340]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-20 09:51:26 [core.py:340]   File "/data4/qinggangying/qingjun/vllm/vllm/model_executor/models/minimax_text_01.py", line 921, in forward
ERROR 03-20 09:51:26 [core.py:340]     if attn_metadata.num_prefills > 0:
ERROR 03-20 09:51:26 [core.py:340]        ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-20 09:51:26 [core.py:340] AttributeError: 'FlashAttentionMetadata' object has no attribute 'num_prefills'

ERROR 03-20 08:32:16 [core.py:340]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1123, in __call__
ERROR 03-20 08:32:16 [core.py:340]     return self._op(*args, **(kwargs or {}))
ERROR 03-20 08:32:16 [core.py:340]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-20 08:32:16 [core.py:340]   File "/data4/qinggangying/qingjun/vllm/vllm/attention/layer.py", line 368, in unified_attention_with_output
ERROR 03-20 08:32:16 [core.py:340]     self.impl.forward(self,
ERROR 03-20 08:32:16 [core.py:340]   File "/data4/qinggangying/qingjun/vllm/vllm/v1/attention/backends/flash_attn.py", line 281, in forward
ERROR 03-20 08:32:16 [core.py:340]     flash_attn_varlen_func(
ERROR 03-20 08:32:16 [core.py:340] TypeError: flash_attn_varlen_func() got an unexpected keyword argument 'q_descale'

There are many more error messages like these.

Investigation

The main problem seems to be related to FlashAttentionMetadata. The recent commits related to FlashAttentionMetadata can be found here: https://github.com/vllm-project/vllm/pull/14570/files.

This is also exactly where the errors occur.

I'm not sure how to fix this code at the moment. I suspect that supporting Flash_attn3 is required to support the corresponding parameters. If I try to comment out some code or do defensive programming, more issues will arise.

There are my run model code:

python3 -m vllm.entrypoints.api_server \
--model /minimax-algeng-rw/minimax-algeng/qingangying/download/MiniMax-Text-01 \
--tensor-parallel-size 8 \
--trust-remote-code \
--quantization experts_int8  \
--max_model_len 4096 \
--dtype bfloat16

Please help me resolve this problem. Thanks!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[pooyadavoodi]

Seems to be caused by #14570

[pooyadavoodi]

@LucasWilkinson @mickaelseznec Do you know what's going on?

[LucasWilkinson]

Looks like this has been resolved by 09d044b ?

[qscqesze]

Looks like this has been resolved by 09d044b ?

No.It's only solve some problem. It still wrong like:

ERROR 03-20 08:32:16 [core.py:340]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1123, in __call__
ERROR 03-20 08:32:16 [core.py:340]     return self._op(*args, **(kwargs or {}))
ERROR 03-20 08:32:16 [core.py:340]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-20 08:32:16 [core.py:340]   File "/data4/qinggangying/qingjun/vllm/vllm/attention/layer.py", line 368, in unified_attention_with_output
ERROR 03-20 08:32:16 [core.py:340]     self.impl.forward(self,
ERROR 03-20 08:32:16 [core.py:340]   File "/data4/qinggangying/qingjun/vllm/vllm/v1/attention/backends/flash_attn.py", line 281, in forward
ERROR 03-20 08:32:16 [core.py:340]     flash_attn_varlen_func(
ERROR 03-20 08:32:16 [core.py:340] TypeError: flash_attn_varlen_func() got an unexpected keyword argument 'q_descale'

[qscqesze]

Hello. This issue has been resolved. We specified the corresponding attention backend by using the environment variable

export VLLM_USE_V1=0
export VLLM_ATTENTION_BACKEND=XFORMERS

which helped us avoid this problem.

[tdoublep]

I was seeing the same error in my dev environment:

ERROR 03-20 08:32:16 [core.py:340] TypeError: flash_attn_varlen_func() got an unexpected keyword argument 'q_descale'

I solved it by recompiling the C parts of vLLM -- looks it pulls in and recompiles a newer version of FA.

[llsj14]

I'm experiencing the same issue, while testing vLLM v0.8.4.
I tried versions of flash_attn as 2.0.4, 2.6.1, 2.7.4.post1, but still get the same error.

Here's the install script I used.

pip uninstall transformer-engine    # to avoid conflict with flash_attn version
pip install -r ./requirements/build.txt
pip install -r ./requirements/common.txt
pip install -r ./requirements/cuda.txt
pip install flash_attn==2.7.4.post1
export VLLM_COMMIT=61c7a1b856e32ef8b12c70abcb9fd9ad22619a13
export VLLM_PRECOMPILED_WHEEL_LOCATION=https://wheels.vllm.ai/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
pip install -e .
pip install -U pynvml

@tdoublep Could you share your solution in more detail?

[mickaelseznec]

You're probably using a uncompatible wheel. Please try to reinstall vllm from source, this should hopefully fix your problem

[llsj14]

@mickaelseznec Thank you.
I changed wheel version and resolved the problem.

pip install -r ./requirements/build.txt
pip install -r ./requirements/common.txt
pip install -r ./requirements/cuda.txt
pip install flash_attn==2.7.4.post1
export VLLM_COMMIT=dc1b4a6f1300003ae27f033afbdff5e2683721ce # last commit id of v0.8.4
export VLLM_PRECOMPILED_WHEEL_LOCATION=https://wheels.vllm.ai/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
pip install -e .
pip install -U pynvml

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!