[MM Encoder] ViT attention performance and consolidation

[ywang96]

🚀 The feature, motivation and pitch

Today many vision transformers on vLLM leverage standard F.scaled_dot_product_attention to compute attention scores.

While there has been some effort in vision.py to help developers easily choose which backend to use, it would be great if vLLM can consolidate non-mask MHA implementations with different backends without caching so that developers can easily plug them in.

We should also investigate integrating FA3 for a few vision models we have and make sure there's no accuracy regression.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[SamitHuang]

should we consider different hardware backends? some attention implementations like FA3 are optimized for GPUs

[ywang96]

should we consider different hardware backends? some attention implementations like FA3 are optimized for GPUs

Totally - we already have a function that is kind of doing that, but only determines which backend to go with. The goal here is that ideally model developers only need to import a VisionAttention module and it'll automatically figure out which backend to go with by default depending on hardware, compute capability, head size, etc (but allow users to override via envvar).

This way the model developer won't need to worry about implementing attention computation themselves like what they currently have to (check Qwen2.5vl for example).

[tanruixiang]

Can I take it? Sorry, it looks like someone is already working on it.