[Feature]: Inflight BNB quantization for Mixtral models

[tristanleclercq]

🚀 The feature, motivation and pitch

I would like to be able to use in-flight BNB quantization of Mixtral models like mistralai/Mixtral-8x7B-Instruct-v0.1 with --quantization bitsandbytes. It currently doesn't work on vllm 0.8.4.

Alternatives

No response

Additional context

Since #2673 a hacky workaround has been implemented to be able to use quantization with Mixtral models here

bitsandbytes is not part of the list of mixtral-supported quantization methods, so vllm falls back to the QuantMixtralForCausalLM (MixtralForCausalLM from mixtral_quant.py) implementation which doesn't rely on fused moe. But weights loading with BitsAndBytes later fails here with

AttributeError: Model MixtralForCausalLM does not support BitsAndBytes quantization yet. No 'packed_modules_mapping' found.

MixtralForCausalLM from mixtral_quant.py does indeed not have the packed_modules_mapping attribute.

I tried two things:

• adding packed_modules_mapping attribute similar to those of MixtralForCausalLM class in QuantMixtralForCausalLM => it now fails here, I'm not sure why
• adding bitsandbytes to the list of mixtral-supported quantization methods => it fails here because FusedMoE is not a LinearBase layer

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[tristanleclercq]

After this PR, a possible workaround should be to use model_impl="transformers".

I'm even wondering if falling back to the transformers implementation wouldn't be an acceptable replacement for the QuantMixtralForCausalLM hack altogether (not sure about the potential impact on performance).

[tristanleclercq]

Related issues (same cause for all, ie "MoE models don't support bitsandbytes quantization in vllm, whether it's pre-quantized or inflight quantization")

• [Bug]: sm75 can not serve qwen3 bnb 4bit model #17337 (about Qwen3-30B-A3B model)
• [Bug]: Loading bnb-community/Llama-4-Scout-17B-16E-Instruct-bnb-4bit error FusedMoE quant_method is None #16713 (about Llama-4-Scout-17B-16E-Instruct model)
• [Bug]: assert self.quant_method is not None error when using llama scout 4bit #16121 (about Llama-4-Scout-17B-16E-Instruct model)

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!