[Feature] support mistral small vlm

[zhyncs]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

https://mistral.ai/fr/news/mistral-small-3-1

Related resources

No response

[draqos]

hello everyone, could i help as a contributor with this issue? if yes, any tips on where to start ?

[rjmehta1993]

is this supported yet? @zhyncs

[FlamingoPg]

I think it not supported yet, maybe you can refer: https://docs.sglang.ai/references/supported_models.html#how-to-support-a-new-vlm

[Nagi-ovo]

@zhaochenyang20 I would like to work on this issue!

[zhaochenyang20]

@zhaochenyang20 I would like to work on this issue!

This should be a proposal, like what's on your plan to support it.

[KivenChen]

M3.1-S looks solid for my use case — mainly need an mmlm that runs on low-spec hw.

That said, I'd mention a few notes:

• it's basically a Mistral text model + Pixtral vision tower. We’ll likely need to add pixtral support, start by bringing in SGL-optimized layer ops and so. [Feature] Support mistralai/Pixtral #2351
• MM data layer seems under heavy dev recently — any doc or example on what guidelines to follow when handling multimodal input for a new model? or any slack threads?
• (update) as I suspect, we may need alignments between mistral vs HF formats. [Feature]: Mistral Small 3.1 HF support vllm-project/vllm#15212

[zhaochenyang20]

M3.1-S looks solid for my use case — mainly need an mmlm that runs on low-spec hw.

That said, I'd mention a few notes:

• it's basically a Mistral text model + Pixtral vision tower. We’ll likely need to add pixtral support, start by bringing in SGL-optimized layer ops and so.
• MM data layer seems under heavy dev recently — any doc or example on what guidelines to follow when handling multimodal input for a new model? or any slack threads?
• (update) as I suspect, we may need alignments between mistral vs HF formats. [Feature]: Mistral Small 3.1 HF support vllm-project/vllm#15212

thanks!

[Nagi-ovo]

As #4518 (comment) matches my preliminary investigation findings. Based on this, our implementation roadmap should be:

1. Pixtral Integration: [Feature] Support mistralai/Pixtral #2351, which can be implemented with reference to Add support for Pixtral huggingface/transformers#33449

2. Mistral Small 3.1 Integration, in relation to [Model] Support Mistral3 in the HF Transformers format vllm-project/vllm#15505 & Release Mistral 3 (Based on v4.49.0))

Current Status and Implementation Plan:

1. Status Quo: sglang already supports Mistral Small 3, but lacks support for the vision tower (Pixtral model) in Mistral Small 3.1.

2. Implementation Strategy: First implement Pixtral support, following sglang's documentation for new VLM integration, building upon existing components like LlavaConfig.

3. Expected Outcome: Full support for Pixtral, followed by complete integration with Mistral Small 3.1.

4. Uncertainties: Given my technical limitations, I welcome any suggestions and collaboration opportunities. Feel free to reach out if interested in contributing.

5. Timeline: Estimated 1 week for Pixtral implementation, followed by swift integration of Mistral Small 3.1.

@zhaochenyang20

[zhaochenyang20]

As #4518 (comment) matches my preliminary investigation findings. Based on this, our implementation roadmap should be:

1. Pixtral Integration: [Feature] Support mistralai/Pixtral #2351, which can be implemented with reference to Add support for Pixtral huggingface/transformers#33449
2. Mistral Small 3.1 Integration, in relation to [Model] Support Mistral3 in the HF Transformers format vllm-project/vllm#15505 & Release Mistral 3 (Based on v4.49.0))

Current Status and Implementation Plan:

1. Status Quo: sglang already supports Mistral Small 3, but lacks support for the vision tower (Pixtral model) in Mistral Small 3.1.
2. Implementation Strategy: First implement Pixtral support, following sglang's documentation for new VLM integration, building upon existing components like LlavaConfig.
3. Expected Outcome: Full support for Pixtral, followed by complete integration with Mistral Small 3.1.
4. Uncertainties: Given my technical limitations, I welcome any suggestions and collaboration opportunities. Feel free to reach out if interested in contributing.
5. Timeline: Estimated 1 week for Pixtral implementation, followed by swift integration of Mistral Small 3.1.

@zhaochenyang20

great! go for it