[Model] Enable encoder DP for MiniCPM-V

[ZJY0516]

Purpose

Enable encoder DP for MiniCPM-V

Related to: #22743

Test Plan

MiniCPM-V-2_6

vllm serve ./hf/MiniCPM-V-2_6/ -tp 2 --trust-remote-code --mm_encoder_tp_mode data

vllm bench serve \
--endpoint-type openai-chat \
--endpoint /v1/chat/completions \
--model "./hf/MiniCPM-V-2_6/" \
--dataset-name random-mm \
--random-mm-base-items-per-request 3 \
--tokenizer ~/hf/MiniCPM-V-2_6/ --trust-remote-code \
--max-concurrency 1 --num-prompts 50

Test Result

encoder dp
============ Serving Benchmark Result ============
Successful requests:                     50        
Maximum request concurrency:             1         
Benchmark duration (s):                  92.09     
Total input tokens:                      51034     
Total generated tokens:                  3652      
Request throughput (req/s):              0.54      
Output token throughput (tok/s):         39.66     
Total Token throughput (tok/s):          593.85    
---------------Time to First Token----------------
Mean TTFT (ms):                          987.47    
Median TTFT (ms):                        955.18    
P99 TTFT (ms):                           1295.46   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          11.86     
Median TPOT (ms):                        11.86     
P99 TPOT (ms):                           11.96     
---------------Inter-token Latency----------------
Mean ITL (ms):                           11.69     
Median ITL (ms):                         11.83     
P99 ITL (ms):                            12.59     
==================================================

default
============ Serving Benchmark Result ============
Successful requests:                     50        
Maximum request concurrency:             1         
Benchmark duration (s):                  95.14     
Total input tokens:                      51034     
Total generated tokens:                  3730      
Request throughput (req/s):              0.53      
Output token throughput (tok/s):         39.21     
Total Token throughput (tok/s):          575.64    
---------------Time to First Token----------------
Mean TTFT (ms):                          1025.98   
Median TTFT (ms):                        1021.61   
P99 TTFT (ms):                           1637.95   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          11.90     
Median TPOT (ms):                        11.87     
P99 TPOT (ms):                           12.15     
---------------Inter-token Latency----------------
Mean ITL (ms):                           11.74     
Median ITL (ms):                         11.85     
P99 ITL (ms):                            12.62     
==================================================

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request correctly enables encoder data parallelism for MiniCPM-V models by passing a use_data_parallel flag to the Idefics2VisionTransformer. However, there are two key issues. First, the supports_encoder_tp_data flag is only set for MiniCPMV4_0, which means this feature will likely not be enabled for versions 2.5, 2.6, and 4.5, despite code changes suggesting they should support it. This is a correctness issue. Second, as detailed in a specific comment, there is significant code duplication in init_vision_module across multiple classes, which impacts maintainability.

[gemini-code-assist]

This init_vision_module implementation is nearly identical across MiniCPMV2_5, MiniCPMV2_6, MiniCPMV4_0, and MiniCPMV4_5. The only significant difference is the conditional logic for quant_config in the v4.x models. This duplication increases maintenance effort. Consider refactoring this into a shared method in a base class to improve code reuse and maintainability.

[ZJY0516]

@DarkLight1337 Could you please take a look at this pr?

[DarkLight1337]

Thanks, can you move the flag introduced by #23325 from the V4 model to the base model so that this is actually enabled for the other models?

[ZJY0516]

Thanks, can you move the flag introduced by #23325 from the V4 model to the base model so that this is actually enabled for the other models?

Done