[VLM] Calculate maximum number of multi-modal tokens by model #6121
Conversation
| from vllm.multimodal import MULTIMODAL_REGISTRY | ||
| @MULTIMODAL_REGISTRY.register_image_input_mapper() | ||
| + @MULTIMODAL_REGISTRY.register_max_image_tokens(<your_calculation>) |
There was a problem hiding this comment.
IMO the relationship between register_max_tokens and register_dummy_data is a bit intricate. There needs to be certain level of consistency here. Hard to get right. Should we mention something here?
There was a problem hiding this comment.
I currently have a note in registry_dummy_data that mentions it should use the max number of tokens from each modality. Is that sufficient?
There was a problem hiding this comment.
IMO the two should be tied together for consistency - see my comment below in phi3v.py.
| from vllm.multimodal import MULTIMODAL_REGISTRY | ||
| @MULTIMODAL_REGISTRY.register_image_input_mapper() | ||
| + @MULTIMODAL_REGISTRY.register_max_image_tokens(<your_calculation>) |
There was a problem hiding this comment.
IMO the two should be tied together for consistency - see my comment below in phi3v.py.
| + (new_height // 336 + 1) * 12 | ||
|
|
||
|
|
||
| def get_max_phi3v_image_tokens(ctx: InputContext): |
There was a problem hiding this comment.
get_max_phi3v_image_tokens and dummy_data_for_phi3v are both based on dummy_height, dummy_width = 8000, 50, so we should make these constants to this file for consistency. I think this will suffice for the purpose of consistency for now, and in the future we can establish more structured protocol between multimodal feature size and dummy data.
There was a problem hiding this comment.
I've made #6146 to address this.
…roject#6121) Signed-off-by: Alvant <[email protected]>
…roject#6121) Signed-off-by: LeiWang1999 <[email protected]>
This PR further extends the multi-modal registry so that each model can specify its own maximum number of multi-modal tokens during memory profiling.
This replaces the function of the user-provided
image_feature_sizeargument in vision language config that was recently removed by #6089.