[Bug]: The accuracy of vllm-Qwen2-VL-7B-Instruct is low.

[xiangxinhello]

Your current environment

from PIL import Image
from transformers import AutoProcessor
from vllm import LLM, SamplingParams
from qwen_vl_utils import process_vision_info

MODEL_PATH = '/workspace/mnt/storage/trt-llama/Qwen2-VL-7B-Instruct'
IMAGE_PATH = '/workspace/mnt/storage/llm_storge/vllm/examples/demo.jpeg'

llm = LLM(
model=MODEL_PATH,
dtype = 'float32',
limit_mm_per_prompt={'image': 10, 'video': 10},
)

sampling_params = SamplingParams(
temperature=0.1, top_p=0.001, repetition_penalty=1.05, max_tokens=256,
stop_token_ids=[],
)

messages = [
{'role': 'system', 'content': 'You are a helpful assistant.'},
{'role': 'user', 'content': [
{
'type': 'image',
'image': IMAGE_PATH,
'max_pixels': 12845056,
},
{
'type': 'text',
'text': '输出击掌的检测框',
},
]},
]

processor = AutoProcessor.from_pretrained(MODEL_PATH)
prompt = processor.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True,
)
image_inputs, video_inputs = process_vision_info(messages)

mm_data = {}
if image_inputs is not None:
mm_data['image'] = image_inputs
if video_inputs is not None:
mm_data['video'] = video_inputs

llm_inputs = {
'prompt': prompt,
'multi_modal_data': mm_data,
}
#击掌(529,516),(583,594)
outputs = llm.generate([llm_inputs], sampling_params=sampling_params)
generated_text = outputs[0].outputs[0].text

print(generated_text)

Model Input Dumps

No response

🐛 Describe the bug

Qwen2-VL-7B-Instruct：vllm-qwenvl-fp16 have a bug, The accuracy between vllm-qwenvl and transformer-qwenvl differs.
击掌(529,513),(584,605) vllm-fp16
击掌(531,516),(581,596) transformers-qwem2-vl-fp16
The coordinates of vllm are (529,513),(584,605).
The coordinates of transformers are (536,509),(588,602).
There is a significant difference in their errors.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

@fyabc can you help look into this if you have time? Thanks!

[fyabc]

@DarkLight1337 @xiangxinhello I will take a look at it.

[fyabc]

@xiangxinhello Hi, you set dtype to 'float32' in your example code. I want to confirm that which dtype do you use in vllm and transformers?

[fyabc]

@xiangxinhello Hi, you set dtype to 'float32' in your example code. I want to confirm that which dtype do you use in vllm and transformers?

If vllm using fp32 and transformers using fp16, the difference may be acceptable... @ShuaiBai623 can you take a look at this diff?

[xiangxinhello]

@xiangxinhello Hi, you set dtype to 'float32' in your example code. I want to confirm that which dtype do you use in vllm and transformers?

If vllm using fp32 and transformers using fp16, the difference may be acceptable... @ShuaiBai623 can you take a look at this diff?

vllm float32 and float16 have the same effect, both have errors.

[xiangxinhello]

Hi, @fyabc, Do you support Qwen-VL-Chat?

[fyabc]

Hi, @fyabc, Do you support Qwen-VL-Chat?

@xiangxinhello #8029 already supported Qwen-VL-Chat, you can try latest vllm-0.6.1.

[xiangxinhello]

@xiangxinhello Hi, you set dtype to 'float32' in your example code. I want to confirm that which dtype do you use in vllm and transformers?

If vllm using fp32 and transformers using fp16, the difference may be acceptable... @ShuaiBai623 can you take a look at this diff?

击掌(531,516),(581,596) transformers-qwem2-vl-fp16
击掌 (529,513),(584,605) vllm-qwem2-vl-fp16
击掌 (529,516),(583,594) vllm-qwem2-vl-fp32

[xiangxinhello]

[xiangxinhello]

Hi @fyabc, The models transformers-qwen-vl-float16 and vllm-qwen-vl-float16 show discrepancies. Could you help me with this?
击掌(536,509),(588,602) transformers-qwen-vl-float16。https://github.com/QwenLM/Qwen-VL
击掌(539,513),(588,604)<|im_end|> this is vllm-qwen-vl-float16

[xiangxinhello]

@xiangxinhello Hi, you set dtype to 'float32' in your example code. I want to confirm that which dtype do you use in vllm and transformers?

HI, @DarkLight1337 @fyabc, https://github.com/QwenLM/Qwen-VL
This is the result for transformers-qwen-vl-float16:击掌(536,509),(588,602)
This is the result for vllm-qwen-vl fp32: 击掌(539,513),(588,604)<|im_end|>
This is the result for vllm-qwen-vl fp16: 击掌(539,513),(588,604)<|im_end|>.

[fyabc]

@xiangxinhello Hi, I have tested Qwen2-VL-7B-Instruct fp16/fp32 on vllm and HF, and got the same outputs '击掌(529,516),(583,594)' in all three different seeds. Can you provide your GPU & environment information, and show your HuggingFace test script?

[xiangxinhello]

@xiangxinhello Hi, I have tested Qwen2-VL-7B-Instruct fp16/fp32 on vllm and HF, and got the same outputs '击掌(529,516),(583,594)' in all three different seeds. Can you provide your GPU & environment information, and show your HuggingFace test script?

Hi @fyabc
My vllm-qwen2-vl-Qwen2-VL-7B-Instruct-fp16 is also (529,516),(583,594).
transformers-Qwen2-VL-7B-Instruct is (531,516),(581,596).
The main issue is that there is a slight difference in the coordinate values between the two.

A100-PCIE-40GB

this transformers-Qwen2-VL-7B-Instruct environment information:
torch 2.4.0 pypi_0 pypi
torchvision 0.19.0 pypi_0 pypi
tqdm 4.66.5 pypi_0 pypi
transformers 4.45.0.dev0 pypi_0 pypi
python 3.10.14 h955ad1f_1

this is transformers test script:
from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info

model = Qwen2VLForConditionalGeneration.from_pretrained(
"/workspace/mnt/storage/trt-llama/Qwen2-VL-7B-Instruct", torch_dtype="auto", device_map="auto"
)

processor = AutoProcessor.from_pretrained("/workspace/mnt/storage/trt-llama/Qwen2-VL-7B-Instruct")

messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "/workspace/mnt/storage/llm_storge/vllm/examples/demo.jpeg",
},
{
"type": "text",
"text": "框出图中击掌的位置"
},
],
}
]

text = processor.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt",
)
inputs = inputs.to("cuda")

generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

[xiangxinhello]

@xiangxinhello Hi, I have tested Qwen2-VL-7B-Instruct fp16/fp32 on vllm and HF, and got the same outputs '击掌(529,516),(583,594)' in all three different seeds. Can you provide your GPU & environment information, and show your HuggingFace test script?

@xiangxinhello Hi, I have tested Qwen2-VL-7B-Instruct fp16/fp32 on vllm and HF, and got the same outputs '击掌(529,516),(583,594)' in all three different seeds. Can you provide your GPU & environment information, and show your HuggingFace test script?

The main issue is that there is a slight difference in the coordinate values between the two.