[Bug]: LLaVA 1.6 in 0.5.1: Exceptions after some bigger image request, stuck in faulty mode

[andrePankraz]

Your current environment

PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.5.0-35-generic-x86_64-with-glibc2.35
Is CUDA available: N/A
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration:
GPU 0: NVIDIA L40
GPU 1: NVIDIA L40
GPU 2: NVIDIA L40
GPU 3: NVIDIA L40

Nvidia driver version: 535.161.08
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 57 bits virtual
Byte Order:                         Little Endian
CPU(s):                             128
On-line CPU(s) list:                0-127
Vendor ID:                          GenuineIntel
Model name:                         Intel(R) Xeon(R) Gold 6438Y+
CPU family:                         6
Model:                              143
Thread(s) per core:                 2
Core(s) per socket:                 32
Socket(s):                          2
Stepping:                           8
BogoMIPS:                           4000.00
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hfi vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Virtualization:                     VT-x
L1d cache:                          3 MiB (64 instances)
L1i cache:                          2 MiB (64 instances)
L2 cache:                           128 MiB (64 instances)
L3 cache:                           120 MiB (2 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102,104,106,108,110,112,114,116,118,120,122,124,126
NUMA node1 CPU(s):                  1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79,81,83,85,87,89,91,93,95,97,99,101,103,105,107,109,111,113,115,117,119,121,123,125,127
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] No relevant packages
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: N/A
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    GPU2    GPU3    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NODE    SYS     SYS     0,2,4,6,8,10    0               N/A
GPU1    NODE     X      SYS     SYS     0,2,4,6,8,10    0               N/A
GPU2    SYS     SYS      X      NODE    1,3,5,7,9,11    1               N/A
GPU3    SYS     SYS     NODE     X      1,3,5,7,9,11    1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

🐛 Describe the bug

Config:

services:
  vllm-mixtral-instruct:
    image: vllm/vllm-openai:v0.5.1
    container_name: vllm-llava
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [ gpu ]
    environment:
      - HTTPS_PROXY=http://proxy.dev.init:3128
      - HTTP_PROXY=http://proxy.dev.init:3128
      - NO_PROXY=10.9.240.0/22,127.0.0.0/8
      - NVIDIA_VISIBLE_DEVICES=2,3
      - HF_TOKEN=$HF_TOKEN
      - VLLM_NO_USAGE_STATS=1
    volumes:
      - /mnt/sda/huggingface:/root/.cache/huggingface
      - .:/opt/vllm
    ports:
      - "8003:8000"
    command:
      - --model=llava-hf/llava-v1.6-mistral-7b-hf
      # - --chat-template=/opt/vllm/template_mixtral.jinja
      # - --max-model-len=24576
      - --tensor-parallel-size=2
      # - --swap-space=5
      - --gpu-memory-utilization=0.3
      - --max-num-batched-tokens=2048
      - --disable-log-requests
      - --enforce-eager
      - --enable-chunked-prefill
    restart: unless-stopped

It works for me via OpenAI Vision compatible API calls, e.g.:

curl 'https://ai1.dev.init/multimodal-llava/v1/chat/completions' -k -H 'Content-Type: application/json' -d @- <<EOF
{
    "model": "llava-hf/llava-v1.6-mistral-7b-hf",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/jpeg;base64,$(curl -s https://upload.wikimedia.org/wikipedia/commons/f/fa/Grayscale_8bits_palette_sample_image.png | base64 -w 0)"
                    }
                },
                {
                    "type": "text",
                    "text": "Was ist in dem Bild?"
                }
            ]
        }
    ],
    "temperature": 0.2,
    "top_p": 0.1,
    "top_k": 20,
    "frequency_penalty": 0.2
}
EOF

But after some bigger image I get the following exception and after that, I have to restart vLLM - doesn't work again even for smaller images.

INFO 07-06 11:30:48 api_server.py:206] vLLM API server version 0.5.1
INFO 07-06 11:30:48 api_server.py:207] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='llava-hf/llava-v1.6-mistral-7b-hf', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=2, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, gpu_memory_utilization=0.3, num_gpu_blocks_override=None, max_num_batched_tokens=2048, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=True, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=True, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, model_loader_extra_config=None, preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=True, max_log_len=None)
INFO 07-06 11:30:48 config.py:698] Defaulting to use mp for distributed inference
INFO 07-06 11:30:48 config.py:787] Chunked prefill is enabled (EXPERIMENTAL).
INFO 07-06 11:30:48 llm_engine.py:169] Initializing an LLM engine (v0.5.1) with config: model='llava-hf/llava-v1.6-mistral-7b-hf', speculative_config=None, tokenizer='llava-hf/llava-v1.6-mistral-7b-hf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=llava-hf/llava-v1.6-mistral-7b-hf, use_v2_block_manager=False, enable_prefix_caching=False)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
(VllmWorkerProcess pid=76) INFO 07-06 11:30:49 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=76) INFO 07-06 11:30:50 utils.py:741] Found nccl from library libnccl.so.2
INFO 07-06 11:30:50 utils.py:741] Found nccl from library libnccl.so.2
INFO 07-06 11:30:50 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=76) INFO 07-06 11:30:50 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 07-06 11:30:50 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /root/.config/vllm/gpu_p2p_access_cache_for_0,1.json
(VllmWorkerProcess pid=76) INFO 07-06 11:30:50 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /root/.config/vllm/gpu_p2p_access_cache_for_0,1.json
INFO 07-06 11:30:51 weight_utils.py:218] Using model weights format ['*.safetensors']
(VllmWorkerProcess pid=76) INFO 07-06 11:30:51 weight_utils.py:218] Using model weights format ['*.safetensors']
INFO 07-06 11:31:04 model_runner.py:255] Loading model weights took 7.1712 GB
(VllmWorkerProcess pid=76) INFO 07-06 11:31:04 model_runner.py:255] Loading model weights took 7.1712 GB
WARNING 07-06 11:31:04 model_runner.py:831] Computed max_num_seqs (min(256, 2048 // 2928)) to be less than 1. Setting it to the minimum value of 1.
(VllmWorkerProcess pid=76) WARNING 07-06 11:31:04 model_runner.py:831] Computed max_num_seqs (min(256, 2048 // 2928)) to be less than 1. Setting it to the minimum value of 1.
INFO 07-06 11:31:05 distributed_gpu_executor.py:56] # GPU blocks: 6521, # CPU blocks: 4096
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 07-06 11:31:09 serving_chat.py:94] Using default chat template:
INFO 07-06 11:31:09 serving_chat.py:94] {{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
WARNING 07-06 11:31:09 serving_embedding.py:141] embedding_mode is False. Embedding API will not work.
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 07-06 11:31:14 metrics.py:295] Avg prompt throughput: 264.9 tokens/s, Avg generation throughput: 0.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.3%, CPU KV cache usage: 0.0%.
INFO:     192.168.5.1:46884 - "POST /v1/chat/completions HTTP/1.0" 200 OK
INFO 07-06 11:31:19 metrics.py:295] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.4 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-06 11:31:28 metrics.py:295] Avg prompt throughput: 92.8 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
INFO:     192.168.5.1:44038 - "POST /v1/chat/completions HTTP/1.0" 200 OK
INFO 07-06 11:31:39 metrics.py:295] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.9 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop: Attempted to assign 2144 = 2144 image tokens to 2043 placeholders, Traceback (most recent call last):
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]     return func(*args, **kwargs)
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 64, in start_worker_execution_loop
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]     output = self.execute_model(execute_model_req=None)
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 271, in execute_model
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]     output = self.model_runner.execute_model(
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]     return func(*args, **kwargs)
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1243, in execute_model
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]     hidden_or_intermediate_states = model_executable(
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]     return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]     return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llava_next.py", line 494, in forward
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]     inputs_embeds = merge_vision_embeddings(
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 35, in merge_vision_embeddings
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]     raise ValueError(
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226] ValueError: Attempted to assign 2144 = 2144 image tokens to 2043 placeholders
(VllmWorkerProcess pid=76) ERROR 07-06 11:31:43 multiproc_worker_utils.py:226]
ERROR 07-06 11:31:43 async_llm_engine.py:53] Engine background task failed
ERROR 07-06 11:31:43 async_llm_engine.py:53] Traceback (most recent call last):
ERROR 07-06 11:31:43 async_llm_engine.py:53]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 43, in _log_task_completion
ERROR 07-06 11:31:43 async_llm_engine.py:53]     return_value = task.result()
ERROR 07-06 11:31:43 async_llm_engine.py:53]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 595, in run_engine_loop
ERROR 07-06 11:31:43 async_llm_engine.py:53]     result = task.result()
ERROR 07-06 11:31:43 async_llm_engine.py:53]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 540, in engine_step
ERROR 07-06 11:31:43 async_llm_engine.py:53]     request_outputs = await self.engine.step_async(virtual_engine)
ERROR 07-06 11:31:43 async_llm_engine.py:53]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 241, in step_async
ERROR 07-06 11:31:43 async_llm_engine.py:53]     output = await self.model_executor.execute_model_async(
ERROR 07-06 11:31:43 async_llm_engine.py:53]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/distributed_gpu_executor.py", line 173, in execute_model_async
ERROR 07-06 11:31:43 async_llm_engine.py:53]     return await self._driver_execute_model_async(execute_model_req)
ERROR 07-06 11:31:43 async_llm_engine.py:53]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 160, in _driver_execute_model_async
ERROR 07-06 11:31:43 async_llm_engine.py:53]     return await self.driver_exec_model(execute_model_req)
ERROR 07-06 11:31:43 async_llm_engine.py:53]   File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR 07-06 11:31:43 async_llm_engine.py:53]     result = self.fn(*self.args, **self.kwargs)
ERROR 07-06 11:31:43 async_llm_engine.py:53]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 271, in execute_model
ERROR 07-06 11:31:43 async_llm_engine.py:53]     output = self.model_runner.execute_model(
ERROR 07-06 11:31:43 async_llm_engine.py:53]   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
ERROR 07-06 11:31:43 async_llm_engine.py:53]     return func(*args, **kwargs)
ERROR 07-06 11:31:43 async_llm_engine.py:53]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1243, in execute_model
ERROR 07-06 11:31:43 async_llm_engine.py:53]     hidden_or_intermediate_states = model_executable(
ERROR 07-06 11:31:43 async_llm_engine.py:53]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
ERROR 07-06 11:31:43 async_llm_engine.py:53]     return self._call_impl(*args, **kwargs)
ERROR 07-06 11:31:43 async_llm_engine.py:53]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
ERROR 07-06 11:31:43 async_llm_engine.py:53]     return forward_call(*args, **kwargs)
ERROR 07-06 11:31:43 async_llm_engine.py:53]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llava_next.py", line 494, in forward
ERROR 07-06 11:31:43 async_llm_engine.py:53]     inputs_embeds = merge_vision_embeddings(
ERROR 07-06 11:31:43 async_llm_engine.py:53]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 35, in merge_vision_embeddings
ERROR 07-06 11:31:43 async_llm_engine.py:53]     raise ValueError(
ERROR 07-06 11:31:43 async_llm_engine.py:53] ValueError: Attempted to assign 2144 = 2144 image tokens to 2043 placeholders
Exception in callback functools.partial(<function _log_task_completion at 0x713f9a053880>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x713f822c8580>>)
handle: <Handle functools.partial(<function _log_task_completion at 0x713f9a053880>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x713f822c8580>>)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 43, in _log_task_completion
    return_value = task.result()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 595, in run_engine_loop
    result = task.result()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 540, in engine_step
    request_outputs = await self.engine.step_async(virtual_engine)
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 241, in step_async
    output = await self.model_executor.execute_model_async(
  File "/usr/local/lib/python3.10/dist-packages/vllm/executor/distributed_gpu_executor.py", line 173, in execute_model_async
    return await self._driver_execute_model_async(execute_model_req)
  File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 160, in _driver_execute_model_async
    return await self.driver_exec_model(execute_model_req)
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 271, in execute_model
    output = self.model_runner.execute_model(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1243, in execute_model
    hidden_or_intermediate_states = model_executable(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llava_next.py", line 494, in forward
    inputs_embeds = merge_vision_embeddings(
  File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 35, in merge_vision_embeddings
    raise ValueError(
ValueError: Attempted to assign 2144 = 2144 image tokens to 2043 placeholders

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 55, in _log_task_completion
    raise AsyncEngineDeadError(
vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for theactual cause.
INFO:     192.168.5.1:46956 - "POST /v1/chat/completions HTTP/1.0" 400 Bad Request

[DarkLight1337]

Can you provide a link to the image that is causing the problem?

[andrePankraz]

Thats just the officiel image, that the project is pushing to Docker Hub

https://hub.docker.com/layers/vllm/vllm-openai/v0.5.1/images/sha256-e58fceffa6f8d3e4d535f9e7128361cd33469b232a8dc670967b62ae62bac5fe?context=explore

[DarkLight1337]

Thats just the officiel image, that the project is pushing to Docker Hub

https://hub.docker.com/layers/vllm/vllm-openai/v0.5.1/images/sha256-e58fceffa6f8d3e4d535f9e7128361cd33469b232a8dc670967b62ae62bac5fe?context=explore

I'm referring to the image that you input to the model.

[andrePankraz]

Oh sry, the word image is totally rewired now in my brain ;)
I was using following image from your own tests in
https://github.com/vllm-project/vllm/blob/main/tests/entrypoints/openai/test_vision.py:

Image: https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg

curl 'https://ai1.dev.init/multimodal-llava/v1/chat/completions' -k -H 'Content-Type: application/json' -d @- <<EOF
{
    "model": "llava-hf/llava-v1.6-mistral-7b-hf",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/jpeg;base64,$(curl -s https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg | base64 -w 0)"
                    }
                },
                {
                    "type": "text",
                    "text": "Was ist in dem Bild?"
                }
            ]
        }
    ],
    "temperature": 0.2,
    "top_p": 0.1,
    "top_k": 20,
    "frequency_penalty": 0.2
}
EOF

[DarkLight1337]

For that particular image, does the same problem happen if you use offline inference? We should have fixed that already...

[andrePankraz]

Sry cannot invest more than that. if next version already fixes it, I'm fine with it. Just wanted to forward the observed issue.
I just use VLLM docker setup on remote GPU server. I would need to much time now to create a different setup with local GPU or create some debugging Docker image (cannot install/run directly there).

[DarkLight1337]

I tried the same command using a local server:

vllm/entrypoints/openai/api_server.py --port 8001 --model llava-hf/llava-v1.6-mistral-7b-hf --enforce-eager

Request:

curl 'http://localhost:8001/v1/chat/completions' -k -H 'Content-Type: application/json' -d @- <<EOF
{
    "model": "llava-hf/llava-v1.6-mistral-7b-hf",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/jpeg;base64,$(curl -s https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg | base64 -w 0)"
                    }
                },
                {
                    "type": "text",
                    "text": "Was ist in dem Bild?"
                }
            ]
        }
    ],
    "temperature": 0.2,
    "top_p": 0.1,
    "top_k": 20,
    "frequency_penalty": 0.2
}
EOF

Output:

{"id":"cmpl-05652f0b1fda4b1ba895beb78d6f412d","object":"chat.completion","created":1720336691,"model":"llava-hf/llava-v1.6-mistral-7b-hf","choices":[{"index":0,"message":{"role":"assistant","content":" Das Bild zeigt eine Landschaft mit einem Weg durch ein Grasfeld. Der Weg ist gepflastert und führt durch eine offene, grasige Fläche, die von einigen Bäumen und Sträuchern begrenzt wird. Im Hintergrund sind weitere Grasflächen und Bäume zu sehen, die unter einem weiten Himmel stehen. Es ist ein schöner, natürlicher Landschaftsbild, das eine ruhige und ungestörte Umgebung darstellt. ","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":2161,"total_tokens":2283,"completion_tokens":122}}

Can you try disabling --enable-chunked-prefill and/or running the model on a single GPU and see if it fixes the problem?

Edit: Actually, --max-num-batched-tokens=2048 might be causing the problem since the image takes up 2144 tokens, and the error message shows that only 2043 (=2048-5 where there are 5 text tokens) image tokens are available.

[andrePankraz]

I have deactivated prefill/maxnumbatch and it works with that. thx.

config was a copy from some other bigger LLM.

[DarkLight1337]

Glad to help!

[andrePankraz]

Yes thx.
May be it deserves some warning/error in startup if it has to be ruled out.
Also it's strange, that it's stuck in this faulty mode, even for other smaller images that worked.
But I'm happy at least. :)