[Bug]: GPU Memory Accounting Issue with Multiple vLLM Instances

[brokenlander]

Your current environment

The output of `python collect_env.py`

System Information Report
GPU Information

Model: NVIDIA GH200 480GB (Hopper Architecture)
Total GPU Memory: 97,871 MiB (~95.57 GiB)
NVIDIA Driver Version: 560.35.03

CUDA/PyTorch Environment

CUDA Version: 12.6 (System)
CUDA Version in Container: 12.6.2

vLLM Details

vLLM version: tested v0.6.4.post1 and v0.6.4.post2.dev71
Models being used:

Llama-3.1-8B-Instruct
Qwen2.5-7B-Instruct



System Details

Running in Docker containers with following settings:

GPU Memory Utilization: 0.3 (30%)
Max Model Length: 16384
Max Batched Tokens: 2048
Max Sequence Length to Capture: 16384
Chunked Prefill: Enabled
Prefix Caching: Enabled
IPC Mode: host
GPU Access: all GPUs exposed to container

Model Input Dumps

docker run --rm --gpus all
-p 8000:8000
--ipc=host
-v /llm-store/Llama-3.1-8B-Instruct:/model
--name vllm-llama
vllm-nvidia:latest
--host 0.0.0.0
--port 8000
--gpu-memory-utilization 0.3
--enable_chunked_prefill True
--enable_prefix_caching
--max_model_len 16384
--max-num-batched-tokens 2048
--max_seq_len_to_capture 16384
--model /model \

docker run --rm --gpus all
-p 8001:8000
--ipc=host
-v /llm-store/Qwen2.5-7B-Instruct:/model
--name vllm-qwen
vllm-nvidia:latest
--host 0.0.0.0
--port 8000
--gpu-memory-utilization 0.3
--enable_chunked_prefill True
--enable_prefix_caching
--max_model_len 16384
--max-num-batched-tokens 2048
--max_seq_len_to_capture 16384
--model /model \

🐛 Describe the bug

When running multiple vLLM instances on the same GPU, the second instance fails to start due to incorrect GPU memory accounting. The second instance appears to include the first instance's memory usage in its calculations, leading to a negative KV cache size and initialization failure.

Note: this setup was working fine in version 0.6.3.post1

First Instance (Works correctly)

Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:41<00:00, 10.38s/it]
INFO 11-25 14:11:42 model_runner.py:1077] Loading model weights took 14.9888 GB
INFO 11-25 14:11:42 worker.py:232] Memory profiling results: total_gpu_memory=95.00GiB initial_memory_usage=15.64GiB peak_torch_memory=16.20GiB memory_usage_post_profile=15.74GiB non_torch_memory=0.72GiB kv_cache_size=11.58GiB gpu_memory_utilization=0.30
INFO 11-25 14:11:42 gpu_executor.py:113] # GPU blocks: 5931, # CPU blocks: 2048
INFO 11-25 14:11:42 gpu_executor.py:117] Maximum concurrency for 16384 tokens per request: 5.79x

Second Instance (Memory issue)

Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:39<00:00,  9.95s/it]
INFO 11-25 14:14:05 model_runner.py:1077] Loading model weights took 14.2487 GB
INFO 11-25 14:14:06 worker.py:232] Memory profiling results: total_gpu_memory=95.00GiB initial_memory_usage=42.61GiB peak_torch_memory=15.67GiB memory_usage_post_profile=42.71GiB non_torch_memory=28.43GiB kv_cache_size=-15.61GiB gpu_memory_utilization=0.30
INFO 11-25 14:14:06 gpu_executor.py:113] # GPU blocks: 0, # CPU blocks: 4681
INFO 11-25 14:14:06 gpu_executor.py:117] Maximum concurrency for 16384 tokens per request: 0.00x
ERROR 11-25 14:14:06 engine.py:366] No available memory for the cache blocks. Try increasing `gpu_memory_utilization` when initializing the engine.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[xxyyy]

同样的问题，一个gpu上想启两个量化后的小模型，第一个initial_memory_usage是量化后的，第二就是原始的大小非量化后，直接报内存不够

[Madoshakalaka]

why was this issue closed? We also had this exactly problem where the second vllm instance fails to start because it produces a negative kv_cache_size during profiling.

More context: our blue-green-deployment proxy requires two vllms to be ran in parallel for a brief period of time...this makes it impossible...

[wehos]

The same error still happens with the latest version of vllm.

[wehos]

The issue is now located:

vllm/vllm/v1/worker/gpu_worker.py

Lines 202 to 211 in 2858830

	torch_allocated_bytes = torch.cuda.memory_stats(
	)["allocated_bytes.all.current"]
	total_allocated_bytes = torch.cuda.mem_get_info(
	)[1] - torch.cuda.mem_get_info()[0]
	non_torch_allocations = total_allocated_bytes - torch_allocated_bytes
	if non_torch_allocations > 0:
	peak_memory += non_torch_allocations
	available_kv_cache_memory = (
	total_gpu_memory * self.cache_config.gpu_memory_utilization -
	peak_memory)

total_allocated_bytes mistakenly includes memory consumed by other vLLM instances. As a result, non_torch_allocations includes the redundant memory, which leads to an unnecessarily low, or even negative, available_kv_cache_memory.

I understand the reason why vLLM includes non_torch_allocations when determining available kv cache, and I don't have a concrete idea for permanent solutions. However, a temporary solution is to modify the source code and build with local files.

[YanickSchraner]

+1

[vquilon]

The key point is that the percentage refers to the available free space on the GPU. I’m able to deploy multiple containers, each running different models with different memory utilization percentages.

I’m using a single GPU: A100 80GiB

• Container 1

  • Process ID: 6578
  • Model: BAAI/bge-m3
  • GPU utilization: 0.1

• Container 2

  • Process ID: 1159434
  • Model: microsoft/phi-4
  • GPU utilization: 0.5

• Container 3

  • Process ID: 1161199
  • Model: microsoft/Phi-4-multimodal-instruct
  • GPU utilization: 0.85

Here’s the actual VRAM usage output (nvidia-smi):

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.86.15              Driver Version: 570.86.15      CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------|
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100 80GB PCIe          Off |   00000001:00:00.0 Off |                    0 |
| N/A   43C    P0             71W /  300W |   68824MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            6578      C   /usr/bin/python3                       3110MiB |
|    0   N/A  N/A         1159434      C   /usr/bin/python3                      38086MiB |
|    0   N/A  N/A         1161199      C   /usr/bin/python3                      27598MiB |
+-----------------------------------------------------------------------------------------+

Calculation steps:

1. Total GPU memory: 81920 MiB

2. For Container 1 (10%):

   • 81920 * 0.1 = 8192 → but only 3110 MiB is actually used

3. Remaining: 81920 - 3110 = 78810 MiB

4. For Container 2 (50% of free):

   • 78810 * 0.5 = ~39405 → actual: 38086 MiB

5. Remaining: 78810 - 38086 = 40724 MiB

6. For Container 3 (85% of free):

   • 40724 * 0.85 = ~34615 → actual: 27598 MiB

Conclusion:
This only a hypothesis, but is the only way I figuere out how to run multiple containers with "limited" VRAM

If you try to start all containers at once using absolute percentages, it won’t work. The trick is that each percentage is calculated based on the currently free VRAM at the time of allocation—even if the configuration appears to be using fixed values. That’s why you must launch the containers sequentially, not simultaneously. Only then will the allocation logic behave as expected.

[wehos]

The key point is that the percentage refers to the available free space on the GPU. I’m able to deploy multiple containers, each running different models with different memory utilization percentages.

I’m using a single GPU: A100 80GiB

• Container 1

  • Process ID: 6578
  • Model: BAAI/bge-m3
  • GPU utilization: 0.1

• Container 2

  • Process ID: 1159434
  • Model: microsoft/phi-4
  • GPU utilization: 0.5

• Container 3

  • Process ID: 1161199
  • Model: microsoft/Phi-4-multimodal-instruct
  • GPU utilization: 0.85

Here’s the actual VRAM usage output (nvidia-smi):

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.86.15 Driver Version: 570.86.15 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------|
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A100 80GB PCIe Off | 00000001:00:00.0 Off | 0 |
| N/A 43C P0 71W / 300W | 68824MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 6578 C /usr/bin/python3 3110MiB |
| 0 N/A N/A 1159434 C /usr/bin/python3 38086MiB |
| 0 N/A N/A 1161199 C /usr/bin/python3 27598MiB |
+-----------------------------------------------------------------------------------------+
Calculation steps:

1. Total GPU memory: 81920 MiB

2. For Container 1 (10%):

   • 81920 * 0.1 = 8192 → but only 3110 MiB is actually used

3. Remaining: 81920 - 3110 = 78810 MiB

4. For Container 2 (50% of free):

   • 78810 * 0.5 = ~39405 → actual: 38086 MiB

5. Remaining: 78810 - 38086 = 40724 MiB

6. For Container 3 (85% of free):

   • 40724 * 0.85 = ~34615 → actual: 27598 MiB

Conclusion: This only a hypothesis, but is the only way I figuere out how to run multiple containers with "limited" VRAM

If you try to start all containers at once using absolute percentages, it won’t work. The trick is that each percentage is calculated based on the currently free VRAM at the time of allocation—even if the configuration appears to be using fixed values. That’s why you must launch the containers sequentially, not simultaneously. Only then will the allocation logic behave as expected.

Thank you for your detailed solutions. However, this does not solve all the problems. Sometime, when the first instance started, the available space went negative due to faulty calculation. In this case, it is impossible to start a new instance.

[YanickSchraner]

Should be fixed now in version 0.9.1 by PR #18974