[Performance]: phi 3.5 vision model consuming high CPU RAM and the process getting killed

[kuladeephx]

Proposal to improve performance

I am trying to run phi3.5 vision instruct model with around 10k prompts. What I noticed with the increase in prompts my CPU RAM consumption keeps increasing and eventually the process gets killed. Its running fine for say small sample like 1000 prompts. My system configuration is 48 GB VRAM and 64GB CPU RAM. Noticed a similar pattern with PIXTRAL-12B-2409. Has anyone faced this issue?

I have tried the implementation by passing in batches of 1000 to llm.generate but still the CPU RAM keeps increasing
Below is the code implementation:
Ima using two images per prompt
from vllm import LLM, SamplingParams
llm = LLM(
model="microsoft/Phi-3.5-vision-instruct",
gpu_memory_utilization=0.7,
trust_remote_code=True,
max_model_len=4096,
limit_mm_per_prompt={"image": 4},
)
sampling_params = SamplingParams(max_tokens=100, temperature=0.0)
outputs = llm.generate(prompt_list, sampling_params=sampling_params)

Report of performance regression

No response

Misc discussion on performance

No response

Your current environment (if you think it is necessary)

The output of `python collect_env.py`

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

It was fixed just earlier today by #9160. Can you try pulling the latest main branch?

[kuladeephx]

I have tried with the latest main branch, its still the same

[DarkLight1337]

cc @Isotr0py

[Isotr0py]

@kuladeephx I noticed you didn't set max_num_seqs when initialized LLM. Can you check if setting max_num_seqs=2 would solve this?

[kuladeephx]

Have tried now with max_num_seqs=2, still the same

[Isotr0py]

Does this issue only encounter on phi-3.5-vision or encounter on other VLMs? I'm reproducing this issue with below commands, but it would cost some time for me to run 10k prompts on T4:

python examples/offline_inference_vision_language.py -m phi3_v --num-prompts 10000

[kuladeephx]

Thanks for the try, so the above trail does it apply 10000 prompts with no repeated images?
yes noticed similar pattern with mistralai/Pixtral-12B-2409

[Isotr0py]

Can you check if other models like Qwen/Qwen2-VL-2B-Instruct and Qwen/Qwen2-VL-7B-Instruct also have this issue?

I'm not sure if this issue relates to composite weight loading, because we have fixed the RAM leaking bug on it previously. We can have a check on Qwen2-VL models since it doesn't use composite weight loading.

[kuladeephx]

Sure will try, was there any reference previously of people trying these above mentioned models with say 10000 prompts or higher. Just curious if anyone has tried in a similar scale

[kuladeephx]

@Isotr0py, can you pls let me know, were you able to reproduce the issue with 10000 prompts

[Isotr0py]

No, I can't reproduce this issue with 10k prompts. I didn't observe significant CPU RAM usage increase during the inference. In fact, my CPU RAM usage is mostly about 25% (about 4GB on my 16GB RAM device) all the time after loading the model.

Can you provide a full script that can reproduce this issue?

[Isotr0py]

The memory leak issue is a little bit similar to #8629 since it also occurred after long running.

Can you check if the solution provided by #8629 (comment) work?

[kuladeephx]

@Isotr0py , I have tried a similar implementation of clearing GPU cache and garbage collection earlier, but it doesn't solve the issue. I am attaching the script and sample data(dummy_image_data.zip), you can try the experiment with total_no_of_prompts = 300 vs total_no_of_prompts = 600 and see the difference. Additional packages which might be required are requests, memory_profiler. I am attaching memory profile screenshots for total_no_of_prompts = 300 vs 600 case. Experiment is carried by passing 150 prompts at once to the LLM class

Link for script -https://github.com/kuladeephx/sample/blob/main/multimodal_sample.py
Link for data - https://github.com/kuladeephx/sample/blob/main/dummy_image_data.zip

[Isotr0py]

Thanks, I can reproduce this issue with the provided script. Let me use vLLM's profiler to see what's wrong.