[Usage]: vLLM's performance significantly slows down in the newer version when using multi-LoRA inference

[zhengwei-gao]

Test Conditions

• Dataset: Alpaca (Chinese)
• Total Requests: 500
• Request Rate: 16 req/s
• Model: Qwen2.5-32B-Instruct
• Hardware: A100_40G × 2

Results

I modified the benchmark code to run the Alpaca dataset. The benchmark duration (in seconds) is as follows:

vLLM Version	0.5.0	0.5.4	0.6.2
Only Base	49.33	48.47	37.93
Base with LoRA	58.96	71.81	83.56

The results indicate that vLLM's performance significantly slows down in the newer version (0.6.2) when using multi-LoRA inference.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[jeejeelee]

Could you perform testing using vllm 0.5.4 and 0.6.2?

[jeejeelee]

Similar issue: #9143

[zhengwei-gao]

Could you perform testing using vllm 0.5.4 and 0.6.2?

Results with vllm-0.5.4 have been updated.

It seems that multi-LoRA inference performance has been hurt since 0.5.4

[jeejeelee]

Thanks. Since version 0.5.4, vllm's multi-lora implementation has been replaced with a Triton-based implementation,see : #5036. We have conducted numerous experiments, and in most cases, it improves performance.

Could you provide your testing environment and testing code? I would be happy to assist in analyzing the issue you've encountered.

[zhengwei-gao]

Thanks. Since version 0.5.4, vllm's multi-lora implementation has been replaced with a Triton-based implementation,see : #5036. We have conducted numerous experiments, and in most cases, it improves performance.

Could you provide your testing environment and testing code? I would be happy to assist in analyzing the issue you've encountered.

Thank you for your warm assistance!

I am running VLLM on a host equipped with 8 A100 40GB GPUs connected via PCIe. Here is my startup command:

docker run --runtime nvidia --gpus '"device=0,1"' --shm-size=40gb -v /mnt_1/deploy/tmp:/tmp/ray  -v /mnt_1/models:/models -p 48003:8000 --ipc=host  vllm:0.6.2 --model /models/Qwen/Qwen2.5-32B-Instruct --served-model-name Qwen --gpu-memory-utilization 0.99 --max-model-len 12288 --tensor-parallel-size 2 --enforce-eager --enable-lora

Below is my testing code:
vllm_benchmark.zip

[jeejeelee]

Can you try canceling or disabling -enforce-eager and test it again?

[zhengwei-gao]

Can you try canceling or disabling -enforce-eager and test it again?

I must use the --enforce-eager option here; otherwise, I encounter an OOM error (my gpus only equip 40G memory each).

[jeejeelee]

If we use this arg, the performance of Triton-based LoRA will indeed decrease significantly. I'll use your code to test it later.

[jeejeelee]

@zhengwei-gao I aslo conducted some tests using your benchmark scirpt, the result are:

I'm using one A800 GPU (80GB) and have set --gpu-memory-utilization to 0.95. I noticed you have two A100 GPUs (40GB). Perhaps you could try reducing the gpu-memory-utilization (e.g., to 0.95) and disabling enforce-eager.

I hope this is helpful to you.

[zhengwei-gao]

@zhengwei-gao I aslo conducted some tests using your benchmark scirpt, the result are:

I'm using one A800 GPU (80GB) and have set --gpu-memory-utilization to 0.95. I noticed you have two A100 GPUs (40GB). Perhaps you could try reducing the gpu-memory-utilization (e.g., to 0.95) and disabling enforce-eager.

I hope this is helpful to you.

Thank you for helping me identify the cause and I know what to do next now.

I will close this issuse now.

[askcs517]

I had the same problem in T4. How did you solve it?
@zhengwei-gao @jeejeelee