How to call/add a new lora module to a live server?

[jayteaftw]

Hi,

I was reading through the documentation for Using Lora in VLLM.

In the documentation when they start the server, it looks like they have to specify which Lora modules are available
--lora-modules sql-lora=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/

Is it possible to do this in real-time instead? That is, start the server and call a recently added Lora module without having to stop and restart the server?

[prd-tuong-nguyen]

@jayteaftw Agree with you

[Erikfcb]

Might be super helpful

[AlphaINF]

mark, I am adding this feature.

[AlphaINF]

#3446

[Liucd0520]

hello，if this problem is fixed now. Indeed, when I need to update the lora model, I must stop the server, and add the new lora model and start the server again. what a stupid operation ! if lora model can be added to online server, the problem can be solved.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!