[Usage]: How to serve embedding model and LLM at the same time

[weiyunfei]

Your current environment

The output of `python collect_env.py`

How would you like to use vllm

Is there any way to serve embedding model and LLM at the same time with vLLM? If so, what should I do?

[DarkLight1337]

Assuming that you're referring to using OpenAI compatible server to serve multiple models at once, no that is not supported. Instead, you can run multiple instances of the server (each serving a different model) at the same time, and have another layer to route the incoming request to the correct server accordingly.

[weiyunfei]

Thanks for your kind reply. BTW, do we have any plan to support multi-models in a single instance ?

[DarkLight1337]

Please refer to the following PRs for more information:

• Support Multiple Models #299
• Data parallel inference #1237
• Add worker registry service for hosting multiple vllm model through single api gateway #1753
• [Feature]: integrated model controller panel support? #4226
• [RFC]: Add control panel support for vLLM #4873

TL;DR: It's generally considered to be outside the scope of vLLM at this moment.