API causes slowdown in batch request handling

[jpeig]

Using the API server and submitting multiple prompts to take advantage of speed benefit returns the following error:

"multiple prompts in a batch is not currently supported"

What's the point of vLLM without being able to send batches to the API?

Of course, I can send multiple seperate requests, but those are handled sequentially and do not benefit from speed improvements.

Correct me if I'm wrong...

[jpeig]

#1636

[simon-mo]

Of course, I can send multiple seperate requests, but those are handled sequentially and do not benefit from speed improvements.

This is not correct. vLLM automatically batches in-flight requests. It is built for the use case of high concurrency of requests. This means, when you are sending multiple individual requests, the underlying engine running in the server perform batching.

[simon-mo]

Further illustrated here, hope the explanation is helpful: #1636 (comment)

[jpeig]

It is thank you for the elaborated answer.

[simon-mo]

Ah one more thing, if you observing sequential behavior, try correct main branch instead of released version. Or turn on the flag --engine-use-ray. In the released version, our AsyncLLMEngine is single threaded and there's a fairly small chance concurrent queries don't get picked up, due to unfairness in Python asyncio.

This should be fixed as we work on #1677

[jpeig]

@simon-mo I'm using asyncio.gather to approach the API (calling the acreate function) so AsyncLLMEngine should be able to handle the queries concurrently. However I am still experiencing semi-sequential behavior whereby requests get sequentially added to the queue with seconds delay in between. I'll try out the main branch.

[simon-mo]

v0.2.2 was released last night. It should include the change. Please try it out and let us know!

[jpeig]

@simon-mo

I'm on main branch (latest).

I still notice 0.5 to 1 seconds between each request being added to the queue.
In the meantime no requests are being processed.

Only after all requests have been added, they do execute concurrently.

Is this expected behavior?

[jpeig]

INFO 11-21 01:07:22 async_llm_engine.py:370] Received request cmpl-78ae9b5f36b241c0b64131e838f2a85f
INFO 11-21 01:07:23 async_llm_engine.py:370] Received request cmpl-cb5e63e3f8d64d8a9b39afc7d9147c5b
INFO 11-21 01:07:24 async_llm_engine.py:370] Received request cmpl-db56d6b1a0f94ef7990f0eb21b98fcd1

etc...

Because I have sent quite a large of requests to the API, no requests are processed by vllm for 14 seconds (no GPU load).

[simon-mo]

Did you turn on engine-use-ray?

[jajj50386]

@jpeig I have the same problem, when sending for example 10 request concurrently then vllm wait around 10 seconds to start generating output for each requests. If in the middle of generation I send a new request then all requests (generating output) stop until this new request is handled.

@simon-mo I have used --engine-use-ray also, no changes. (api_server with stream)