[Feature]: Implement Concurrent Partial Prefills In V1 Engine

[robertgshaw2-redhat]

🚀 The feature, motivation and pitch

In V0, we support concurrent partial prefills to avoid TTFT latency with long requests. Implement it in V1

cc @WoosukKwon

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[plops655]

I can take this up.

[ccw1996]

I'm interested in contributing this.

[houseroad]

Shall we mark this as done? Since I feel in the context of chunked prefill, probably doesn't make sense to add support for max_num_partial_prefills and max_long_partial_prefills. cc: @comaniac

[comaniac]

Shall we mark this as done? Since I feel in the context of chunked prefill, probably doesn't make sense to add support for max_num_partial_prefills and max_long_partial_prefills. cc: @comaniac

IIRC in v0 we still apply max_num_partial_prefills for example even when chunked prefill is enabled, but I'm not fully sure which one (max_num_partial_prefills or long_prefill_token_threshold has a higher priority). This is a topic worth to discuss. cc @joerunde @njhill

[anirmohammadnoori72]

Hi! I’d like to take this issue and start working on it. I’ve already begun reviewing the code. Please let me know if it’s still available.

Thanks!

[whuLames]

Is there any progress about the support of chunked prefill in V1

[hibukipanim]

Is there any progress about the support of chunked prefill in V1

V1 Supports chunk prefill, and it's enabled by default.

What is still not supported is "Concurrent Partial Prefills", which this PR for V0 explains well: #10235

[pansicheng]

@comaniac @joerunde @njhill Hi! Mind taking a look at my PR (#21651)? It limits the number of concurrent long-text prefills processed per step by using the max_long_partial_prefills parameter in v1.