Skip to content

[WIP] Two batch overlap #18415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 82 commits into from
Closed

[WIP] Two batch overlap #18415

wants to merge 82 commits into from

Conversation

@LucasWilkinson
Copy link
Collaborator

@LucasWilkinson LucasWilkinson commented May 20, 2025 •
edited by github-actions bot
Loading

Different streams high-level
image

Different Schedules
image

@github-actions
Copy link

github-actions bot commented May 20, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@mergify mergify bot added documentation Improvements or additions to documentation v1 labels May 20, 2025
@mgoin mgoin self-requested a review May 20, 2025 15:22
@mergify
Copy link

mergify bot commented May 21, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label May 21, 2025
LucasWilkinson and others added 14 commits May 22, 2025 20:51
@LucasWilkinson
wip
8293182 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
enable naive microbatching
37c9bab 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
support MLA
df8f889 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
support more args in dp example
f93bdd3 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
fix dummy mode
9ccfd09 
Signed-off-by: Lucas Wilkinson <[email protected]>
@SageMoore @LucasWilkinson
added multhreading support
020269c 
Signed-off-by: Sage Moore <[email protected]>
@LucasWilkinson
manually manage stream
ffb740a 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
working but only on the same stream
04f11d9 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
use vllm current_stream
2259b47 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
tp1 working multistream tp > 1 broken
9c60a62 
Signed-off-by: Lucas Wilkinson <[email protected]>
@SageMoore @LucasWilkinson
fix hang
2a7f25f
@LucasWilkinson
dp working no yields
a8439e2 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
seperate gpu wait
00f526f 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
wip
18bf91e 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson LucasWilkinson force-pushed the lwilkinson/attn-slicing branch from 713e4df to 18bf91e Compare May 23, 2025 03:32
LucasWilkinson and others added 9 commits May 23, 2025 03:32
@LucasWilkinson
wip
2dc3b8b 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
debugging hang
9edd082 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
tone down prints
952f3c5 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
better debug utils
e4419df 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
better logging
37bdf9f 
Signed-off-by: Lucas Wilkinson <[email protected]>
@SageMoore
fix dp=2 tp=2 hang
020d9b0 
Signed-off-by: Sage Moore <[email protected]>
@LucasWilkinson
add comment
2f39206 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
wip seperate comm and compute threads
7b31e8a 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
fixes
a743a35 
Signed-off-by: Lucas Wilkinson <[email protected]>
@mergify mergify bot removed the needs-rebase label Jun 18, 2025
@SageMoore
more fixes
ff2dd13 
Signed-off-by: Sage Moore <[email protected]>
@mergify
Copy link

mergify bot commented Jun 19, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Jun 19, 2025
@gx16377
Copy link

gx16377 commented Jun 19, 2025

Genius implementation method

SageMoore and others added 15 commits June 24, 2025 21:20
@SageMoore
setup deepepll for ubatching
a4def24 
Signed-off-by: Sage Moore <[email protected]>
@SageMoore
yields now work with deepep_ll
930efd0 
Signed-off-by: Sage Moore <[email protected]>
@SageMoore
added initial code for cuda graph capturing ubatches
96c0c4e 
Signed-off-by: Sage Moore <[email protected]>
@SageMoore
fix correctness issue with full-cudagraphs + attn splitting
97dbafa 
Signed-off-by: Sage Moore <[email protected]>
@SageMoore
initial full cudagraphs support. normal runs are working. ubatching d…
144b148 
…oes not

Signed-off-by: Sage Moore <[email protected]>
@SageMoore
add attention splitting to dummy runs
44a2b34 
Signed-off-by: Sage Moore <[email protected]>
@SageMoore
factored out some of the context creation code along with misc commet…
e2ba707 
…ed infra

Signed-off-by: Sage Moore <[email protected]>
@SageMoore
more refactoring
0e2b4bd 
Signed-off-by: Sage Moore <[email protected]>
@SageMoore
delete any notion of dummy_ubatch
54deb61 
Signed-off-by: Sage Moore <[email protected]>
@SageMoore
refactor a bunch of misc parameters into a UbatchMetadata class
78228a6 
Signed-off-by: Sage Moore <[email protected]>
@SageMoore
reintegrate full cudagraphs
af68574 
Signed-off-by: Sage Moore <[email protected]>
@SageMoore
capture works replay does not
4672c72 
Signed-off-by: Sage Moore <[email protected]>
@SageMoore
random push
d833982 
Signed-off-by: Sage Moore <[email protected]>
@liewegas
misc
88fbc04 
Signed-off-by: Sage Moore <[email protected]>
@LucasWilkinson
Merge remote-tracking branch 'origin/main' into lwilkinson/attn-slicing
f6db576 
Signed-off-by: Lucas Wilkinson <[email protected]>
@mergify mergify bot removed the needs-rebase label Jul 1, 2025
@LucasWilkinson
wip
9ce5044 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
no cuda-graph for now
b6b44d2 
Signed-off-by: Lucas Wilkinson <[email protected]>
@mergify
Copy link

mergify bot commented Jul 1, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Jul 1, 2025
@LucasWilkinson LucasWilkinson mentioned this pull request Jul 4, 2025
Merged
4 tasks
@lhtin
Copy link
Contributor

lhtin commented Sep 8, 2025

@LucasWilkinson Will this patch continue to be developed?

@LucasWilkinson
Copy link
Collaborator Author

LucasWilkinson commented Sep 8, 2025

@LucasWilkinson Will this patch continue to be developed?

Replaced by: #23693

@LucasWilkinson
Copy link
Collaborator Author

LucasWilkinson commented Sep 8, 2025

Replaced by: #23693

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ProExpertProg ProExpertProg ProExpertProg left review comments

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

@pavanimajety pavanimajety Awaiting requested review from pavanimajety pavanimajety will be requested when the pull request is marked ready for review pavanimajety is a code owner

@zou3519 zou3519 Awaiting requested review from zou3519 zou3519 will be requested when the pull request is marked ready for review zou3519 is a code owner

@youkaichao youkaichao Awaiting requested review from youkaichao youkaichao will be requested when the pull request is marked ready for review youkaichao is a code owner

@aarnphm aarnphm Awaiting requested review from aarnphm aarnphm will be requested when the pull request is marked ready for review aarnphm is a code owner

@chaunceyjiang chaunceyjiang Awaiting requested review from chaunceyjiang chaunceyjiang will be requested when the pull request is marked ready for review chaunceyjiang is a code owner

Assignees

@youkaichao youkaichao

Labels

documentation Improvements or additions to documentation frontend needs-rebase v1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@LucasWilkinson @gglin001 @gx16377 @lhtin @ProExpertProg @youkaichao @SageMoore @liewegas