Skip to content

Automatically configure KV cache size #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 17 commits into from
Mar 12, 2023
Merged

Automatically configure KV cache size #6

merged 17 commits into from
Mar 12, 2023

Conversation

@WoosukKwon
Copy link
Collaborator

@WoosukKwon WoosukKwon commented Mar 3, 2023 •
edited
Loading

This PR adds OPT memory analyzer to the system, and uses it to automatically determine the KV cache size.

Tested models:

  • OPT-125M
  • OPT-350M
  • OPT-1.3B
  • OPT-2.7B
  • OPT-6.7B
  • OPT-13B

Tested GPUs:

  • A100

@WoosukKwon WoosukKwon merged commit e9d3f2f into main Mar 12, 2023
@WoosukKwon WoosukKwon deleted the autoconfig branch March 12, 2023 07:23
@TheBloke TheBloke mentioned this pull request Jul 20, 2023
Closed
xiangyuT added a commit to xiangyuT/vllm that referenced this pull request Oct 24, 2023
@gc-fu @xiangyuT
Support to run batched inference in underlying changes (vllm-project#6)
f78c169 
* finish changing scheduler

* finish merge

* fix model

* Fix (vllm-project#5)

* fix problems

* fix

* delete unused params

* remove redundant comments

---------

Co-authored-by: Xiangyu Tian <[email protected]>
@shanshanpt shanshanpt mentioned this pull request Nov 17, 2023
Closed
@junior-zsy junior-zsy mentioned this pull request Nov 20, 2023
Closed
hongxiayang pushed a commit to hongxiayang/vllm that referenced this pull request Feb 13, 2024
@WoosukKwon
Add memory analyzer & utomatically configure KV cache size (vllm-proj…
de10960 
…ect#6)
slyalin pushed a commit to slyalin/vllm that referenced this pull request Mar 21, 2024
@ilya-lavrenov
Merge pull request vllm-project#6 from mzegla/extended_requirements
2922b06 
Add missing Python requirements
mzusman added a commit to mzusman/vllm that referenced this pull request Apr 16, 2024
@mzusman
dtype (vllm-project#6)
00bce1f 
Co-authored-by: Mor Zusman <[email protected]>
dtrifiro referenced this pull request in dtrifiro/vllm Apr 26, 2024
@z103cb
Merge pull request #6 from z103cb/ibm_main_docker_ubi_updates
91e4a51 
[CI/Build] Dockerfile.ubi : Remove test stage
@dlopes78 dlopes78 mentioned this pull request May 8, 2024
Closed
Starmys pushed a commit to Starmys/vllm that referenced this pull request May 20, 2024
@xiaoxiawu-microsoft
Merge pull request vllm-project#6 from wenxcs/wenxh/fp8-on-a100
4e56e27 
FP8 on A100 for PHIMOE
@oliver-li oliver-li mentioned this pull request Jul 5, 2024
Closed
lsy323 pushed a commit to lsy323/vllm that referenced this pull request Mar 31, 2025
@yaochengji
Merge pull request vllm-project#6 from lsy323/lsiyuan/update-torch-whl
afbf9aa 
update torchwhl
@alokkrsahu alokkrsahu mentioned this pull request Apr 9, 2025
Closed
1 task
@jifa513 jifa513 mentioned this pull request Apr 10, 2025
Closed
1 task
@qiuhaining qiuhaining mentioned this pull request Apr 10, 2025
Closed
1 task
@southfreebird southfreebird mentioned this pull request Apr 11, 2025
Closed
1 task
@QuanhuiGuan QuanhuiGuan mentioned this pull request Apr 14, 2025
Closed
1 task
maobaolong pushed a commit to maobaolong/vllm that referenced this pull request Apr 14, 2025
@Flechman
Add MLA support for v1 disagg connector (vllm-project#6)
406d6bf 
Signed-off-by: remi <[email protected]>
tlrmchlsmth referenced this pull request in tlrmchlsmth/vllm Apr 15, 2025
@ApostaC
Squashed commit of the following:
1cea76f 
commit 406d6bf
Author: Rémi Delacourt <[email protected]>
Date:   Fri Apr 11 00:47:40 2025 +0200

    Add MLA support for v1 disagg connector (#6)

    Signed-off-by: remi <[email protected]>

commit 1d8415d
Author: [email protected] <[email protected]>
Date:   Thu Apr 10 21:59:54 2025 +0000

    rename

    Signed-off-by: [email protected] <[email protected]>

commit 9c4159c
Author: [email protected] <[email protected]>
Date:   Thu Apr 10 21:41:20 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 54e1491
Author: [email protected] <[email protected]>
Date:   Thu Apr 10 20:31:35 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 8e1eadc
Author: [email protected] <[email protected]>
Date:   Thu Apr 10 20:26:37 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 05349a5
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 22:10:50 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 7f57f3c
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 17:13:31 2025 +0000

    update lifecycle

    Signed-off-by: [email protected] <[email protected]>

commit 7c31e29
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 17:03:55 2025 +0000

    nits

    Signed-off-by: [email protected] <[email protected]>

commit 74af233
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:44:01 2025 +0000

    done with nits

    Signed-off-by: [email protected] <[email protected]>

commit e64f745
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:28:51 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 40e5d81
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:25:04 2025 +0000

    refactor

    Signed-off-by: [email protected] <[email protected]>

commit 25c9592
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:20:41 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit fc58dd5
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:13:39 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 20decdf
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:06:15 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 5145566
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:52:03 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 62e1421
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:47:40 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 689379e
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:39:40 2025 +0000

    updaed

    Signed-off-by: [email protected] <[email protected]>

commit b1310fd
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:36:37 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 7b64acb
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:32:47 2025 +0000

    clean up code

    Signed-off-by: [email protected] <[email protected]>

commit 7766ca5
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:31:33 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit b0629bd
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:27:58 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit eca7a49
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:25:24 2025 +0000

    cleaning

    Signed-off-by: [email protected] <[email protected]>

commit 1881aa5
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:24:55 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 7833645
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:11:18 2025 +0000

    updared

    Signed-off-by: [email protected] <[email protected]>

commit e72e5e4
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:08:34 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 48c2eb2
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:07:37 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit de1e487
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:01:14 2025 +0000

    fix nit

    Signed-off-by: [email protected] <[email protected]>

commit 0163070
Merge: e2ecc14 8b3f606
Author: Robert Shaw <[email protected]>
Date:   Wed Apr 9 10:39:46 2025 -0400

    Merge pull request #4 from robertgshaw2-redhat/rob-changes

    Rob changes

commit 8b3f606
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 14:29:26 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 90e8c53
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 14:29:17 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit da019df
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 14:23:47 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 4ebcc3e
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 13:44:41 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 00df670
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 13:20:14 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit a73721a
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 13:19:17 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 31d807e
Author: [email protected] <[email protected]>
Date:   Tue Apr 8 20:58:28 2025 +0000

    stash

    Signed-off-by: [email protected] <[email protected]>

commit 5accb53
Author: [email protected] <[email protected]>
Date:   Tue Apr 8 16:00:29 2025 +0000

    stash

    Signed-off-by: [email protected] <[email protected]>

Signed-off-by: ApostaC <[email protected]>
tlrmchlsmth referenced this pull request in tlrmchlsmth/vllm Apr 15, 2025
@ApostaC
Squashed commit of the following:
ea8df07 
commit 3a24897
Author: ApostaC <[email protected]>
Date:   Thu Apr 10 18:31:47 2025 -0700

    [Fix] memory leak problem by proper clean up

    Signed-off-by: ApostaC <[email protected]>

commit 406d6bf
Author: Rémi Delacourt <[email protected]>
Date:   Fri Apr 11 00:47:40 2025 +0200

    Add MLA support for v1 disagg connector (#6)

    Signed-off-by: remi <[email protected]>

commit 1d8415d
Author: [email protected] <[email protected]>
Date:   Thu Apr 10 21:59:54 2025 +0000

    rename

    Signed-off-by: [email protected] <[email protected]>

commit 9c4159c
Author: [email protected] <[email protected]>
Date:   Thu Apr 10 21:41:20 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 54e1491
Author: [email protected] <[email protected]>
Date:   Thu Apr 10 20:31:35 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 8e1eadc
Author: [email protected] <[email protected]>
Date:   Thu Apr 10 20:26:37 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 05349a5
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 22:10:50 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 7f57f3c
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 17:13:31 2025 +0000

    update lifecycle

    Signed-off-by: [email protected] <[email protected]>

commit 7c31e29
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 17:03:55 2025 +0000

    nits

    Signed-off-by: [email protected] <[email protected]>

commit 74af233
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:44:01 2025 +0000

    done with nits

    Signed-off-by: [email protected] <[email protected]>

commit e64f745
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:28:51 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 40e5d81
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:25:04 2025 +0000

    refactor

    Signed-off-by: [email protected] <[email protected]>

commit 25c9592
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:20:41 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit fc58dd5
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:13:39 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 20decdf
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 16:06:15 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 5145566
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:52:03 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 62e1421
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:47:40 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 689379e
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:39:40 2025 +0000

    updaed

    Signed-off-by: [email protected] <[email protected]>

commit b1310fd
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:36:37 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 7b64acb
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:32:47 2025 +0000

    clean up code

    Signed-off-by: [email protected] <[email protected]>

commit 7766ca5
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:31:33 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit b0629bd
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:27:58 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit eca7a49
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:25:24 2025 +0000

    cleaning

    Signed-off-by: [email protected] <[email protected]>

commit 1881aa5
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:24:55 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 7833645
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:11:18 2025 +0000

    updared

    Signed-off-by: [email protected] <[email protected]>

commit e72e5e4
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:08:34 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 48c2eb2
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:07:37 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit de1e487
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 15:01:14 2025 +0000

    fix nit

    Signed-off-by: [email protected] <[email protected]>

commit 0163070
Merge: e2ecc14 8b3f606
Author: Robert Shaw <[email protected]>
Date:   Wed Apr 9 10:39:46 2025 -0400

    Merge pull request #4 from robertgshaw2-redhat/rob-changes

    Rob changes

commit 8b3f606
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 14:29:26 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 90e8c53
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 14:29:17 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit da019df
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 14:23:47 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 4ebcc3e
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 13:44:41 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 00df670
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 13:20:14 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit a73721a
Author: [email protected] <[email protected]>
Date:   Wed Apr 9 13:19:17 2025 +0000

    updated

    Signed-off-by: [email protected] <[email protected]>

commit 31d807e
Author: [email protected] <[email protected]>
Date:   Tue Apr 8 20:58:28 2025 +0000

    stash

    Signed-off-by: [email protected] <[email protected]>

commit 5accb53
Author: [email protected] <[email protected]>
Date:   Tue Apr 8 16:00:29 2025 +0000

    stash

    Signed-off-by: [email protected] <[email protected]>

Signed-off-by: ApostaC <[email protected]>
@hao-cold hao-cold mentioned this pull request May 13, 2025
Closed
1 task
@markmc markmc mentioned this pull request May 21, 2025
Closed
1 task
@zerosurplus zerosurplus mentioned this pull request Jun 16, 2025
Open
1 task
@xiaocode337317439 xiaocode337317439 mentioned this pull request Jun 27, 2025
Open
1 task
@Chris113113 Chris113113 mentioned this pull request Jul 10, 2025
Open
1 task
@shrijayan shrijayan mentioned this pull request Jul 12, 2025
Closed
@tyxiong23 tyxiong23 mentioned this pull request Jul 30, 2025
Closed
1 task
@xiaomofang xiaomofang mentioned this pull request Jul 31, 2025
Open
1 task
@devops724 devops724 mentioned this pull request Aug 3, 2025
Open
1 task
@fernandaspets fernandaspets mentioned this pull request Aug 8, 2025
Open
@crischeng crischeng mentioned this pull request Aug 12, 2025
Closed
1 task
@ruisearch42 ruisearch42 mentioned this pull request Aug 22, 2025
Open
1 task
@shaamil101-etched shaamil101-etched mentioned this pull request Aug 25, 2025
Open
1 task
@ZJY0516 ZJY0516 mentioned this pull request Aug 31, 2025
Closed
@wyn1015 wyn1015 mentioned this pull request Sep 19, 2025
Open
1 task
@zhanghb55 zhanghb55 mentioned this pull request Sep 25, 2025
Open
1 task
dcmaddix pushed a commit to dcmaddix/vllm that referenced this pull request Oct 10, 2025
@wcwuwc
Merge pull request vllm-project#6 from dcmaddix/marlin_experts_mxfp4
ac3a49e 
Add support for mxfp4 through marlin experts
@tina0852 tina0852 mentioned this pull request Oct 11, 2025
Open
1 task
jsboige pushed a commit to jsboige/vllm that referenced this pull request Oct 22, 2025
fix(grid-search): Resolve bugs vllm-project#5 (rope_scaling) and vllm…
464bb08 
…-project#6 (KV cache parsing)

Bug vllm-project#5: Fix JSON escaping in rope_scaling parameter
- Line 379: Correct rope_scaling JSON format with proper escaping
- Prevents malformed YAML in docker compose files

Bug vllm-project#6: Update regex patterns to match actual log format
- Lines 851-856: Update KV cache detection patterns
- Match actual vLLM log output format

All 6 grid search bugs now resolved (Missions 14a-14k)
Grid search validation successful with 36 configurations tested

Refs: Mission 14k, Mission 15
@Michel-debug Michel-debug mentioned this pull request Oct 23, 2025
Open
1 task
@whwangovo whwangovo mentioned this pull request Oct 23, 2025
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@WoosukKwon