Enable CPU nightly performance benchmark and its Markdown report #18444
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
louie-tsai
force-pushed
the
nightly_cpu_benchmark
branch
9 times, most recently
from
913611c to
298aba1
Compare
louie-tsai
force-pushed
the
nightly_cpu_benchmark
branch
3 times, most recently
from
1299794 to
51ede3a
Compare
|
@bigPYJ1151 could you help to review this PR? |
louie-tsai
force-pushed
the
nightly_cpu_benchmark
branch
2 times, most recently
from
2f11d5e to
21ce351
Compare
|
@xuechendi please help to review and merge the PR. thanks |
louie-tsai
force-pushed
the
nightly_cpu_benchmark
branch
2 times, most recently
from
5bf0667 to
433ed5c
Compare
|
@louie-tsai , It looks like the rename for GPU script is not necessary, do you think it is OK to drop changes to existing GPU script and only add for CPU? |
|
@bigPYJ1151 , please take a look of this PR, the custom facing team want to provide CPU benchmark script from VLLM upstream repo, so customer can reproduce the number easily. Please check if current test settings makes sense to you. |
louie-tsai
force-pushed
the
nightly_cpu_benchmark
branch
4 times, most recently
from
7eb926e to
f5243fc
Compare
addressed it accordingly |
bigPYJ1151
left a comment
bigPYJ1151
left a comment
There was a problem hiding this comment.
Frankly speaking, I'm not sure reusing the benchmark script between CPU and GPU is a good idea.
Meanwhile, this PR also relates to vllm CI infra, needs to setup benchmark machines and pipelines. Do we have plan to establish these?
There was a problem hiding this comment.
I also think adding -gpu suffix is not necessary as it will introduce lots of changes.
There was a problem hiding this comment.
changed it accordingly.
There was a problem hiding this comment.
Perhaps numa should be added to requirements like test.in.
There was a problem hiding this comment.
added them accordingly.
We are targeting on release docker image. should we use test image instead?
if yes, do you have build command for test image?
There was a problem hiding this comment.
Right, the test image included more dictionaries and packages for testing/benchmarking.
To build it, just add --target=vllm-test when building the CPU image.
There was a problem hiding this comment.
Why is this change required?
There was a problem hiding this comment.
used to face a parsing issue on CPU results, but no issue on the latest code. remove the changes.
louie-tsai
force-pushed
the
nightly_cpu_benchmark
branch
2 times, most recently
from
a876145 to
2353aa2
Compare
Signed-off-by: Tsai, Louie <[email protected]>
Signed-off-by: Tsai, Louie <[email protected]>
Signed-off-by: Tsai, Louie <[email protected]>
Signed-off-by: Tsai, Louie <[email protected]>
Signed-off-by: Tsai, Louie <[email protected]>
Signed-off-by: Tsai, Louie <[email protected]>
Signed-off-by: Tsai, Louie <[email protected]>
Signed-off-by: Tsai, Louie <[email protected]>
Signed-off-by: Tsai, Louie <[email protected]>
Signed-off-by: Tsai, Louie <[email protected]>
Signed-off-by: Tsai, Louie <[email protected]>
Signed-off-by: Tsai, Louie <[email protected]>
Signed-off-by: Tsai, Louie <[email protected]> fix
Signed-off-by: Tsai, Louie <[email protected]>
…-to-markdown.py Co-authored-by: Eero Tamminen <[email protected]> Signed-off-by: Tsai, Louie <[email protected]>
Signed-off-by: Tsai, Louie <[email protected]>
louie-tsai
force-pushed
the
nightly_cpu_benchmark
branch
from
8b05811 to
104c444
Compare
|
rebase before #20200 to avoid CI/CD issue from it |
…m-project#18444) Signed-off-by: Tsai, Louie <[email protected]>
…m-project#18444) Signed-off-by: Tsai, Louie <[email protected]> Signed-off-by: Jinzhen Lin <[email protected]>
ghost
mentioned this pull request
6 participants
Need to standardize vLLM CPU benchmarks among customers and intel users by using vLLM benchmark suite.
Also hope to enable CPU perf numbers on vLLM performance dashboard.
Enable vLLM benchmark suite for CPU and below are snapshot of serving benchmark report.
numbers are aligned with our Xeon EMR numbers.
How to run it on CPU under vllm folder:
ON_CPU=1 bash .buildkite/nightly-benchmarks/scripts/run-performance-benchmarks.shalso added a new section "Platform Information" section to list out CPU info
Here is the full report.
benchmark_results_0527_3.md
overall, it took ~2 hours for current tests.
it also needs to have auto OMP thread binding from below PR.
#17930
Also added a compare-json-results.py to compare among different benchmark-results.json files
