-
-
Notifications
You must be signed in to change notification settings - Fork 10.7k
[Spec Decode][Benchmark] Add Spec Bench Dataset for benchmarking #23563
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This pull request has merge conflicts that must be resolved before it can be |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request adds support for the Spec Bench dataset, which is a valuable addition for benchmarking. The implementation leverages the existing CustomDataset
class effectively. However, I've identified two high-severity issues that should be addressed. First, a variable name for an argument group is reused, which is confusing and could lead to bugs. Second, the load_data
method is called twice when initializing a SpecBench
object, leading to unnecessary overhead. The provided code suggestions aim to fix these issues.
This pull request has merge conflicts that must be resolved before it can be |
@RoyNissim, you might find this PR relevant to your recent efforts in advancing and standardizing benchmarking. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the contribution - Could you also create a section under https://github.com/vllm-project/vllm/blob/main/benchmarks/README.md specifically for spec decode if we're going to have multiple benchmark datasets under this category? (This can be in a follow-up PR)
Signed-off-by: Ekagra Ranjan <[email protected]>
Head branch was pushed to by a user without write access
5749f55
to
d562f98
Compare
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: Ekagra Ranjan <[email protected]>
…m-project#23563) Signed-off-by: Ekagra Ranjan <[email protected]>
…m-project#23563) Signed-off-by: Ekagra Ranjan <[email protected]>
…m-project#23563) Signed-off-by: Ekagra Ranjan <[email protected]>
…m-project#23563) Signed-off-by: Ekagra Ranjan <[email protected]> Signed-off-by: xuebwang-amd <[email protected]>
…m-project#23563) Signed-off-by: Ekagra Ranjan <[email protected]>
--spec-bench-category "summarization"
.Test
cmd
time VLLM_USE_HYBRID_MEM=0 VLLM_USE_V1=1 python3 examples/offline_inference/spec_decode.py --method eagle --num_spec_tokens 3 --tp 1 --dataset-name spec_bench --dataset-path "/host/vllm-cohere/data/spec_bench/question.jsonl" --num-prompts -1 --print-output
Output
cmd
time VLLM_USE_HYBRID_MEM=0 VLLM_USE_V1=1 python3 examples/offline_inference/spec_decode.py --method eagle --num_spec_tokens 3 --tp 1 --dataset-name spec_bench --dataset-path "/host/vllm-cohere/data/spec_bench/question.jsonl" --num-prompts -1 --print-output --spec-bench-category "summarization"
Output
cmd
time VLLM_USE_HYBRID_MEM=0 VLLM_USE_V1=1 python3 examples/offline_inference/spec_decode.py --method eagle --num_spec_tokens 3 --tp 1 --dataset-name spec_bench --dataset-path "/host/vllm-cohere/data/spec_bench/question.jsonl" --num-prompts -1 --print-output --spec-bench-category "math_reasoning"
Output