[Spec Decode][Benchmark] Add Spec Bench Dataset for benchmarking

[ekagra-ranjan]

• Adds Spec Bench Dataset https://github.com/hemingkx/Spec-Bench to vLLM benchmark suite.
• Can benchmark on a specific category only, e.g., --spec-bench-category "summarization".
• It leverages CustomDataset sampling to reduce code.
• Setting num prompt <=0 in dataset that inherit CustomDataset will load all the data

Test

cmd
time VLLM_USE_HYBRID_MEM=0 VLLM_USE_V1=1 python3 examples/offline_inference/spec_decode.py --method eagle --num_spec_tokens 3 --tp 1 --dataset-name spec_bench --dataset-path "/host/vllm-cohere/data/spec_bench/question.jsonl" --num-prompts -1 --print-output

Output

--------------------------------------------------
--------------------------------------------------
total_num_output_tokens: 68355
num_drafts: 31214
num_draft_tokens: 93642
num_accepted_tokens: 37294
mean acceptance length: 2.19
--------------------------------------------------
acceptance at token 0: 0.66
acceptance at token 1: 0.36
acceptance at token 2: 0.17

cmd
time VLLM_USE_HYBRID_MEM=0 VLLM_USE_V1=1 python3 examples/offline_inference/spec_decode.py --method eagle --num_spec_tokens 3 --tp 1 --dataset-name spec_bench --dataset-path "/host/vllm-cohere/data/spec_bench/question.jsonl" --num-prompts -1 --print-output --spec-bench-category "summarization"

Output

--------------------------------------------------
--------------------------------------------------
total_num_output_tokens: 17626
num_drafts: 8433
num_draft_tokens: 25299
num_accepted_tokens: 9193
mean acceptance length: 2.09
--------------------------------------------------
acceptance at token 0: 0.65
acceptance at token 1: 0.31
acceptance at token 2: 0.13

cmd
time VLLM_USE_HYBRID_MEM=0 VLLM_USE_V1=1 python3 examples/offline_inference/spec_decode.py --method eagle --num_spec_tokens 3 --tp 1 --dataset-name spec_bench --dataset-path "/host/vllm-cohere/data/spec_bench/question.jsonl" --num-prompts -1 --print-output --spec-bench-category "math_reasoning"

Output

--------------------------------------------------
--------------------------------------------------
total_num_output_tokens: 12641
num_drafts: 4963
num_draft_tokens: 14889
num_accepted_tokens: 7701
mean acceptance length: 2.55
--------------------------------------------------
acceptance at token 0: 0.78
acceptance at token 1: 0.50
acceptance at token 2: 0.27

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ekagra-ranjan.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[gemini-code-assist]

Code Review

This pull request adds support for the Spec Bench dataset, which is a valuable addition for benchmarking. The implementation leverages the existing CustomDataset class effectively. However, I've identified two high-severity issues that should be addressed. First, a variable name for an argument group is reused, which is confusing and could lead to bugs. Second, the load_data method is called twice when initializing a SpecBench object, leading to unnecessary overhead. The provided code suggestions aim to fix these issues.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ekagra-ranjan.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[keyboardAnt]

@RoyNissim, you might find this PR relevant to your recent efforts in advancing and standardizing benchmarking.

[ywang96]

Thanks for the contribution - Could you also create a section under https://github.com/vllm-project/vllm/blob/main/benchmarks/README.md specifically for spec decode if we're going to have multiple benchmark datasets under this category? (This can be in a follow-up PR)

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ekagra-ranjan.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork