Update Kimi-K2.md by adding '-dcp 8' and corresponding bench #63
Conversation
Signed-off-by: wangln19 <[email protected]>
Summary of ChangesHello @wangln19, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request enhances the documentation for the Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the Kimi-K2 documentation by adding a -dcp 8 flag to a vllm serve command and including a new benchmark result. My review focuses on improving the clarity and consistency of the documentation. I've suggested explaining the new command-line flag and restructuring the new benchmark section to be more informative and consistent with the rest of the document.
|
|
||
| # node 0: | ||
| vllm serve moonshotai/Kimi-K2-Instruct --trust-remote-code --tokenizer-mode auto --tensor-parallel-size 8 --pipeline-parallel-size 2 --dtype bfloat16 --quantization fp8 --max-model-len 2048 --max-num-seqs 1 --max-num-batched-tokens 1024 --enable-chunked-prefill --disable-log-requests --kv-cache-dtype fp8 | ||
| vllm serve moonshotai/Kimi-K2-Instruct --trust-remote-code --tokenizer-mode auto --tensor-parallel-size 8 --pipeline-parallel-size 2 --dtype bfloat16 --quantization fp8 --max-model-len 2048 --max-num-seqs 1 --max-num-batched-tokens 1024 --enable-chunked-prefill --disable-log-requests --kv-cache-dtype fp8 -dcp 8 |
There was a problem hiding this comment.
The -dcp 8 parameter has been added to the command, but it's not a standard vllm argument and isn't explained in this document. To help users understand how to use this configuration, please add a description of what -dcp does to the "Key parameter notes" section.
| After adding '-dcp 8': | ||
| ```bash | ||
| ============ Serving Benchmark Result ============ | ||
| Successful requests: 16 | ||
| Request rate configured (RPS): 10000.00 | ||
| Benchmark duration (s): 47.14 | ||
| Total input tokens: 128000 | ||
| Total generated tokens: 16000 | ||
| Request throughput (req/s): 0.34 | ||
| Output token throughput (tok/s): 339.38 | ||
| Peak output token throughput (tok/s): 384.00 | ||
| Peak concurrent requests: 16.00 | ||
| Total Token throughput (tok/s): 3054.46 | ||
| ---------------Time to First Token---------------- | ||
| Mean TTFT (ms): 2007.87 | ||
| Median TTFT (ms): 1932.03 | ||
| P99 TTFT (ms): 4680.76 | ||
| -----Time per Output Token (excl. 1st token)------ | ||
| Mean TPOT (ms): 45.01 | ||
| Median TPOT (ms): 45.10 | ||
| P99 TPOT (ms): 46.51 | ||
| ---------------Inter-token Latency---------------- | ||
| Mean ITL (ms): 45.01 | ||
| Median ITL (ms): 42.01 | ||
| P99 ITL (ms): 52.01 | ||
| ================================================== | ||
| ``` |
There was a problem hiding this comment.
Thank you for adding the benchmark results. To make this section clearer and more consistent with the rest of the document, please consider the following improvements:
- Placement: The
vllm servecommand was updated for the16xH800setup, but this benchmark result is placed after the16xH200benchmark. Please move this section to follow the16xH800benchmark results for consistency. - Missing Command: For reproducibility, please include the
vllm bench servecommand that was used to generate these results. - Section Title: The title "After adding '-dcp 8':" could be more descriptive. Consider making it a sub-heading like
#### With -dcp 8under the appropriate benchmark section. - Code Block Language: This code block is marked as
bash, while others are marked asshell. Using a consistent language specifier (shellorbash) would improve the document's consistency. - Final Newline: The file is missing a newline character at the end. It's a common convention to end files with a newline.
Signed-off-by: wangln19 [email protected]