[V1][Metrics] Add TTFT and TPOT histograms #12530
Conversation
Compute these intervals on the API server side as EngineCoreOutputs are processed. The OutputProcessor needs to track per-request arrival times and the generation timestamp of the most recent token generation. Prefill intervals are calculated when the first EngineCoreOutput is received, indicating prefill completion. The histogram buckets match the v0 implementation, but they should probably be improved in future. Signed-off-by: Mark McLoughlin <[email protected]>
markmc
requested review from
DarkLight1337,
WoosukKwon,
alexm-redhat,
comaniac,
njhill,
robertgshaw2-redhat,
simon-mo and
ywang96
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
| return | ||
|
|
||
| num_new_generation_tokens = len(output.new_token_ids) | ||
| now = time.time() |
There was a problem hiding this comment.
I have seen mixed use of time.time() and time.monotonic() - I think we should standardize on one or the other so we don't have a footgun down the road
There was a problem hiding this comment.
Yes!
I had initially changed to time.monotonic() but I backed it out because it looks like arrival_time being wall-clock time is part of the public API, so not something I wanted to rush into.
I'll make sure to do it or capture it as a TODO/FIXME before I wrap this up 👍
robertgshaw2-redhat
added
the
ready
Signed-off-by: Mark McLoughlin <[email protected]>
Signed-off-by: Mark McLoughlin <[email protected]> Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: Mark McLoughlin <[email protected]>
Signed-off-by: Mark McLoughlin <[email protected]>
2 participants
Follow on from #12516, part of #10582
Compute these intervals on the API server side as EngineCoreOutputs are processed.
The OutputProcessor needs to track per-request arrival times and the generation timestamp of the most recent token generation.
Prefill intervals are calculated when the first EngineCoreOutput is received, indicating prefill completion.
The histogram buckets match the v0 implementation, but they should probably be improved in future.