[Model] Allow users to control skip reading cache per request.

[noooop]

Purpose

Even if prefix caching is enabled, it cannot read caching in the following two scenarios:

• prompt logprobs
• all pooling

Otherwise, the output might be less than n_prompt_tokens.

skip_reading_caching can still write to caching to accelerate following requests

Address #27145 (comment)

Test Plan

tests/models/language/pooling/test_extract_hidden_states.py

Test Result

main:

Even if chunked_prefill is not enabled, prefix_caching + all pooling will still cause the following error.
(EngineCore_DP0 pid=2893225) AssertionError: partial prefill not supported with ALL pooling

n_prompt_tokens:55 n_output_tokens:55 num_cached_tokens:0
n_prompt_tokens:56 n_output_tokens:8 num_cached_tokens:48
n_prompt_tokens:57 n_output_tokens:8 num_cached_tokens:48

n_prompt_tokens:55 n_output_tokens:48 num_cached_tokens:7
n_prompt_tokens:56 n_output_tokens:8 num_cached_tokens:48
n_prompt_tokens:57 n_output_tokens:8 num_cached_tokens:48

this pr:

Turn off prefix_caching during all pooling requests.

n_prompt_tokens:55 n_output_tokens:55 num_cached_tokens:0
n_prompt_tokens:56 n_output_tokens:56 num_cached_tokens:0
n_prompt_tokens:57 n_output_tokens:57  num_cached_tokens:0

n_prompt_tokens:55 n_output_tokens:55  num_cached_tokens:0
n_prompt_tokens:56 n_output_tokens:56  num_cached_tokens:0
n_prompt_tokens:57 n_output_tokens:57 num_cached_tokens:0

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces a mechanism to disable prefix caching on a per-request basis for scenarios where it can lead to incorrect output, specifically for requests involving prompt_logprobs or certain pooling tasks. The changes correctly add a not_using_prefix_caching flag to PoolingParams and SamplingParams and use it in the KVCacheManager. However, there is a critical bug in the implementation for SamplingParams where the logic to set this flag is inverted, which would cause significant performance degradation and incorrect behavior. I have provided a comment with a suggested fix for this critical issue.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[noooop]

/gemini review

[noooop]

cc @DarkLight1337
Ready for review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[gemini-code-assist]

Code Review

This pull request introduces a mechanism to control prefix caching on a per-request basis, which is necessary for features like prompt logprobs and 'all' pooling that are incompatible with prefix caching. The changes involve adding a disable_prefix_caching flag to SamplingParams and PoolingParams and updating the KV cache manager to respect this flag. The implementation for pooling parameters is correct, but there is a critical logic error in how the flag is set for sampling parameters, which I've commented on. The rest of the changes and the overall approach look good.

[heheda12345]

To accelerate it a bit, you can:

1. initialize the value of skip_reading_cache in process_inputs of vllm/v1/engine/processor.py
2. Put it in EngineCoreRequest
3. copy from EngineCoreRequest to Request by from_engine_core_request

In this way, skip_reading_cache will be computed in the frontend process rather than the engine core busy loop. (Though there won't be too much speedup, I want to avoid unnecessary operation in KVCacheManager as much as possible)

[noooop]

I think the get_skip_reading_prefix_cache logic best belongs in the Request.

I have cached the result of skip_reading_prefix_cache.

[heheda12345]

yeah caching also make sense to me

[heheda12345]

emmm... what about skip_reading_prefix_cache? There are too many types of cache in vllm

[heheda12345]

LGTM!