[Usage]: How to get "num_gpu_blocks" in V1？

[yansh97]

Your current environment

The output of `python collect_env.py`

How would you like to use vllm

In V0, I can get "num_gpu_blocks" through "llm.llm_engine.cache_config.num_gpu_blocks".

But in V1, LLM and EngineCore are in different processes. How can I get "num_gpu_blocks"?

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[yansh97]

Is it possible to provide an API to get the num_gpu_blocks after EngineCore initializes KV cache, or to write back to the vllm_config of LLMEngine?

[lengrongfu]

Maybe we should provide an API to query vllm_config data. @WoosukKwon What do you think?

[WoosukKwon]

@yansh97 @lengrongfu Makes sense. More specifically, I think we need an API to query the size of KV cache in terms of the number of tokens (rather than blocks) as we'd like to avoid exposing the concept of block to the users.

[yansh97]

I'd like to query num_gpu_tokens rather than num_gpu_blocks too. @WoosukKwon @lengrongfu

In V0, I always calculate llm.llm_engine.cache_config.block_size*llm.llm_engine.cache_config.num_gpu_blocks. This is too detailed and verbose for most users.

[lengrongfu]

@yansh97 Could you please explain whether you are using this variable by accessing the metric api or in what way? like this issue #15719

[yansh97]

Let me explain why I need this variable. I need to run a large number of conversations with similar length in batches. The online part is through OpenAI API, and the offline part is through LLM class. I calculate a reasonable concurrency value by num_gpu_tokens, and then send HTTP requests or call the chat API in LLM class in batches.

Therefore, I want to get this variable not only through the Metric API, but also through the Python API in LLM class. @lengrongfu

[lengrongfu]

/assign

Wait this #15755 PR merge after, i can work on this issue.

@yansh97 @lengrongfu Makes sense. More specifically, I think we need an API to query the size of KV cache in terms of the number of tokens (rather than blocks) as we'd like to avoid exposing the concept of block to the users.

Than I need add a method in class LLMEngine, For example, def num_gpu_tokens(self) -> int, Its calculation logic is llm.llm_engine.cache_config.block_size*llm.llm_engine.cache_config.num_gpu_blocks.

Is this what you expected? @WoosukKwon

[yansh97]

@lengrongfu I found out that the #15755 PR has been merged. How do I get the num_gpu_blocks by accessing the metric api?

[lengrongfu]

curl http://localhost:8000/metrics, Do we also need to provide a get_kv_cache_token_capacity method by calculating num_gpu_blocks*block_size.