[Bug]: PD demo example failed to run benchmark

[gaowayne]

Your current environment

The output of python collect_env.py

Your output of `python collect_env.py` here

(vast) root@salab-hpedl380g11-02:/mnt/nvme2n1/wayne/vast/perf/VUA# python collect_env.py
INFO 07-04 07:47:23 [init.py:244] Automatically detected platform cuda.
Collecting environment information...

BIOS Vendor ID: Intel(R) Corporation
Model name: INTEL(R) XEON(R) GOLD 6548Y+
BIOS Model name: INTEL(R) XEON(R) GOLD 6548Y+ CPU @ 2.5GHz
BIOS CPU family: 179
CPU family: 6
Model: 207
Thread(s) per core: 2
Core(s) per socket: 32
Socket(s): 2
Stepping: 2
CPU(s) scaling MHz: 20%
CPU max MHz: 4100.0000
CPU min MHz: 800.0000
BogoMIPS: 5000.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect user_shstk avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_pkg_req hfi vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Virtualization: VT-x
L1d cache: 3 MiB (64 instances)
L1i cache: 2 MiB (64 instances)
L2 cache: 128 MiB (64 instances)
L3 cache: 120 MiB (2 instances)
NUMA node(s): 2

GPU0 X NODE SYS SYS 32-63,96-127 1 N/A
GPU1 NODE X SYS SYS 32-63,96-127 1 N/A
NIC0 SYS SYS X PIX
NIC1 SYS SYS PIX X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1

==============================
Environment Variables

NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

run PD example application, then run benchmark, I suffer crashed. everything works fine without PD separation.

steps

1. start P

CUDA_VISIBLE_DEVICES=0 VLLM_LOGGING_LEVEL=INFO VLLM_USE_V1=1  bin/vua-vllm serve  mistralai/Mistral-7B-Instruct-v0.2  --gpu-memory-utilization 0.80   --kv-transfer-config '{"kv_connector":"VUAStorageConnector_V1","kv_role":"kv_producer","kv_connector_extra_config": {"shared_storage_path": "/mnt/nvme1n1/nfsrdma/test10"}}' --port 8001

2. start D

CUDA_VISIBLE_DEVICES=1 VLLM_LOGGING_LEVEL=INFO VLLM_USE_V1=1  bin/vua-vllm serve  mistralai/Mistral-7B-Instruct-v0.2  --gpu-memory-utilization 0.80   --kv-transfer-config '{"kv_connector":"VUAStorageConnector_V1","kv_role":"kv_consumer","kv_connector_extra_config": {"shared_storage_path": "/mnt/nvme1n1/nfsrdma/test10"}}' --port 8002

3. start example proxy server

python3 examples/online_serving/disaggregated_serving/disagg_proxy_demo.py  --model mistralai/Mistral-7B-Instruct-v0.2  --prefill localhost:8001   --decode localhost:8002  --port 8000

4. start bench on proxy server it hangs because P process dead

VLLM_LOGGING_LEVEL=INFO bin/vua-vllm bench serve --model mistralai/Mistral-7B-Instruct-v0.2 --random-input-len 20000 --random-output-len 4000 --port 8000
INFO 07-04 08:07:30 [__init__.py:244] Automatically detected platform cuda.
Namespace(subparser='bench', bench_type='serve', dispatch_function=<function BenchmarkServingSubcommand.cmd at 0x789d06771d00>, seed=0, num_prompts=1000, dataset_name='random', dataset_path=None, custom_output_len=256, custom_skip_chat_template=False, sonnet_input_len=550, sonnet_output_len=150, sonnet_prefix_len=200, sharegpt_output_len=None, random_input_len=20000, random_output_len=4000, random_range_ratio=0.0, random_prefix_len=0, hf_subset=None, hf_split=None, hf_output_len=None, endpoint_type='openai', label=None, base_url=None, host='127.0.0.1', port=8000, endpoint='/v1/completions', max_concurrency=None, model='mistralai/Mistral-7B-Instruct-v0.2', tokenizer=None, use_beam_search=False, logprobs=None, request_rate=inf, burstiness=1.0, trust_remote_code=False, disable_tqdm=False, profile=False, save_result=False, save_detailed=False, append_result=False, metadata=None, result_dir=None, result_filename=None, ignore_eos=False, percentile_metrics='ttft,tpot,itl', metric_percentiles='99', goodput=None, top_p=None, top_k=None, min_p=None, temperature=None, tokenizer_mode='auto', served_model_name=None, lora_modules=None)
/mnt/nvme2n1/wayne/vast/perf/vllm/vllm/benchmarks/serve.py:793: FutureWarning: It is strongly recommended to run mistral models with `--tokenizer-mode "mistral"` to ensure correct encoding and decoding.
  tokenizer = get_tokenizer(tokenizer_id,
INFO 07-04 08:07:34 [datasets.py:353] Sampling input_len from [19999, 19999] and output_len from [4000, 4000]
Starting initial single prompt test run...
Initial test run completed. Starting main benchmark run...
Traffic request rate: inf
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: None
 46%|████████████████████████████████████████████████████████▌                                                                  | 460/1000 [00:20<00:01, 368.53it/s]

5. P process is dead

8113, 18114, 18115, 18116, 18117, 18118, 18119, 18120, 18121, ַ18122, 18123, 1ั8124, 18125, 18126, 18127, 18128, 18129, 18130, 18131, 18132, 18133, 1ִ8134, ِ18135, 18136, 181்37, 18138, 18139, 18140, 18141, 18142, 18143, 18144, 18145, 18146, 18147, 18148, 18149, 18150, 18151, 18152, 18153ু, 18154, 18155, 18156, 18157, 18158, 18159, 18160, 18161, 18162, 18163, 18164, 18165, 18166, 18167, 18168, 181็್ু69, ี18170, 18171ิ, 18172, 18173, 18174, 18175, 1ेُ8176, 18177, 18178, 18179, 18180, 18181, 18182, 18183ֹ̯, 18184, 18185, 18186, 18187, 18188, 18189, 18190, 18191, 18192,্ْ́ 18193, 18194, 786, 444, 546, 18197, 18198, 4537, 2037, 270, 293, 4455, 1074, 18202, 18203, 18204, 28256, 28727, 377, 18207, 18208, 5927, 81្9, 903, 18211, 18212, 18213, 18214, 1ّ8215, 18216, 18217, 18218, 18͡219, 18220, 18221, 5461, 28721, 17773, 517, 8176, 18224, 18225, 1്8226, 18227, 18228, 18229, 18230, 18️231, 18232,َ 18233, 18234, 18235, 18236, 18237, 18238, 18239, 18240, 18241, 18242, 18243, 18244, 18245, 18246, 18247, 18̄248, 18249, 18250ּ, 18251, 18252, 18253, 18254, 18255, 18256, 18257, 18258, 18259̂, 18260, 18261, 18262, 18263,್ ু18264, 18265, 18266, 18267, 18268, 18269, 18270, 182712, 18272, 18273, 18274്, 18275, 18276, 18277, 18278, 18279, 18280,ु 3582, 1150, 10906,่ 1828्3, 18284, 18285, 18ְ286, 18287,้ 18288, 18289, 18290, 18291, 18292, 12808, 7752, 313, 160̄9, 4197, 1̯774, 18297, 18298, ֵ18299, 18300, 1830ั1, 18302, 18303, 183్04, 18305, 18306, 18307, 18308, 18309, 18310, 18311, 18312,ַ 183ֹ13, 18314, 18315, 183162, 18317, 18318, 18319, 18320, 18321, 1់83̂22, 18323, 18324, 18325, 18326, 18327, 18328, 18329, 28790, 1841, 2728, 18332, 183ू33, 18334, 18335, 18336, 183്37, 18338, 183239], prompt_embeds shape: None, lora_request: Nonีe, prompt_adapter_request: None.INFO:ֵ     127.0.0.1:46316 - ̥"POST /v1/̈completions HTTP/1.1" 200 OKិ్INFO ់07-04 08:10:38 [async_llṃ.py:345] Request cmpl-464ু667714f6a4f63affa7bfc4a3daeee-0 failed (engine dead).INFO:     127.0.0.1ု:463়22 - "POST /v1/compl්etions HTTP/1.1" 200ី OKINFO 07-04 ์08:10:38 [async_llmٰ.py:345] Request ̶cmpl-fdbab1865f5a44ֹོืe5a8ce03692a2e7f4f-0 failed (engine dead).INFO:     127.0.0.1:46336 - "POST /v1/completions HTTP/1.1" 200 OKINFO 07-04 08:10:38 [asًync_llm.py:345] Request ాcmpl-d9212ಿcf33ea64cb29c11532d1ffc8724-0 failed (engine dead).INFO:     127.0.0.1:46338 - "POST /v1/completions HTTP/1.1" 200 OKINFOි 07-04 08:10:38 [async_llm.py:345] Rིequest cmpl-bbै0f59b4b0424d98889e404d1f771e03-0 failed (engine dead).INFO:     Waiting for application shutdown.્INFO:     Applicationུ shutdown complete.़̇INFO:     Finished server process [560828]

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[panpan0000]

please provide the correct formatting prefill log , and as well , the example proxy logs, thanks.

[gaowayne]

please provide the correct formatting prefill log , and as well , the example proxy logs, thanks.

hello dear peter, it is me Wayne. :)

here is prefill log. it has no error.

(vast) root@salab-hpedl380g11-02:/mnt/nvme2n1/wayne/vast/perf/VUA# CUDA_VISIBLE_DEVICES=0 VLLM_LOGGING_LEVEL=INFO VLLM_USE_V1=1  bin/vua-vllm serve  mistralai/Mistral-7B-Instruct-v0.2  --gpu-memory-utilization 0.80   --kv-transfer-config '{"kv_connector":"VUAStorageConnector_V1","kv_role":"kv_producer","kv_connector_extra_config": {"shared_storage_path": "/mnt/nvme1n1/nfsrdma/test10"}}' --port 8001
INFO 07-10 10:15:06 [__init__.py:244] Automatically detected platform cuda.
INFO 07-10 10:15:09 [api_server.py:1287] vLLM API server version 0.9.2.dev166+g6f68c4922.d20250622
INFO 07-10 10:15:10 [cli_args.py:309] non-default args: {'port': 8001, 'model': 'mistralai/Mistral-7B-Instruct-v0.2', 'gpu_memory_utilization': 0.8, 'kv_transfer_config': KVTransferConfig(kv_connector='VUAStorageConnector_V1', engine_id='3bf32c37-0bb4-4ad7-a1ee-33b5a157d0ef', kv_buffer_device='cuda', kv_buffer_size=1000000000.0, kv_role='kv_producer', kv_rank=None, kv_parallel_size=1, kv_ip='127.0.0.1', kv_port=14579, kv_connector_extra_config={'shared_storage_path': '/mnt/nvme1n1/nfsrdma/test10'}, kv_connector_module_path=None)}
INFO 07-10 10:15:17 [config.py:831] This model supports multiple tasks: {'embed', 'score', 'reward', 'classify', 'generate'}. Defaulting to 'generate'.
INFO 07-10 10:15:17 [config.py:1444] Using max model len 32768
INFO 07-10 10:15:17 [config.py:2197] Chunked prefill is enabled with max_num_batched_tokens=2048.
/mnt/nvme2n1/wayne/vast/perf/vllm/vllm/transformers_utils/tokenizer_group.py:24: FutureWarning: It is strongly recommended to run mistral models with `--tokenizer-mode "mistral"` to ensure correct encoding and decoding.
  self.tokenizer = get_tokenizer(self.tokenizer_id, **tokenizer_config)
INFO 07-10 10:15:20 [__init__.py:244] Automatically detected platform cuda.
INFO 07-10 10:15:22 [core.py:459] Waiting for init message from front-end.
INFO 07-10 10:15:22 [core.py:69] Initializing a V1 LLM engine (v0.9.2.dev166+g6f68c4922.d20250622) with config: model='mistralai/Mistral-7B-Instruct-v0.2', speculative_config=None, tokenizer='mistralai/Mistral-7B-Instruct-v0.2', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=mistralai/Mistral-7B-Instruct-v0.2, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null}
WARNING 07-10 10:15:23 [utils.py:2756] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x739b5e69b2c0>
INFO 07-10 10:15:23 [parallel_state.py:1072] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
INFO 07-10 10:15:23 [factory.py:74] Creating v1 connector with name: VUAStorageConnector_V1 and engine_id: 3bf32c37-0bb4-4ad7-a1ee-33b5a157d0ef
WARNING 07-10 10:15:23 [base.py:71] Initializing KVConnectorBase_V1. This API is experimental and subject to change in the future as we iterate the design.
WARNING 07-10 10:15:23 [topk_topp_sampler.py:59] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
INFO 07-10 10:15:23 [gpu_model_runner.py:1691] Starting to load model mistralai/Mistral-7B-Instruct-v0.2...
INFO 07-10 10:15:23 [gpu_model_runner.py:1696] Loading model from scratch...
INFO 07-10 10:15:23 [cuda.py:259] Using Flash Attention backend on V1 engine.
INFO 07-10 10:15:24 [weight_utils.py:292] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/3 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  33% Completed | 1/3 [00:00<00:01,  1.75it/s]
Loading safetensors checkpoint shards:  67% Completed | 2/3 [00:01<00:00,  1.57it/s]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:01<00:00,  1.58it/s]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:01<00:00,  1.59it/s]

INFO 07-10 10:15:26 [default_loader.py:272] Loading weights took 1.95 seconds
INFO 07-10 10:15:26 [gpu_model_runner.py:1720] Model loading took 13.4967 GiB and 2.687737 seconds
INFO 07-10 10:15:31 [backends.py:508] Using cache directory: /root/.cache/vllm/torch_compile_cache/53d43ca849/rank_0_0/backbone for vLLM's torch.compile
INFO 07-10 10:15:31 [backends.py:519] Dynamo bytecode transform time: 4.45 s
INFO 07-10 10:15:35 [backends.py:155] Directly load the compiled graph(s) for shape None from the cache, took 3.318 s
INFO 07-10 10:15:35 [monitor.py:34] torch.compile takes 4.45 s in total
INFO 07-10 10:15:36 [gpu_worker.py:232] Available KV cache memory: 21.63 GiB
INFO 07-10 10:15:36 [kv_cache_utils.py:716] GPU KV cache size: 177,152 tokens
INFO 07-10 10:15:36 [kv_cache_utils.py:720] Maximum concurrency for 32,768 tokens per request: 5.41x
Capturing CUDA graphs: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 67/67 [00:18<00:00,  3.66it/s]
INFO 07-10 10:15:55 [gpu_model_runner.py:2196] Graph capturing finished in 18 secs, took 0.54 GiB
INFO 07-10 10:15:55 [core.py:172] init engine (profile, create kv cache, warmup model) took 28.31 seconds
/mnt/nvme2n1/wayne/vast/perf/vllm/vllm/transformers_utils/tokenizer_group.py:24: FutureWarning: It is strongly recommended to run mistral models with `--tokenizer-mode "mistral"` to ensure correct encoding and decoding.
  self.tokenizer = get_tokenizer(self.tokenizer_id, **tokenizer_config)
INFO 07-10 10:15:55 [factory.py:74] Creating v1 connector with name: VUAStorageConnector_V1 and engine_id: 3bf32c37-0bb4-4ad7-a1ee-33b5a157d0ef
WARNING 07-10 10:15:55 [base.py:71] Initializing KVConnectorBase_V1. This API is experimental and subject to change in the future as we iterate the design.
INFO 07-10 10:15:55 [loggers.py:137] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 11072
INFO 07-10 10:15:55 [api_server.py:1349] Starting vLLM API server 0 on http://0.0.0.0:8001
INFO 07-10 10:15:55 [launcher.py:29] Available routes are:
INFO 07-10 10:15:55 [launcher.py:37] Route: /openapi.json, Methods: GET, HEAD
INFO 07-10 10:15:55 [launcher.py:37] Route: /docs, Methods: GET, HEAD
INFO 07-10 10:15:55 [launcher.py:37] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 07-10 10:15:55 [launcher.py:37] Route: /redoc, Methods: GET, HEAD
INFO 07-10 10:15:55 [launcher.py:37] Route: /health, Methods: GET
INFO 07-10 10:15:55 [launcher.py:37] Route: /load, Methods: GET
INFO 07-10 10:15:55 [launcher.py:37] Route: /ping, Methods: POST
INFO 07-10 10:15:55 [launcher.py:37] Route: /ping, Methods: GET
INFO 07-10 10:15:55 [launcher.py:37] Route: /tokenize, Methods: POST
INFO 07-10 10:15:55 [launcher.py:37] Route: /detokenize, Methods: POST
INFO 07-10 10:15:55 [launcher.py:37] Route: /v1/models, Methods: GET
INFO 07-10 10:15:55 [launcher.py:37] Route: /version, Methods: GET
INFO 07-10 10:15:55 [launcher.py:37] Route: /v1/chat/completions, Methods: POST
INFO 07-10 10:15:55 [launcher.py:37] Route: /v1/completions, Methods: POST
INFO 07-10 10:15:55 [launcher.py:37] Route: /v1/embeddings, Methods: POST
INFO 07-10 10:15:55 [launcher.py:37] Route: /pooling, Methods: POST
INFO 07-10 10:15:55 [launcher.py:37] Route: /classify, Methods: POST
INFO 07-10 10:15:55 [launcher.py:37] Route: /score, Methods: POST
INFO 07-10 10:15:55 [launcher.py:37] Route: /v1/score, Methods: POST
INFO 07-10 10:15:55 [launcher.py:37] Route: /v1/audio/transcriptions, Methods: POST
INFO 07-10 10:15:55 [launcher.py:37] Route: /rerank, Methods: POST
INFO 07-10 10:15:55 [launcher.py:37] Route: /v1/rerank, Methods: POST
INFO 07-10 10:15:55 [launcher.py:37] Route: /v2/rerank, Methods: POST
INFO 07-10 10:15:55 [launcher.py:37] Route: /invocations, Methods: POST
INFO 07-10 10:15:55 [launcher.py:37] Route: /metrics, Methods: GET
INFO:     Started server process [584618]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     127.0.0.1:53226 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:52718 - "GET /v1/models HTTP/1.1" 200 OK
INFO 07-10 10:16:33 [chat_utils.py:420] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
INFO 07-10 10:16:33 [logger.py:43] Received request chatcmpl-ad9b5d5172e94ae197230c14cd6c9c8c: prompt: '<s> [INST] hello [/INST]', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.
INFO:     127.0.0.1:54410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 07-10 10:16:33 [async_llm.py:270] Added request chatcmpl-ad9b5d5172e94ae197230c14cd6c9c8c.
In connector.start_load_kv, but the attn_metadata is None
INFO 07-10 10:16:35 [loggers.py:118] Engine 000: Avg prompt throughput: 1.0 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
INFO 07-10 10:16:45 [loggers.py:118] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
INFO 07-10 10:16:50 [logger.py:43] Received request chatcmpl-93fb228f8cc6428fa6d7d78b313ff577: prompt: '<s> [INST] Below is a paper. Memorize the paper and answer my question after the paper.\n The paper begins. \n Abstract\nTo build Video Question Answering VideoQA) systems in daily activities,  seeking capable of assisting humans answers from long-form videos with diverse and complex Existing multi-modal VQA models achieve events is a must. promising performance video images or  short clips; on of large-scale   multi- especially with the recent success However; when extending these meth- modal pre-training: challenges arise: What are some potential directions to further improve the proposed model to address its current limitations? [/INST]', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.0, top_p=1.0, top_k=0, min_p=0.0, seed=0, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.
INFO:     127.0.0.1:36062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 07-10 10:16:50 [async_llm.py:270] Added request chatcmpl-93fb228f8cc6428fa6d7d78b313ff577.
In connector.start_load_kv, but the attn_metadata is None
INFO 07-10 10:16:55 [loggers.py:118] Engine 000: Avg prompt throughput: 790.7 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
INFO 07-10 10:17:05 [loggers.py:118] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%

decode has no error too.

evalscope perf have errors. I use vllm default bench works fine too.

(vast) root@salab-hpedl380g11-02:/mnt/nvme2n1/wayne/vast/perf/VUA# evalscope perf  --url "http://127.0.0.1:8000/v1/chat/completions"  --parallel 1 --model mistralai/Mistral-7B-Instruct-v0.2 --number 2 --api openai --dataset longalpaca --read-timeout 600 --connect-timeout 600 --log-every-n-query 20
2025-07-10 10:16:33,142 - evalscope - INFO - Save the result to: outputs/20250710_101633/Mistral-7B-Instruct-v0.2
2025-07-10 10:16:33,143 - evalscope - INFO - Starting benchmark with args: 
2025-07-10 10:16:33,143 - evalscope - INFO - {
    "model": "mistralai/Mistral-7B-Instruct-v0.2",
    "model_id": "Mistral-7B-Instruct-v0.2",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": null,
    "port": 8877,
    "url": "http://127.0.0.1:8000/v1/chat/completions",
    "headers": {},
    "connect_timeout": 600,
    "read_timeout": 600,
    "api_key": null,
    "no_test_connection": false,
    "number": 2,
    "parallel": 1,
    "rate": -1,
    "log_every_n_query": 20,
    "debug": false,
    "wandb_api_key": null,
    "swanlab_api_key": null,
    "name": null,
    "outputs_dir": "outputs/20250710_101633/Mistral-7B-Instruct-v0.2",
    "max_prompt_length": 9223372036854775807,
    "min_prompt_length": 0,
    "prefix_length": 0,
    "prompt": null,
    "query_template": null,
    "apply_chat_template": true,
    "dataset": "longalpaca",
    "dataset_path": null,
    "frequency_penalty": null,
    "repetition_penalty": null,
    "logprobs": null,
    "max_tokens": 2048,
    "min_tokens": null,
    "n_choices": null,
    "seed": 0,
    "stop": null,
    "stop_token_ids": null,
    "stream": true,
    "temperature": 0.0,
    "top_p": null,
    "top_k": null,
    "extra_args": {}
}
2025-07-10 10:16:34,855 - evalscope - INFO - Test connection successful.
2025-07-10 10:16:50,012 - evalscope - INFO - Save the data base to: outputs/20250710_101633/Mistral-7B-Instruct-v0.2/benchmark_data.db
Processing:   0%|                                                                                                                                                                                                                                                    | 0/2 [00:08<?, ?it/s]
2025-07-10 10:16:58,914 - evalscope - ERROR - Exception in async function 'statistic_benchmark_metric': Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/benchmark.py", line 140, in statistic_benchmark_metric
    metrics.update_metrics(benchmark_data, api_plugin)
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 109, in update_metrics
    benchmark_data._calculate_tokens(api_plugin)
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 45, in _calculate_tokens
    api_plugin.parse_responses(self.response_messages, request=self.request)
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/plugin/api/openai_api.py", line 110, in parse_responses
    last_response_js = json.loads(responses[-1])
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/benchmark.py", line 140, in statistic_benchmark_metric
    metrics.update_metrics(benchmark_data, api_plugin)
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 109, in update_metrics
    benchmark_data._calculate_tokens(api_plugin)
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 45, in _calculate_tokens
    api_plugin.parse_responses(self.response_messages, request=self.request)
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/plugin/api/openai_api.py", line 110, in parse_responses
    last_response_js = json.loads(responses[-1])
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/nvme2n1/wayne/vast/bin/evalscope", line 8, in <module>
    sys.exit(run_cmd())
             ^^^^^^^^^
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/cli/cli.py", line 25, in run_cmd
    cmd.execute()
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/cli/start_perf.py", line 33, in execute
    run_perf_benchmark(self.args)
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/main.py", line 94, in run_perf_benchmark
    return run_one_benchmark(args, output_path=output_path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/main.py", line 41, in run_one_benchmark
    return loop.run_until_complete(benchmark(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/usr/lib/python3.12/asyncio/base_events.py", line 1987, in _run_once
    handle._run()
  File "/usr/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/handler.py", line 20, in async_wrapper
    sys.exit(1)
SystemExit: 1
2025-07-10 10:16:59,032 - evalscope - ERROR - Exception in async function 'send_request': Event loop is closed
Traceback (most recent call last):
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/benchmark.py", line 91, in send_request
    async with client:
GeneratorExit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/benchmark.py", line 89, in send_request
    async with semaphore:
  File "/usr/lib/python3.12/asyncio/locks.py", line 20, in __aexit__
    self.release()
  File "/usr/lib/python3.12/asyncio/locks.py", line 406, in release
    self._wake_up_next()
  File "/usr/lib/python3.12/asyncio/locks.py", line 416, in _wake_up_next
    fut.set_result(True)
  File "/usr/lib/python3.12/asyncio/base_events.py", line 795, in call_soon
    self._check_closed()
  File "/usr/lib/python3.12/asyncio/base_events.py", line 541, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Exception ignored in: <coroutine object send_request at 0x7adb9c8b7a40>
Traceback (most recent call last):
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/handler.py", line 20, in async_wrapper
    sys.exit(1)
SystemExit: 1
2025-07-10 10:16:59,032 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-5' coro=<send_request() done, defined at /mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/handler.py:14> cb=[gather.<locals>._done_callback() at /usr/lib/python3.12/asyncio/tasks.py:767]>
2025-07-10 10:16:59,032 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-6' coro=<send_request() done, defined at /mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/handler.py:14> wait_for=<Future finished result=True> cb=[gather.<locals>._done_callback() at /usr/lib/python3.12/asyncio/tasks.py:767]>
(vast) root@salab-hpedl380g11-02:/mnt/nvme2n1/wayne/vast/perf/VUA# 

proxy server show no log, but I configure right. :)

(vast) root@salab-hpedl380g11-02:/mnt/nvme2n1/wayne/vast/vllm# python3 examples/online_serving/disaggregated_serving/disagg_proxy_demo.py  --model mistralai/Mistral-7B-Instruct-v0.2  --prefill localhost:8001   --decode localhost:8002  --port 8000
INFO:     Started server process [585683]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     127.0.0.1:34386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:60616 - "POST /v1/chat/completions HTTP/1.1" 200 OK

I see the requested passed down to prefill server via proxy already in the log. :(

[gaowayne]

@panpan0000 could you please take a look at this issue modelscope/evalscope#696

[chaunceyjiang]

@gaowayne Hi, the latest version of vLLM is not compatible with VUAStorageConnector_V1. The main reason is that PR #20232 changed the type of SchedulerOutput.scheduled_cached_reqs.

This causes an exception to be thrown at the following line in kv_connector_v1.py:

    for cached_req in scheduler_output.scheduled_cached_reqs:
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'CachedRequestData' object is not iterable

This needs to be addressed on the VUA side in order to restore compatibility.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!