[Bug]: Compile inductor / CUDA Graph build before the memory profiling

[houseroad]

Your current environment

Running Llama4 Maverick on H100x8

🐛 Describe the bug

Otherwise, it's easy to get OOM. Inductor and CUDA graph themselves may consume a lot of memory, especially, inductor may leverage some profiling to search the best config for the kernels.

export LLAMA_DIR=meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8; export PORT=8081 VLLM_LOGGING_LEVEL=DEBUG VLLM_DISABLE_COMPILE_CACHE=1 SAFETENSORS_FAST_GPU=1 vllm serve $LLAMA_DIR --disable-log-requests -tp 8 --host :: --port $PORT --served-model-name default --no-enable-prefix-caching --max-model-len 4096 --gpu-memory-utilization 0.8 2>&1 | tee marverik_fp8_no_compile.log

If we use 0.9 or 0.95, it's easy to reproduce the issue on H100x8 machines.
0.8 may be okay.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[houseroad]

cc: @yeqcharlotte @WoosukKwon

[yeqcharlotte]

Probably hard to do it without doing 2 profile runs that introduces a lot of start up overhead.

We need to assign some blocks before start profiling. Wondering if using some heuristics to reserve additional memory for CUDAGraphs makes sense.

[yeqcharlotte]

#20628 should make things slightly better

[BoyuanFeng]

On CUDAGraph memory, we may estimate based on the peak activation memory. One thing to notice is that peak activation memory is profiled based on max_num_tokens (e.g. usually the user-specified model length such as 8192) which might be very different from max cudagraph capture size (512). So we can estimate via Estimated cudagraph memory = peak activation memory * (max_capture_size / max_num_tokens) * factor. Factor is some extra buffer.

Also checked the inductor memory overhead but it is relatively small (~40 MB comparing 1st and 2nd profiling run). So I did not handle it.

nccl memory has been correctly included in non_torch_increase since nccl initialization happens after inital memory snapshot.

More investigation here and #21489 should fix the issue.

[yinghai]

This is an interesting problem. I wonder if this problem can be solved by

1. Reserve a big range of memory address but allocate a small amount of KV memory using subset of those memory address so that we can do the profile and cudagraph capturing.
2. Deallocate the KV memory but hold the KV memory address. Estimate how much KV we can build now that we have information of cudagraph memory usage
3. Allocate the KV memory and map to the same reserved memory address for KV.

In this case the unified memory address captured by cudagraph early would still work.