[Usage]: vllm0.6 tensor parallel with cuda graph output is incorrect

[baoqian426]

Your current environment

Due to project requirements, I'm currently still using vLLM version 0.6.3. I noticed that when using CUDA Graph, the output is correct with TP=1, but becomes garbled with TP=2. Has anyone encountered this issue? Are there any related issues that have addressed and resolved it? Thank you very much!

How would you like to use vllm

I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[jeejeelee]

I know that 0.6.3 contains a cudagraph bug, see: https://github.com/vllm-project/vllm/blob/v0.6.3/vllm/worker/model_runner.py#L835. This bug had been fixed in #9549.

[baoqian426]

I know that 0.6.3 contains a cudagraph bug, see: https://github.com/vllm-project/vllm/blob/v0.6.3/vllm/worker/model_runner.py#L835. This bug had been fixed in #9549.

Thanks!I checked my code,it is already "max_decode_seq_len=max_decode_seq_len"

[jeejeelee]

Well, considering this version is a bit old, I suggest you try the new version

[baoqian426]

Well, considering this version is a bit old, I suggest you try the new version

I found that the issue lies in the all-reduce operation. When I directly return the embedding output, I see that hidden_states on rank 0 is all zeros ([0, 0, 0, ...]), while on rank 1, hidden_states contains actual values. In theory, the hidden_states on both ranks should be the same after the all-reduce.

I have confirmed that torch.distributed.init_process_group is executed on all ranks with the correct parameters. Where could the problem be? Why does it work fine in eager mode?