[Bug]: pytest -v compile/test_full_graph.py failing for PyTorch 2.9

[zou3519]

Your current environment

PyTorch 2.9, started showing up on #24994 after a recent rebase

🐛 Describe the bug

https://buildkite.com/vllm/ci/builds/34479#0199d273-f27f-48c2-857d-831e8793f572/163-5686, pytest -v compile/test_full_graph.py -k test_custom_compile_config[compilation_config12-model_info12]

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[zou3519]

I think this is some combination of the caching issue (we said somewhere that AOTAutograd cache didn't work with graph partition right? @ProExpertProg @BoyuanFeng), the cudagraph partition issue (#26735 changes the error message), and something else I am still digging into

[zou3519]

Okay so there are three issues:

1. AOTAutograd cache doesn't work with graph partition. Resolution is unfortunately to monkey patch our way to victory.
2. cudagraph partition issue. [BugFix] Patch inductor partitioning logic #26735 fixes that problem
3. this might need to be a spawn instead of a fork, but I don't understand that too well

To unblock @huydhn, we can just disable that test for now and bring it back later. It's too late to change PyTorch at this point, and said test that is failing only runs under 2.9

[BoyuanFeng]

this might need to be a spawn instead of a fork, but I don't understand that too well

https://buildkite.com/vllm/ci/builds/34479#0199d273-f27f-48c2-857d-831e8793f572/163-5686

these tests are vllm cache issues with inductor graph partition config. It should pass after #26116 landed.

[ProExpertProg]

this might need to be a spawn instead of a fork, but I don't understand that too well

Yeah, 2.9 initializes CUDA on a torch API call it did not initialize CUDA on before. So after forking we get this error. IIUC @huydhn is trying to find the location of this CUDA init happening before fork that we're trying to get rid of.

For now, I've forced spawn in both the test util and in vllm MP code on #26738 to see if that gets further in CI.

[ProExpertProg]

It seems that we've resolved all issues apart from the AOT (including spawn/fork), as CI passes for test_full_graph.py in #26738

[ProExpertProg]

I believe we've resolved this