[Bug]: RuntimeError with tensor_parallel_size > 1 in Process Bootstrapping Phase

[zixuzixu]

Your current environment

The output of `python collect_env.py`
Collecting environment information...
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Rocky Linux release 8.9 (Green Obsidian) (x86_64)
GCC version: (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20)
Clang version: 16.0.6 (Red Hat 16.0.6-2.module+el8.9.0+1651+e10a8f6d)
CMake version: version 3.29.5
Libc version: glibc-2.28

Python version: 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] (64-bit runtime)
Python platform: Linux-4.18.0-513.24.1.el8_9.x86_64-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100-PCIE-40GB
GPU 1: NVIDIA A100-PCIE-40GB
GPU 2: NVIDIA A100-PCIE-40GB
GPU 3: NVIDIA A100-PCIE-40GB
GPU 4: NVIDIA A100-PCIE-40GB
GPU 5: NVIDIA A100-PCIE-40GB
GPU 6: NVIDIA A100-PCIE-40GB
GPU 7: NVIDIA A100-PCIE-40GB

Nvidia driver version: 550.54.15
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              48
On-line CPU(s) list: 0-47
Thread(s) per core:  1
Core(s) per socket:  24
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               106
Model name:          Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz
Stepping:            6
CPU MHz:             3500.000
CPU max MHz:         3500.0000
CPU min MHz:         800.0000
BogoMIPS:            5600.00
Virtualization:      VT-x
L1d cache:           48K
L1i cache:           32K
L2 cache:            1280K
L3 cache:            36864K
NUMA node0 CPU(s):   0-23
NUMA node1 CPU(s):   24-47
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid fsrm md_clear pconfig flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] pytorch-lightning==2.3.0
[pip3] torch==2.3.0
[pip3] torchaudio==2.3.1
[pip3] torchmetrics==1.4.0.post0
[pip3] torchvision==0.18.1
[pip3] transformers==4.42.0.dev0
[pip3] triton==2.3.0
[conda] blas                      2.116                       mkl    conda-forge
[conda] blas-devel                3.9.0            16_linux64_mkl    conda-forge
[conda] libblas                   3.9.0            16_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            16_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            16_linux64_mkl    conda-forge
[conda] liblapacke                3.9.0            16_linux64_mkl    conda-forge
[conda] libopenvino-pytorch-frontend 2024.1.0             he02047a_7    conda-forge
[conda] mkl                       2022.1.0           h84fe81f_915    conda-forge
[conda] mkl-devel                 2022.1.0           ha770c72_916    conda-forge
[conda] mkl-include               2022.1.0           h84fe81f_915    conda-forge
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-nccl-cu12          2.20.5                   pypi_0    pypi
[conda] pytorch-cuda              12.1                 ha16c6d3_5    pytorch
[conda] pytorch-lightning         2.3.0              pyhd8ed1ab_0    conda-forge
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torch                     2.3.0                    pypi_0    pypi
[conda] torchaudio                2.3.1               py310_cu121    pytorch
[conda] torchmetrics              1.4.0.post0        pyhd8ed1ab_0    conda-forge
[conda] torchvision               0.18.1              py310_cu121    pytorch
[conda] transformers              4.42.0.dev0              pypi_0    pypi
[conda] triton                    2.3.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.0
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV12    SYS     SYS     SYS     SYS     SYS     SYS     0-23    0               N/A
GPU1    NV12     X      SYS     SYS     SYS     SYS     SYS     SYS     0-23    0               N/A
GPU2    SYS     SYS      X      NV12    SYS     SYS     SYS     SYS     0-23    0               N/A
GPU3    SYS     SYS     NV12     X      SYS     SYS     SYS     SYS     0-23    0               N/A
GPU4    SYS     SYS     SYS     SYS      X      NV12    SYS     SYS     24-47   1               N/A
GPU5    SYS     SYS     SYS     SYS     NV12     X      SYS     SYS     24-47   1               N/A
GPU6    SYS     SYS     SYS     SYS     SYS     SYS      X      NV12    24-47   1               N/A
GPU7    SYS     SYS     SYS     SYS     SYS     SYS     NV12     X      24-47   1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

🐛 Describe the bug

Description

When setting tensor_parallel_size to a value greater than 1, the program gets stuck and raises a RuntimeError related to the bootstrapping phase of new processes. This issue does not occur when using version v0.4.3, but persists in versions v0.5.0.post1 and v0.5.0.

Steps to Reproduce

1. Install version v0.5.0.post1 or v0.5.0 of the library.

2. Run the following Python script with tensor_parallel_size set to 2:

   import os
   import argparse
   import ray
   from vllm import SamplingParams, LLM
   from dotenv import load_dotenv, find_dotenv
   _ = load_dotenv(find_dotenv())
   
   ray.init(address="auto")
   
   def parse_args():
       parser = argparse.ArgumentParser()
       parser.add_argument("--model_path", type=str, default=os.path.join(os.environ.get("LLAMA_MODEL_FOLDER")))
       return parser.parse_args()    
   
   args = parse_args()
   MODEL_PATH = args.model_path
   
   llm = LLM(model=MODEL_PATH, tensor_parallel_size=2)

python load.py  --model_path $JOBFS/fine_tuned_models/checkpoint-1857

Expected Behavior

The program should run without any issues regarding process bootstrapping, similar to how it behaves with version v0.4.3.

Observed Behavior

The program raises the following RuntimeError when tensor_parallel_size is set to 2:

python load.py  --model_path $JOBFS/fine_tuned_models/checkpoint-1857
INFO 06-18 21:35:42 config.py:623] Defaulting to use mp for distributed inference
INFO 06-18 21:35:42 llm_engine.py:161] Initializing an LLM engine (v0.5.0) with config: model='/scratch/pbs.5401450.kman.restech.unsw.edu.au/fine_tuned_models/checkpoint-1857', speculative_config=None, tokenizer='/scratch/pbs.5401450.kman.restech.unsw.edu.au/fine_tuned_models/checkpoint-1857', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/scratch/pbs.5401450.kman.restech.unsw.edu.au/fine_tuned_models/checkpoint-1857)
INFO 06-18 21:35:44 config.py:623] Defaulting to use mp for distributed inference
INFO 06-18 21:35:44 llm_engine.py:161] Initializing an LLM engine (v0.5.0) with config: model='/scratch/pbs.5401450.kman.restech.unsw.edu.au/fine_tuned_models/checkpoint-1857', speculative_config=None, tokenizer='/scratch/pbs.5401450.kman.restech.unsw.edu.au/fine_tuned_models/checkpoint-1857', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/scratch/pbs.5401450.kman.restech.unsw.edu.au/fine_tuned_models/checkpoint-1857)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/runpy.py", line 289, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/codes/kgqa/load.py", line 20, in <module>
    llm = LLM(model = MODEL_PATH, tensor_parallel_size=2)
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 144, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 360, in from_engine_args
    engine = cls(
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 223, in __init__
    self.model_executor = executor_class(
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 25, in __init__
    super().__init__(*args, **kwargs)
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 41, in __init__
    self._init_executor()
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 48, in _init_executor
    self.workers = [
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 49, in <listcomp>
    ProcessWorkerWrapper(
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/executor/multiproc_worker_utils.py", line 162, in __init__
    self.process.start()
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "/scratch/pbs.5401450.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
ERROR 06-18 21:35:45 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 3449547 died, exit code: 1
INFO 06-18 21:35:45 multiproc_worker_utils.py:123] Killing local vLLM worker processes

Environment

• OS: GNU/Linux
• Python Version: Python 3.10.14
• Library Version: v0.5.0.post1 and v0.5.0

Additional Context

Reverting back to version v0.4.3 resolves the issue. It appears there might be a change in how processes are handled in newer versions that could be causing this error.

[bkchang]

We are seeing the same error for LLM, LLMEngine, and AsyncLLMEngine.

Interestingly, we find wrapping everything in the python script within if __name__ == '__main__': can temporarily bypass the issue.

For test.py being the following

import os
from vllm import LLM, SamplingParams

prompts = ['<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful chatbot who always responds to requests.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWhat is ramen?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n']

sampling_params = SamplingParams(temperature=0.0, max_tokens=512)

llm = LLM(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    tensor_parallel_size=2,
    disable_log_stats=False,
)

outputs = llm.generate(prompts, sampling_params)

Running python test.py gives

(venv-vllm) ben@sakura-h100-3:/nvme0n1/ben/test$ python test.py
/nvme0n1/ben/venvs/venv-vllm/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
INFO 06-19 01:40:01 config.py:632] Defaulting to use mp for distributed inference
INFO 06-19 01:40:01 llm_engine.py:161] Initializing an LLM engine (v0.5.0.post1) with config: model='meta-llama/Meta-Llama-3-8B-Instruct', speculative_config=None, tokenizer='meta-llama/Meta-Llama-3-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=meta-llama/Meta-Llama-3-8B-Instruct)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
/nvme0n1/ben/venvs/venv-vllm/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
INFO 06-19 01:40:04 config.py:632] Defaulting to use mp for distributed inference
INFO 06-19 01:40:04 llm_engine.py:161] Initializing an LLM engine (v0.5.0.post1) with config: model='meta-llama/Meta-Llama-3-8B-Instruct', speculative_config=None, tokenizer='meta-llama/Meta-Llama-3-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=meta-llama/Meta-Llama-3-8B-Instruct)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/usr/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/usr/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/usr/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/usr/lib/python3.10/runpy.py", line 289, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/nvme0n1/ben/test/test.py", line 8, in <module>
    llm = LLM(
  File "/nvme0n1/ben/vllm/vllm/entrypoints/llm.py", line 144, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
  File "/nvme0n1/ben/vllm/vllm/engine/llm_engine.py", line 371, in from_engine_args
    engine = cls(
  File "/nvme0n1/ben/vllm/vllm/engine/llm_engine.py", line 223, in __init__
    self.model_executor = executor_class(
  File "/nvme0n1/ben/vllm/vllm/executor/distributed_gpu_executor.py", line 25, in __init__
    super().__init__(*args, **kwargs)
  File "/nvme0n1/ben/vllm/vllm/executor/executor_base.py", line 41, in __init__
    self._init_executor()
  File "/nvme0n1/ben/vllm/vllm/executor/multiproc_gpu_executor.py", line 48, in _init_executor
    self.workers = [
  File "/nvme0n1/ben/vllm/vllm/executor/multiproc_gpu_executor.py", line 49, in <listcomp>
    ProcessWorkerWrapper(
  File "/nvme0n1/ben/vllm/vllm/executor/multiproc_worker_utils.py", line 162, in __init__
    self.process.start()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/usr/lib/python3.10/multiprocessing/spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "/usr/lib/python3.10/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
ERROR 06-19 01:40:05 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 2334411 died, exit code: 1
INFO 06-19 01:40:05 multiproc_worker_utils.py:123] Killing local vLLM worker processes

For test-wrap.py being the following

import os
from vllm import LLM, SamplingParams

if __name__ == '__main__':

    prompts = ['<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful chatbot who always responds to requests.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWhat is ramen?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n']

    sampling_params = SamplingParams(temperature=0.0, max_tokens=512)

    llm = LLM(
        model="meta-llama/Meta-Llama-3-8B-Instruct",
        tensor_parallel_size=2,
        disable_log_stats=False,
    )

    outputs = llm.generate(prompts, sampling_params)

Running python test-wrap.py gives

(venv-vllm) ben@sakura-h100-3:/nvme0n1/ben/test$ python test-wrapped.py 
/nvme0n1/ben/venvs/venv-vllm/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
INFO 06-19 01:43:57 config.py:632] Defaulting to use mp for distributed inference
INFO 06-19 01:43:57 llm_engine.py:161] Initializing an LLM engine (v0.5.0.post1) with config: model='meta-llama/Meta-Llama-3-8B-Instruct', speculative_config=None, tokenizer='meta-llama/Meta-Llama-3-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=meta-llama/Meta-Llama-3-8B-Instruct)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:00 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:00 utils.py:672] Found nccl from library libnccl.so.2
INFO 06-19 01:44:00 utils.py:672] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:00 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 06-19 01:44:00 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 06-19 01:44:01 custom_all_reduce_utils.py:196] generating GPU P2P access cache in /home/ben/.config/vllm/gpu_p2p_access_cache_for_0,1.json
INFO 06-19 01:44:06 custom_all_reduce_utils.py:208] reading GPU P2P access cache from /home/ben/.config/vllm/gpu_p2p_access_cache_for_0,1.json
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:06 custom_all_reduce_utils.py:208] reading GPU P2P access cache from /home/ben/.config/vllm/gpu_p2p_access_cache_for_0,1.json
INFO 06-19 01:44:07 weight_utils.py:218] Using model weights format ['*.safetensors']
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:07 weight_utils.py:218] Using model weights format ['*.safetensors']
INFO 06-19 01:44:09 model_runner.py:160] Loading model weights took 7.4829 GB
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:10 model_runner.py:160] Loading model weights took 7.4829 GB
INFO 06-19 01:44:11 distributed_gpu_executor.py:56] # GPU blocks: 61915, # CPU blocks: 4096
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:12 model_runner.py:889] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:12 model_runner.py:893] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 06-19 01:44:12 model_runner.py:889] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 06-19 01:44:12 model_runner.py:893] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:16 custom_all_reduce.py:267] Registering 2275 cuda graph addresses
INFO 06-19 01:44:16 custom_all_reduce.py:267] Registering 2275 cuda graph addresses
INFO 06-19 01:44:16 model_runner.py:965] Graph capturing finished in 4 secs.
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:16 model_runner.py:965] Graph capturing finished in 4 secs.
Processed prompts: 100%|███████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.98s/it, est. speed input: 16.18 toks/s, output: 145.13 toks/s]
[rank0]:[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
/usr/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Note that there is a CUDA IPC Tensor Error and a Leaked Semaphore Objects error at the end, but at least the text generation task can be successfully completed.

This might be related to unprotected usage of python's concurrent.futures library, although the only place I could find vllm using this library is here.

[bkchang]

Just checked. For v0.4.3, default backend (ray) works, albeit with a small error [rank0]:[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] at the end.
If we set distributed_executor_backend='mp', it is broken with the same error described in the threads above.

[youkaichao]

cc @njhill

[youkaichao]

hi, folks, can you try to set an environment variable export VLLM_WORKER_MULTIPROC_METHOD=fork , and then run vllm again?

[jeejeelee]

hi, folks, can you try to set an environment variable export VLLM_WORKER_MULTIPROC_METHOD=fork , and then run vllm again?

I also encountered a similar error above, adding this environment variable resolved it.

[zixuzixu]

I attempted to set the environment variable export VLLM_WORKER_MULTIPROC_METHOD=fork as suggested and reran the VLLM application. Unfortunately, I'm still encountering errors. This time, a KeyError occurred in the multiprocessing.resource_tracker module, indicating a potential issue with process management under the fork start method.

The traceback highlights a removal operation on a missing key in a resource cache.

Here’s the relevant part of the error message:

(llm) z5327441@k091:/scratch/pbs.5466392.kman.restech.unsw.edu.au $ export VLLM_WORKER_MULTIPROC_METHOD=fork
(llm) z5327441@k091:/scratch/pbs.5466392.kman.restech.unsw.edu.au $ python test.py 
2024-06-19 20:27:18,261 INFO worker.py:1568 -- Connecting to existing Ray cluster at address: 10.197.40.91:6379...
2024-06-19 20:27:18,269 INFO worker.py:1744 -- Connected to Ray cluster. View the dashboard at 10.197.40.91:8265 
INFO 06-19 20:27:18 config.py:623] Defaulting to use mp for distributed inference
INFO 06-19 20:27:18 llm_engine.py:161] Initializing an LLM engine (v0.5.0.post1) with config: model='/scratch/pbs.5466392.kman.restech.unsw.edu.au/models/Meta-Llama-3-8B-Instruct', speculative_config=None, tokenizer='/scratch/pbs.5466392.kman.restech.unsw.edu.au/models/Meta-Llama-3-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/scratch/pbs.5466392.kman.restech.unsw.edu.au/models/Meta-Llama-3-8B-Instruct)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
(VllmWorkerProcess pid=3599698) INFO 06-19 20:27:20 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3599698) INFO 06-19 20:27:20 utils.py:637] Found nccl from library libnccl.so.2
INFO 06-19 20:27:20 utils.py:637] Found nccl from library libnccl.so.2
INFO 06-19 20:27:20 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3599698) INFO 06-19 20:27:20 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 06-19 20:27:20 custom_all_reduce_utils.py:170] generating GPU P2P access cache in /home/z5327441/.config/vllm/gpu_p2p_access_cache_for_7,6,5,4,3,2,1,0.json
2024-06-19 20:27:22,430 INFO worker.py:1568 -- Connecting to existing Ray cluster at address: 10.197.40.91:6379...
2024-06-19 20:27:22,438 INFO worker.py:1744 -- Connected to Ray cluster. View the dashboard at 10.197.40.91:8265 
2024-06-19 20:27:22,475 INFO worker.py:1568 -- Connecting to existing Ray cluster at address: 10.197.40.91:6379...
2024-06-19 20:27:22,483 INFO worker.py:1744 -- Connected to Ray cluster. View the dashboard at 10.197.40.91:8265 
INFO 06-19 20:27:22 config.py:623] Defaulting to use mp for distributed inference
INFO 06-19 20:27:22 llm_engine.py:161] Initializing an LLM engine (v0.5.0.post1) with config: model='/scratch/pbs.5466392.kman.restech.unsw.edu.au/models/Meta-Llama-3-8B-Instruct', speculative_config=None, tokenizer='/scratch/pbs.5466392.kman.restech.unsw.edu.au/models/Meta-Llama-3-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/scratch/pbs.5466392.kman.restech.unsw.edu.au/models/Meta-Llama-3-8B-Instruct)
INFO 06-19 20:27:22 config.py:623] Defaulting to use mp for distributed inference
INFO 06-19 20:27:22 llm_engine.py:161] Initializing an LLM engine (v0.5.0.post1) with config: model='/scratch/pbs.5466392.kman.restech.unsw.edu.au/models/Meta-Llama-3-8B-Instruct', speculative_config=None, tokenizer='/scratch/pbs.5466392.kman.restech.unsw.edu.au/models/Meta-Llama-3-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/scratch/pbs.5466392.kman.restech.unsw.edu.au/models/Meta-Llama-3-8B-Instruct)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
(VllmWorkerProcess pid=3600016) INFO 06-19 20:27:23 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3600019) INFO 06-19 20:27:23 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
INFO 06-19 20:27:23 utils.py:637] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3600016) INFO 06-19 20:27:23 utils.py:637] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3600016) INFO 06-19 20:27:23 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 06-19 20:27:23 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 06-19 20:27:23 utils.py:637] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3600019) INFO 06-19 20:27:23 utils.py:637] Found nccl from library libnccl.so.2
INFO 06-19 20:27:23 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3600019) INFO 06-19 20:27:23 pynccl.py:63] vLLM is using nccl==2.20.5
Traceback (most recent call last):
  File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
KeyError: '/psm_deecb814'
INFO 06-19 20:27:24 custom_all_reduce_utils.py:170] generating GPU P2P access cache in /home/z5327441/.config/vllm/gpu_p2p_access_cache_for_7,6,5,4,3,2,1,0.json
[rank0]: Traceback (most recent call last):
[rank0]:   File "<string>", line 1, in <module>
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
[rank0]:     exitcode = _main(fd, parent_sentinel)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
[rank0]:     prepare(preparation_data)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
[rank0]:     _fixup_main_from_path(data['init_main_from_path'])
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
[rank0]:     main_content = runpy.run_path(main_path,
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/runpy.py", line 289, in run_path
[rank0]:     return _run_module_code(code, init_globals, run_name,
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/runpy.py", line 96, in _run_module_code
[rank0]:     _run_code(code, mod_globals, init_globals,
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]:     exec(code, run_globals)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/test.py", line 13, in <module>
[rank0]:     llm = LLM(
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 144, in __init__
[rank0]:     self.llm_engine = LLMEngine.from_engine_args(
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 363, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 223, in __init__
[rank0]:     self.model_executor = executor_class(
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 25, in __init__
[rank0]:     super().__init__(*args, **kwargs)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 41, in __init__
[rank0]:     self._init_executor()
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 65, in _init_executor
[rank0]:     self._run_workers("init_device")
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 119, in _run_workers
[rank0]:     driver_worker_output = driver_worker_method(*args, **kwargs)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/worker/worker.py", line 115, in init_device
[rank0]:     init_worker_distributed_environment(self.parallel_config, self.rank,
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/worker/worker.py", line 357, in init_worker_distributed_environment
[rank0]:     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 655, in ensure_model_parallel_initialized
[rank0]:     initialize_model_parallel(tensor_model_parallel_size,
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 616, in initialize_model_parallel
[rank0]:     _TP = GroupCoordinator(
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 157, in __init__
[rank0]:     self.ca_comm = CustomAllreduce(
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 174, in __init__
[rank0]:     if not _can_p2p(rank, world_size):
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 78, in _can_p2p
[rank0]:     if not gpu_p2p_access_check(rank, i):
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/distributed/device_communicators/custom_all_reduce_utils.py", line 174, in gpu_p2p_access_check
[rank0]:     cache[f"{_i}->{_j}"] = can_actually_p2p(_i, _j)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/distributed/device_communicators/custom_all_reduce_utils.py", line 123, in can_actually_p2p
[rank0]:     pi.start()
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/process.py", line 121, in start
[rank0]:     self._popen = self._Popen(self)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
[rank0]:     return Popen(process_obj)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
[rank0]:     super().__init__(process_obj)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
[rank0]:     self._launch(process_obj)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 42, in _launch
[rank0]:     prep_data = spawn.get_preparation_data(process_obj._name)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 154, in get_preparation_data
[rank0]:     _check_not_importing_main()
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 134, in _check_not_importing_main
[rank0]:     raise RuntimeError('''
[rank0]: RuntimeError: 
[rank0]:         An attempt has been made to start a new process before the
[rank0]:         current process has finished its bootstrapping phase.

[rank0]:         This probably means that you are not using fork to start your
[rank0]:         child processes and you have forgotten to use the proper idiom
[rank0]:         in the main module:

[rank0]:             if __name__ == '__main__':
[rank0]:                 freeze_support()
[rank0]:                 ...

[rank0]:         The "freeze_support()" line can be omitted if the program
[rank0]:         is not going to be frozen to produce an executable.
Traceback (most recent call last):
  File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
KeyError: '/psm_2038dbcf'
INFO 06-19 20:27:24 custom_all_reduce_utils.py:170] generating GPU P2P access cache in /home/z5327441/.config/vllm/gpu_p2p_access_cache_for_7,6,5,4,3,2,1,0.json
[rank0]: Traceback (most recent call last):
[rank0]:   File "<string>", line 1, in <module>
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
[rank0]:     exitcode = _main(fd, parent_sentinel)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
[rank0]:     prepare(preparation_data)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
[rank0]:     _fixup_main_from_path(data['init_main_from_path'])
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
[rank0]:     main_content = runpy.run_path(main_path,
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/runpy.py", line 289, in run_path
[rank0]:     return _run_module_code(code, init_globals, run_name,
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/runpy.py", line 96, in _run_module_code
[rank0]:     _run_code(code, mod_globals, init_globals,
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]:     exec(code, run_globals)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/test.py", line 13, in <module>
[rank0]:     llm = LLM(
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 144, in __init__
[rank0]:     self.llm_engine = LLMEngine.from_engine_args(
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 363, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 223, in __init__
[rank0]:     self.model_executor = executor_class(
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 25, in __init__
[rank0]:     super().__init__(*args, **kwargs)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 41, in __init__
[rank0]:     self._init_executor()
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 65, in _init_executor
[rank0]:     self._run_workers("init_device")
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 119, in _run_workers
[rank0]:     driver_worker_output = driver_worker_method(*args, **kwargs)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/worker/worker.py", line 115, in init_device
[rank0]:     init_worker_distributed_environment(self.parallel_config, self.rank,
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/worker/worker.py", line 357, in init_worker_distributed_environment
[rank0]:     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 655, in ensure_model_parallel_initialized
[rank0]:     initialize_model_parallel(tensor_model_parallel_size,
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 616, in initialize_model_parallel
[rank0]:     _TP = GroupCoordinator(
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 157, in __init__
[rank0]:     self.ca_comm = CustomAllreduce(
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 174, in __init__
[rank0]:     if not _can_p2p(rank, world_size):
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 78, in _can_p2p
[rank0]:     if not gpu_p2p_access_check(rank, i):
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/distributed/device_communicators/custom_all_reduce_utils.py", line 174, in gpu_p2p_access_check
[rank0]:     cache[f"{_i}->{_j}"] = can_actually_p2p(_i, _j)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/site-packages/vllm/distributed/device_communicators/custom_all_reduce_utils.py", line 123, in can_actually_p2p
[rank0]:     pi.start()
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/process.py", line 121, in start
[rank0]:     self._popen = self._Popen(self)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
[rank0]:     return Popen(process_obj)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
[rank0]:     super().__init__(process_obj)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
[rank0]:     self._launch(process_obj)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 42, in _launch
[rank0]:     prep_data = spawn.get_preparation_data(process_obj._name)
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 154, in get_preparation_data
[rank0]:     _check_not_importing_main()
[rank0]:   File "/scratch/pbs.5466392.kman.restech.unsw.edu.au/miniforge3/envs/llm/lib/python3.10/multiprocessing/spawn.py", line 134, in _check_not_importing_main
[rank0]:     raise RuntimeError('''
[rank0]: RuntimeError: 
[rank0]:         An attempt has been made to start a new process before the
[rank0]:         current process has finished its bootstrapping phase.

[rank0]:         This probably means that you are not using fork to start your
[rank0]:         child processes and you have forgotten to use the proper idiom
[rank0]:         in the main module:

[rank0]:             if __name__ == '__main__':
[rank0]:                 freeze_support()
[rank0]:                 ...

[rank0]:         The "freeze_support()" line can be omitted if the program
[rank0]:         is not going to be frozen to produce an executable.
*** SIGTERM received at time=1718792844 on cpu 39 ***
PC: @     0x14ee9615645c  (unknown)  pthread_cond_wait@@GLIBC_2.3.2
    @     0x14ee9615acf0  (unknown)  (unknown)
[2024-06-19 20:27:24,847 E 3600016 3599783] logging.cc:343: *** SIGTERM received at time=1718792844 on cpu 39 ***
[2024-06-19 20:27:24,847 E 3600016 3599783] logging.cc:343: PC: @     0x14ee9615645c  (unknown)  pthread_cond_wait@@GLIBC_2.3.2
[2024-06-19 20:27:24,847 E 3600016 3599783] logging.cc:343:     @     0x14ee9615acf0  (unknown)  (unknown)
*** SIGTERM received at time=1718792844 on cpu 37 ***
PC: @     0x14bb61b8c45c  (unknown)  pthread_cond_wait@@GLIBC_2.3.2
    @     0x14bb61b90cf0  (unknown)  (unknown)
[2024-06-19 20:27:24,892 E 3600019 3599784] logging.cc:343: *** SIGTERM received at time=1718792844 on cpu 37 ***
[2024-06-19 20:27:24,892 E 3600019 3599784] logging.cc:343: PC: @     0x14bb61b8c45c  (unknown)  pthread_cond_wait@@GLIBC_2.3.2
[2024-06-19 20:27:24,892 E 3600019 3599784] logging.cc:343:     @     0x14bb61b90cf0  (unknown)  (unknown)

[youkaichao]

@zixuzixu two separate issues:

1. KeyError: '/psm_deecb814' , see [Bug][v0.5.0]: Benign error reported by Python multiprocessing resource_tracker #5468 . it should not crash your program.
2. An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This is a bug, and I think the latest code [distributed][misc] use fork by default for mp #5669 should solve the problem. Please have a try.

[bkchang]

For the latest commit in the main branch #5648 adding export VLLM_WORKER_MULTIPROC_METHOD=fork does not work:

(venv-vllm) ben@sakura-h100-3:/nvme0n1/ben/test$ export VLLM_WORKER_MULTIPROC_METHOD=fork
(venv-vllm) ben@sakura-h100-3:/nvme0n1/ben/test$ python test.py
/nvme0n1/ben/venvs/venv-vllm/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
INFO 06-20 01:29:02 llm_engine.py:164] Initializing an LLM engine (v0.5.0.post1) with config: model='meta-llama/Meta-Llama-3-8B-Instruct', speculative_config=None, tokenizer='meta-llama/Meta-Llama-3-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=meta-llama/Meta-Llama-3-8B-Instruct)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
(VllmWorkerProcess pid=3681069) INFO 06-20 01:29:07 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3681068) INFO 06-20 01:29:07 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3681067) INFO 06-20 01:29:07 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
INFO 06-20 01:29:08 utils.py:672] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3681068) INFO 06-20 01:29:08 utils.py:672] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3681067) INFO 06-20 01:29:08 utils.py:672] Found nccl from library libnccl.so.2
INFO 06-20 01:29:08 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3681069) INFO 06-20 01:29:08 utils.py:672] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3681068) INFO 06-20 01:29:08 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3681067) INFO 06-20 01:29:08 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3681069) INFO 06-20 01:29:08 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 06-20 01:29:11 custom_all_reduce_utils.py:196] generating GPU P2P access cache in /home/ben/.config/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
/nvme0n1/ben/venvs/venv-vllm/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
/nvme0n1/ben/venvs/venv-vllm/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
INFO 06-20 01:29:13 llm_engine.py:164] Initializing an LLM engine (v0.5.0.post1) with config: model='meta-llama/Meta-Llama-3-8B-Instruct', speculative_config=None, tokenizer='meta-llama/Meta-Llama-3-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=meta-llama/Meta-Llama-3-8B-Instruct)
INFO 06-20 01:29:13 llm_engine.py:164] Initializing an LLM engine (v0.5.0.post1) with config: model='meta-llama/Meta-Llama-3-8B-Instruct', speculative_config=None, tokenizer='meta-llama/Meta-Llama-3-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=meta-llama/Meta-Llama-3-8B-Instruct)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
(VllmWorkerProcess pid=3681346) INFO 06-20 01:29:17 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3681347) INFO 06-20 01:29:17 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3681350) INFO 06-20 01:29:17 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3681352) INFO 06-20 01:29:17 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3681345) INFO 06-20 01:29:17 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3681351) INFO 06-20 01:29:17 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3681352) INFO 06-20 01:29:18 utils.py:672] Found nccl from library libnccl.so.2
INFO 06-20 01:29:18 utils.py:672] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3681351) INFO 06-20 01:29:18 utils.py:672] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3681350) INFO 06-20 01:29:18 utils.py:672] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3681352) INFO 06-20 01:29:18 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 06-20 01:29:18 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3681351) INFO 06-20 01:29:18 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3681350) INFO 06-20 01:29:18 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3681345) INFO 06-20 01:29:18 utils.py:672] Found nccl from library libnccl.so.2
INFO 06-20 01:29:18 utils.py:672] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3681345) INFO 06-20 01:29:18 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 06-20 01:29:18 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3681346) INFO 06-20 01:29:18 utils.py:672] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3681347) INFO 06-20 01:29:18 utils.py:672] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3681346) INFO 06-20 01:29:18 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=3681347) INFO 06-20 01:29:18 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 06-20 01:29:23 custom_all_reduce_utils.py:196] generating GPU P2P access cache in /home/ben/.config/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
[rank0]: Traceback (most recent call last):
[rank0]:   File "<string>", line 1, in <module>
[rank0]:   File "/usr/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
[rank0]:     exitcode = _main(fd, parent_sentinel)
[rank0]:   File "/usr/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
[rank0]:     prepare(preparation_data)
[rank0]:   File "/usr/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
[rank0]:     _fixup_main_from_path(data['init_main_from_path'])
[rank0]:   File "/usr/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
[rank0]:     main_content = runpy.run_path(main_path,
[rank0]:   File "/usr/lib/python3.10/runpy.py", line 289, in run_path
[rank0]:     return _run_module_code(code, init_globals, run_name,
[rank0]:   File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
[rank0]:     _run_code(code, mod_globals, init_globals,
[rank0]:   File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]:     exec(code, run_globals)
[rank0]:   File "/nvme0n1/ben/test/test.py", line 10, in <module>
[rank0]:     llm = LLM(
[rank0]:   File "/nvme0n1/ben/vllm/vllm/entrypoints/llm.py", line 144, in __init__
[rank0]:     self.llm_engine = LLMEngine.from_engine_args(
[rank0]:   File "/nvme0n1/ben/vllm/vllm/engine/llm_engine.py", line 384, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/nvme0n1/ben/vllm/vllm/engine/llm_engine.py", line 230, in __init__
[rank0]:     self.model_executor = executor_class(
[rank0]:   File "/nvme0n1/ben/vllm/vllm/executor/distributed_gpu_executor.py", line 25, in __init__
[rank0]:     super().__init__(*args, **kwargs)
[rank0]:   File "/nvme0n1/ben/vllm/vllm/executor/executor_base.py", line 41, in __init__
[rank0]:     self._init_executor()
[rank0]:   File "/nvme0n1/ben/vllm/vllm/executor/multiproc_gpu_executor.py", line 68, in _init_executor
[rank0]:     self._run_workers("init_device")
[rank0]:   File "/nvme0n1/ben/vllm/vllm/executor/multiproc_gpu_executor.py", line 122, in _run_workers
[rank0]:     driver_worker_output = driver_worker_method(*args, **kwargs)
[rank0]:   File "/nvme0n1/ben/vllm/vllm/worker/worker.py", line 115, in init_device
[rank0]:     init_worker_distributed_environment(self.parallel_config, self.rank,
[rank0]:   File "/nvme0n1/ben/vllm/vllm/worker/worker.py", line 358, in init_worker_distributed_environment
[rank0]:     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
[rank0]:   File "/nvme0n1/ben/vllm/vllm/distributed/parallel_state.py", line 656, in ensure_model_parallel_initialized
[rank0]:     initialize_model_parallel(tensor_model_parallel_size,
[rank0]:   File "/nvme0n1/ben/vllm/vllm/distributed/parallel_state.py", line 617, in initialize_model_parallel
[rank0]:     _TP = GroupCoordinator(
[rank0]:   File "/nvme0n1/ben/vllm/vllm/distributed/parallel_state.py", line 158, in __init__
[rank0]:     self.ca_comm = CustomAllreduce(
[rank0]:   File "/nvme0n1/ben/vllm/vllm/distributed/device_communicators/custom_all_reduce.py", line 174, in __init__
[rank0]:     if not _can_p2p(rank, world_size):
[rank0]:   File "/nvme0n1/ben/vllm/vllm/distributed/device_communicators/custom_all_reduce.py", line 78, in _can_p2p
[rank0]:     if not gpu_p2p_access_check(rank, i):
[rank0]:   File "/nvme0n1/ben/vllm/vllm/distributed/device_communicators/custom_all_reduce_utils.py", line 201, in gpu_p2p_access_check
[rank0]:     result = can_actually_p2p(batch_src, batch_tgt)
[rank0]:   File "/nvme0n1/ben/vllm/vllm/distributed/device_communicators/custom_all_reduce_utils.py", line 138, in can_actually_p2p
[rank0]:     p_src.start()
[rank0]:   File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
[rank0]:     self._popen = self._Popen(self)
[rank0]:   File "/usr/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
[rank0]:     return Popen(process_obj)
[rank0]:   File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
[rank0]:     super().__init__(process_obj)
[rank0]:   File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
[rank0]:     self._launch(process_obj)
[rank0]:   File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 42, in _launch
[rank0]:     prep_data = spawn.get_preparation_data(process_obj._name)
[rank0]:   File "/usr/lib/python3.10/multiprocessing/spawn.py", line 154, in get_preparation_data
[rank0]:     _check_not_importing_main()
[rank0]:   File "/usr/lib/python3.10/multiprocessing/spawn.py", line 134, in _check_not_importing_main
[rank0]:     raise RuntimeError('''
[rank0]: RuntimeError: 
[rank0]:         An attempt has been made to start a new process before the
[rank0]:         current process has finished its bootstrapping phase.

[rank0]:         This probably means that you are not using fork to start your
[rank0]:         child processes and you have forgotten to use the proper idiom
[rank0]:         in the main module:

[rank0]:             if __name__ == '__main__':
[rank0]:                 freeze_support()
[rank0]:                 ...

[rank0]:         The "freeze_support()" line can be omitted if the program
[rank0]:         is not going to be frozen to produce an executable.
INFO 06-20 01:29:23 custom_all_reduce_utils.py:196] generating GPU P2P access cache in /home/ben/.config/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
[rank0]: Traceback (most recent call last):
[rank0]:   File "<string>", line 1, in <module>
[rank0]:   File "/usr/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
[rank0]:     exitcode = _main(fd, parent_sentinel)
[rank0]:   File "/usr/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
[rank0]:     prepare(preparation_data)
[rank0]:   File "/usr/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
[rank0]:     _fixup_main_from_path(data['init_main_from_path'])
[rank0]:   File "/usr/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
[rank0]:     main_content = runpy.run_path(main_path,
[rank0]:   File "/usr/lib/python3.10/runpy.py", line 289, in run_path
[rank0]:     return _run_module_code(code, init_globals, run_name,
[rank0]:   File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
[rank0]:     _run_code(code, mod_globals, init_globals,
[rank0]:   File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]:     exec(code, run_globals)
[rank0]:   File "/nvme0n1/ben/test/test.py", line 10, in <module>
[rank0]:     llm = LLM(
[rank0]:   File "/nvme0n1/ben/vllm/vllm/entrypoints/llm.py", line 144, in __init__
[rank0]:     self.llm_engine = LLMEngine.from_engine_args(
[rank0]:   File "/nvme0n1/ben/vllm/vllm/engine/llm_engine.py", line 384, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/nvme0n1/ben/vllm/vllm/engine/llm_engine.py", line 230, in __init__
[rank0]:     self.model_executor = executor_class(
[rank0]:   File "/nvme0n1/ben/vllm/vllm/executor/distributed_gpu_executor.py", line 25, in __init__
[rank0]:     super().__init__(*args, **kwargs)
[rank0]:   File "/nvme0n1/ben/vllm/vllm/executor/executor_base.py", line 41, in __init__
[rank0]:     self._init_executor()
[rank0]:   File "/nvme0n1/ben/vllm/vllm/executor/multiproc_gpu_executor.py", line 68, in _init_executor
[rank0]:     self._run_workers("init_device")
[rank0]:   File "/nvme0n1/ben/vllm/vllm/executor/multiproc_gpu_executor.py", line 122, in _run_workers
[rank0]:     driver_worker_output = driver_worker_method(*args, **kwargs)
[rank0]:   File "/nvme0n1/ben/vllm/vllm/worker/worker.py", line 115, in init_device
[rank0]:     init_worker_distributed_environment(self.parallel_config, self.rank,
[rank0]:   File "/nvme0n1/ben/vllm/vllm/worker/worker.py", line 358, in init_worker_distributed_environment
[rank0]:     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
[rank0]:   File "/nvme0n1/ben/vllm/vllm/distributed/parallel_state.py", line 656, in ensure_model_parallel_initialized
[rank0]:     initialize_model_parallel(tensor_model_parallel_size,
[rank0]:   File "/nvme0n1/ben/vllm/vllm/distributed/parallel_state.py", line 617, in initialize_model_parallel
[rank0]:     _TP = GroupCoordinator(
[rank0]:   File "/nvme0n1/ben/vllm/vllm/distributed/parallel_state.py", line 158, in __init__
[rank0]:     self.ca_comm = CustomAllreduce(
[rank0]:   File "/nvme0n1/ben/vllm/vllm/distributed/device_communicators/custom_all_reduce.py", line 174, in __init__
[rank0]:     if not _can_p2p(rank, world_size):
[rank0]:   File "/nvme0n1/ben/vllm/vllm/distributed/device_communicators/custom_all_reduce.py", line 78, in _can_p2p
[rank0]:     if not gpu_p2p_access_check(rank, i):
[rank0]:   File "/nvme0n1/ben/vllm/vllm/distributed/device_communicators/custom_all_reduce_utils.py", line 201, in gpu_p2p_access_check
[rank0]:     result = can_actually_p2p(batch_src, batch_tgt)
[rank0]:   File "/nvme0n1/ben/vllm/vllm/distributed/device_communicators/custom_all_reduce_utils.py", line 138, in can_actually_p2p
[rank0]:     p_src.start()
[rank0]:   File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
[rank0]:     self._popen = self._Popen(self)
[rank0]:   File "/usr/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
[rank0]:     return Popen(process_obj)
[rank0]:   File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
[rank0]:     super().__init__(process_obj)
[rank0]:   File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
[rank0]:     self._launch(process_obj)
[rank0]:   File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 42, in _launch
[rank0]:     prep_data = spawn.get_preparation_data(process_obj._name)
[rank0]:   File "/usr/lib/python3.10/multiprocessing/spawn.py", line 154, in get_preparation_data
[rank0]:     _check_not_importing_main()
[rank0]:   File "/usr/lib/python3.10/multiprocessing/spawn.py", line 134, in _check_not_importing_main
[rank0]:     raise RuntimeError('''
[rank0]: RuntimeError: 
[rank0]:         An attempt has been made to start a new process before the
[rank0]:         current process has finished its bootstrapping phase.

[rank0]:         This probably means that you are not using fork to start your
[rank0]:         child processes and you have forgotten to use the proper idiom
[rank0]:         in the main module:

[rank0]:             if __name__ == '__main__':
[rank0]:                 freeze_support()
[rank0]:                 ...

[rank0]:         The "freeze_support()" line can be omitted if the program
[rank0]:         is not going to be frozen to produce an executable.
ERROR 06-20 01:29:24 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 3681352 died, exit code: -15
INFO 06-20 01:29:24 multiproc_worker_utils.py:123] Killing local vLLM worker processes
ERROR 06-20 01:29:24 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 3681347 died, exit code: -15
INFO 06-20 01:29:24 multiproc_worker_utils.py:123] Killing local vLLM worker processes

For #5669, it works with or without export VLLM_WORKER_MULTIPROC_METHOD=fork, albeit with a small error after text generation is completed:

INFO 06-20 02:08:42 model_runner.py:965] Graph capturing finished in 6 secs.
Processed prompts: 100%|███████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.71s/it, est. speed input: 18.70 toks/s, output: 168.34 toks/s]
ERROR 06-20 02:08:45 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 3690565 died, exit code: -15
INFO 06-20 02:08:45 multiproc_worker_utils.py:123] Killing local vLLM worker processes

[youkaichao]

This is expected. #5669 contains fixes for the error you mentioned.

[zixuzixu]

After building from the latest code #5669, everything is working now! (I faced some challenges with setting the g++ version and the cudatoolkit version without sudo permissions, but I managed to resolve them.)

Thank you so much for your help! I appreciate your efforts. Should I close this issue?

[youkaichao]

you can keep it open until #5669 is merged. I have to fix some test failure there.

[yaronr]

Hi. Encountering the same issue. Is there a workaround?
VLLM_WORKER_MULTIPROC_METHOD=fork not working.