[Bug]: Qwen3 30b a3b awq not working with vllm docker v0.8.5.post1

[kulievvitaly]

Your current environment

docker v0.8.5.post1 ubuntu 24.04 rtx 3090ti

sudo docker run --ipc=host --log-opt max-size=10m --log-opt max-file=1 --rm -it --gpus '"device=0"' --env "VLLM_USE_MODELSCOPE=True" --env "VLLM_USE_V1=0" -p 8000:8000 --mount type=bind,source=/home/me/.cache,target=/root/.cache vllm/vllm-openai:v0.8.5 --model swift/Qwen3-30B-A3B-AWQ --max-model-len 4000 --dtype half --gpu-memory-utilization 0.92 --disable-log-requests --no-enable-prefix-caching -q awq --tensor-parallel-size 1

🐛 Describe the bug

sudo docker run --ipc=host --log-opt max-size=10m --log-opt max-file=1 --rm -it --gpus '"device=0"' --env "VLLM_USE_MODELSCOPE=True" --env "VLLM_USE_V1=0" -p 8000:8000 --mount type=bind,source=/home/me/.cache,target=/root/.cache vllm/vllm-openai:v0.8.5 --model swift/Qwen3-30B-A3B-AWQ --max-model-len 4000 --dtype half --gpu-memory-utilization 0.92 --disable-log-requests --no-enable-prefix-caching -q awq --tensor-parallel-size 1

me@am5:~$ sudo docker run --ipc=host --log-opt max-size=10m --log-opt max-file=1 --rm -it --gpus '"device=0"' --env "VLLM_USE_MODELSCOPE=True" --env "VLLM_USE_V1=0" -p 8000:8000 --mount type=bind,source=/home/m
e/.cache,target=/root/.cache vllm/vllm-openai:v0.8.5.post1 --model swift/Qwen3-30B-A3B-AWQ --max-model-len 4000 --dtype half --gpu-memory-utilization 0.92 --disable-log-requests --no-enable-prefix-caching -q aw
q --tensor-parallel-size 1
[sudo] password for me:
INFO 05-06 12:20:53 [init.py:239] Automatically detected platform cuda.
INFO 05-06 12:20:55 [api_server.py:1043] vLLM API server version 0.8.5.post1
INFO 05-06 12:20:55 [api_server.py:1044] args: Namespace(host=None, port=8000, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=[''], allowed_methods=[''],
allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_cert
s=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choi
ce=False, tool_call_parser=None, tool_parser_plugin='', model='swift/Qwen3-30B-A3B-AWQ', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer
revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config={}, use_tqdm_on_load=True, config_format=<ConfigFo
rmat.AUTO: 'auto'>, dtype='half', max_model_len=4000, guided_decoding_backend='xgrammar', reasoning_parser=None, logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_par
allel_size=1, tensor_parallel_size=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, gpu_m
emory_utilization=0.92, swap_space=4, kv_cache_dtype='auto', num_gpu_blocks_override=None, enable_prefix_caching=False, prefix_caching_hash_algo='builtin', cpu_offload_gb=0, calculate_kv_scales=False, disable_s
liding_window=False, use_v2_block_manager=True, seed=None, max_logprobs=20, disable_log_stats=False, quantization='awq', rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=False
, max_seq_len_to_capture=8192, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lor
a=None, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=
None, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', speculative_config=None, ignore_patterns=[], served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None
, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, max_num_batched_tokens=None, max_num_seqs=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill
token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, scheduling_policy='fcfs', enable_chunked_prefill=None, disable_
chunked_mm_input=False, scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_exten
sion_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, additional_config=None, enable_reasoning=False, disable_cascade_attn=False, disable_log_requests=True, max_log_le
n=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False)
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/swift/Qwen3-30B-A3B-AWQ
2025-05-06 12:20:59,705 - modelscope - WARNING - Using branch: master as version is unstable, use with caution
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/swift/Qwen3-30B-A3B-AWQ
2025-05-06 12:21:02,873 - modelscope - WARNING - Using branch: master as version is unstable, use with caution
INFO 05-06 12:21:08 [config.py:717] This model supports multiple tasks: {'generate', 'embed', 'score', 'reward', 'classify'}. Defaulting to 'generate'.
INFO 05-06 12:21:08 [awq_marlin.py:117] Detected that the model can run with awq_marlin, however you specified quantization=awq explicitly, so forcing awq. Use quantization=awq_marlin for faster inference
WARNING 05-06 12:21:08 [config.py:830] awq quantization is not fully optimized yet. The speed can be slower than non-quantized models.
INFO 05-06 12:21:08 [api_server.py:246] Started engine process with PID 81 INFO 05-06 12:21:10 [init.py:239] Automatically detected platform cuda. Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/swift/Qwen3-30B-A3B-AWQ
INFO 05-06 12:21:10 [llm_engine.py:240] Initializing a V0 LLM engine (v0.8.5.post1) with config: model='swift/Qwen3-30B-A3B-AWQ', speculative_config=None, tokenizer='swift/Qwen3-30B-A3B-AWQ', skip_tokenizer_ini
t=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=4000, download_dir=None, load_format=LoadFormat.AUTO,
tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=awq, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decodin
g_backend='xgrammar', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False),
seed=None, served_model_name=swift/Qwen3-30B-A3B-AWQ, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_p
reprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,15
2,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True,
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/swift/Qwen3-30B-A3B-AWQ
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/swift/Qwen3-30B-A3B-AWQ
2025-05-06 12:21:19,777 - modelscope - WARNING - Using branch: master as version is unstable, use with caution
INFO 05-06 12:21:20 [cuda.py:292] Using Flash Attention backend.
INFO 05-06 12:21:20 [parallel_state.py:1004] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 05-06 12:21:20 [model_runner.py:1108] Starting to load model swift/Qwen3-30B-A3B-AWQ...
WARNING 05-06 12:21:20 [utils.py:168] The model class Qwen3MoeForCausalLM has not defined packed_modules_mapping, this may lead to incorrect mapping of quantized or ignored modules
ERROR 05-06 12:21:20 [engine.py:448]
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 436, in run_mp_engine
engine = MQLLMEngine.from_vllm_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 128, in from_vllm_config
return cls(
^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 82, in init
self.engine = LLMEngine(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 275, in init
self.model_executor = executor_class(vllm_config=vllm_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 52, in init
self._init_executor()
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 47, in _init_executor
self.collective_rpc("load_model")
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
answer = run_method(self.driver_worker, method, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2456, in run_method
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 203, in load_model
self.model_runner.load_model()
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1111, in load_model
self.model = get_model(vllm_config=self.vllm_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/init.py", line 14, in get_model
return loader.load_model(vllm_config=vllm_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 452, in load_model
model = _initialize_model(vllm_config=vllm_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 133, in _initialize_model
return model_class(vllm_config=vllm_config, prefix=prefix)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 488, in init
self.model = Qwen3MoeModel(vllm_config=vllm_config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 151, in init
old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 334, in init
self.start_layer, self.end_layer, self.layers = make_layers(
^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 610, in make_layers
maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 336, in
lambda prefix: Qwen3MoeDecoderLayer(config=config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 278, in init
self.mlp = Qwen3MoeSparseMoeBlock(config=config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 113, in init
self.experts = FusedMoE(num_experts=config.num_experts,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/layer.py", line 499, in init
assert self.quant_method is not None
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
Process SpawnProcess-1:
Traceback (most recent call last):
File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 450, in run_mp_engine
raise e
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 436, in run_mp_engine
engine = MQLLMEngine.from_vllm_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 128, in from_vllm_config
return cls(
^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 82, in init
self.engine = LLMEngine(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 275, in init
self.model_executor = executor_class(vllm_config=vllm_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 52, in init
self._init_executor()
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 47, in _init_executor
self.collective_rpc("load_model")
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
answer = run_method(self.driver_worker, method, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2456, in run_method
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 203, in load_model
self.model_runner.load_model()
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1111, in load_model
self.model = get_model(vllm_config=self.vllm_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/init.py", line 14, in get_model
return loader.load_model(vllm_config=vllm_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 452, in load_model
model = _initialize_model(vllm_config=vllm_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 133, in _initialize_model
return model_class(vllm_config=vllm_config, prefix=prefix)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 488, in init
self.model = Qwen3MoeModel(vllm_config=vllm_config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 151, in init
old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 334, in init
self.start_layer, self.end_layer, self.layers = make_layers(
^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 610, in make_layers
maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 336, in
lambda prefix: Qwen3MoeDecoderLayer(config=config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 278, in init
self.mlp = Qwen3MoeSparseMoeBlock(config=config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 113, in init
self.experts = FusedMoE(num_experts=config.num_experts,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/layer.py", line 499, in init
assert self.quant_method is not None
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
[rank0]:[W506 12:21:20.872054709 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/
docs/stable/distributed.html#shutdown (function operator())
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1130, in
uvloop.run(run_server(args))
File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 109, in run
return __asyncio.run(
^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 61, in wrapper
return await main
^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1078, in run_server
async with build_async_engine_client(args) as engine_client:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 269, in build_async_engine_client_from_engine_args
raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[mehamednews]

I too am experiencing a persistent CUDA kernel error when trying to run Qwen3 32B AWQ.

Error Message

RuntimeError: CUDA error: no kernel image is available for execution on the device

What I've Tried

I've attempted multiple approaches to resolve this issue:

• Built the Docker image from source using the latest commit.
• Tested different argument configurations:
  • AWQ vs AWQ Marling
  • Engine v0 vs v1
  • xformers vs flash_attention v2
  • Enabling eager mode

None of these solutions have resolved the error.

(base) ➜  qwen3 docker compose up     
[+] Running 1/1
 ✔ Container qwen3-vllm-1  Recreated                                                                                                                                                      0.2s 
Attaching to vllm-1
vllm-1  | INFO 05-06 22:32:45 [__init__.py:239] Automatically detected platform cuda.
vllm-1  | INFO 05-06 22:32:49 [api_server.py:1042] vLLM API server version 0.8.5.dev469+g999328be0.d20250506
vllm-1  | INFO 05-06 22:32:49 [api_server.py:1043] args: Namespace(host=None, port=8000, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='Qwen/Qwen3-32B-AWQ', task='auto', tokenizer=None, tokenizer_mode='auto', trust_remote_code=True, dtype='auto', seed=None, hf_config_path=None, allowed_local_media_path='', revision=None, code_revision=None, rope_scaling={}, rope_theta=None, tokenizer_revision=None, max_model_len=None, quantization=None, enforce_eager=True, max_seq_len_to_capture=8192, max_logprobs=20, disable_sliding_window=False, disable_cascade_attn=False, skip_tokenizer_init=False, enable_prompt_embeds=False, served_model_name=None, disable_async_output_proc=False, config_format='auto', hf_token=None, hf_overrides={}, override_neuron_config={}, override_pooler_config=None, logits_processor_pattern=None, generation_config='auto', override_generation_config={}, enable_sleep_mode=False, model_impl='auto', load_format='auto', download_dir=None, model_loader_extra_config={}, ignore_patterns=None, use_tqdm_on_load=True, qlora_adapter_name_or_path=None, pt_load_map_location='cpu', guided_decoding_backend='auto', guided_decoding_disable_fallback=False, guided_decoding_disable_any_whitespace=False, guided_decoding_disable_additional_properties=False, enable_reasoning=True, reasoning_parser='deepseek_r1', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, worker_cls='auto', worker_extension_cls='', block_size=None, gpu_memory_utilization=0.9, swap_space=4, kv_cache_dtype='auto', num_gpu_blocks_override=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', cpu_offload_gb=0, calculate_kv_scales=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=None, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=None, max_prompt_adapters=1, max_prompt_adapter_token=0, device='cuda', speculative_config=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, max_num_batched_tokens=None, max_num_seqs=None, max_num_partial_prefills=1, max_long_partial_prefills=1, cuda_graph_sizes=[512], long_prefill_token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, scheduling_policy='fcfs', enable_chunked_prefill=None, disable_chunked_mm_input=False, scheduler_cls='vllm.core.scheduler.Scheduler', compilation_config=None, kv_transfer_config=None, kv_events_config=None, additional_config=None, use_v2_block_manager=True, disable_log_stats=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False)
vllm-1  | INFO 05-06 22:32:54 [config.py:752] This model supports multiple tasks: {'score', 'classify', 'reward', 'embed', 'generate'}. Defaulting to 'generate'.
vllm-1  | INFO 05-06 22:32:54 [awq_marlin.py:115] The model is convertible to awq_marlin during runtime. Using awq_marlin kernel.
vllm-1  | INFO 05-06 22:32:54 [config.py:2057] Chunked prefill is enabled with max_num_batched_tokens=2048.
vllm-1  | WARNING 05-06 22:32:54 [cuda.py:93] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
vllm-1  | INFO 05-06 22:32:56 [__init__.py:239] Automatically detected platform cuda.
vllm-1  | INFO 05-06 22:32:58 [core.py:59] Initializing a V1 LLM engine (v0.8.5.dev469+g999328be0.d20250506) with config: model='Qwen/Qwen3-32B-AWQ', speculative_config=None, tokenizer='Qwen/Qwen3-32B-AWQ', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=40960, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=awq_marlin, enforce_eager=True, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend='deepseek_r1'), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=None, served_model_name=Qwen/Qwen3-32B-AWQ, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[],"max_capture_size":0}
vllm-1  | WARNING 05-06 22:32:58 [utils.py:2540] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7a608cbebf80>
vllm-1  | INFO 05-06 22:32:59 [parallel_state.py:1004] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
vllm-1  | INFO 05-06 22:32:59 [cuda.py:221] Using Flash Attention backend on V1 engine.
vllm-1  | INFO 05-06 22:32:59 [topk_topp_sampler.py:59] Using FlashInfer for top-p & top-k sampling.
vllm-1  | INFO 05-06 22:32:59 [gpu_model_runner.py:1335] Starting to load model Qwen/Qwen3-32B-AWQ...
vllm-1  | INFO 05-06 22:32:59 [weight_utils.py:265] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:00<00:01,  1.65it/s]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:01<00:01,  1.55it/s]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:01<00:00,  1.56it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:02<00:00,  1.55it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:02<00:00,  1.56it/s]
vllm-1  | 
vllm-1  | INFO 05-06 22:33:02 [loader.py:459] Loading weights took 2.66 seconds
vllm-1  | ERROR 05-06 22:33:02 [core.py:400] EngineCore failed to start.
vllm-1  | ERROR 05-06 22:33:02 [core.py:400] Traceback (most recent call last):
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 391, in run_engine_core
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]     engine_core = EngineCoreProc(*args, **kwargs)
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 333, in __init__
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]     super().__init__(vllm_config, executor_class, log_stats,
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 65, in __init__
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]     self.model_executor = executor_class(vllm_config)
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 52, in __init__
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]     self._init_executor()
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 47, in _init_executor
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]     self.collective_rpc("load_model")
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]     answer = run_method(self.driver_worker, method, args, kwargs)
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2474, in run_method
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]     return func(*args, **kwargs)
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]            ^^^^^^^^^^^^^^^^^^^^^
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 162, in load_model
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]     self.model_runner.load_model()
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1338, in load_model
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]     self.model = get_model(vllm_config=self.vllm_config)
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/__init__.py", line 14, in get_model
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]     return loader.load_model(vllm_config=vllm_config)
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 472, in load_model
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]     _process_weights_after_loading(model, model_config, target_device)
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 180, in _process_weights_after_loading
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]     quant_method.process_weights_after_loading(module)
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/awq_marlin.py", line 282, in process_weights_after_loading
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]     marlin_scales = marlin_permute_scales(
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]                     ^^^^^^^^^^^^^^^^^^^^^^
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/marlin_utils.py", line 239, in marlin_permute_scales
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]     s = s.reshape((-1, len(scale_perm)))[:, scale_perm]
vllm-1  | ERROR 05-06 22:33:02 [core.py:400]         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
vllm-1  | ERROR 05-06 22:33:02 [core.py:400] RuntimeError: CUDA error: no kernel image is available for execution on the device
vllm-1  | ERROR 05-06 22:33:02 [core.py:400] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
vllm-1  | ERROR 05-06 22:33:02 [core.py:400] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
vllm-1  | ERROR 05-06 22:33:02 [core.py:400] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
vllm-1  | ERROR 05-06 22:33:02 [core.py:400] 
vllm-1  | Process EngineCore_0:
vllm-1  | Traceback (most recent call last):
vllm-1  |   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
vllm-1  |     self.run()
vllm-1  |   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
vllm-1  |     self._target(*self._args, **self._kwargs)
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 404, in run_engine_core
vllm-1  |     raise e
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 391, in run_engine_core
vllm-1  |     engine_core = EngineCoreProc(*args, **kwargs)
vllm-1  |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 333, in __init__
vllm-1  |     super().__init__(vllm_config, executor_class, log_stats,
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 65, in __init__
vllm-1  |     self.model_executor = executor_class(vllm_config)
vllm-1  |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 52, in __init__
vllm-1  |     self._init_executor()
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 47, in _init_executor
vllm-1  |     self.collective_rpc("load_model")
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
vllm-1  |     answer = run_method(self.driver_worker, method, args, kwargs)
vllm-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2474, in run_method
vllm-1  |     return func(*args, **kwargs)
vllm-1  |            ^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 162, in load_model
vllm-1  |     self.model_runner.load_model()
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1338, in load_model
vllm-1  |     self.model = get_model(vllm_config=self.vllm_config)
vllm-1  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/__init__.py", line 14, in get_model
vllm-1  |     return loader.load_model(vllm_config=vllm_config)
vllm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 472, in load_model
vllm-1  |     _process_weights_after_loading(model, model_config, target_device)
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 180, in _process_weights_after_loading
vllm-1  |     quant_method.process_weights_after_loading(module)
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/awq_marlin.py", line 282, in process_weights_after_loading
vllm-1  |     marlin_scales = marlin_permute_scales(
vllm-1  |                     ^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/marlin_utils.py", line 239, in marlin_permute_scales
vllm-1  |     s = s.reshape((-1, len(scale_perm)))[:, scale_perm]
vllm-1  |         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
vllm-1  | RuntimeError: CUDA error: no kernel image is available for execution on the device
vllm-1  | CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
vllm-1  | For debugging consider passing CUDA_LAUNCH_BLOCKING=1
vllm-1  | Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
vllm-1  | 
vllm-1  | [rank0]:[W506 22:33:03.582829614 ProcessGroupNCCL.cpp:1476] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
vllm-1  | Traceback (most recent call last):
vllm-1  |   File "<frozen runpy>", line 198, in _run_module_as_main
vllm-1  |   File "<frozen runpy>", line 88, in _run_code
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1129, in <module>
vllm-1  |     uvloop.run(run_server(args))
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run
vllm-1  |     return __asyncio.run(
vllm-1  |            ^^^^^^^^^^^^^^
vllm-1  |   File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
vllm-1  |     return runner.run(main)
vllm-1  |            ^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
vllm-1  |     return self._loop.run_until_complete(task)
vllm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper
vllm-1  |     return await main
vllm-1  |            ^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1077, in run_server
vllm-1  |     async with build_async_engine_client(args) as engine_client:
vllm-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
vllm-1  |     return await anext(self.gen)
vllm-1  |            ^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
vllm-1  |     async with build_async_engine_client_from_engine_args(
vllm-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
vllm-1  |     return await anext(self.gen)
vllm-1  |            ^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 178, in build_async_engine_client_from_engine_args
vllm-1  |     async_llm = AsyncLLM.from_vllm_config(
vllm-1  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 151, in from_vllm_config
vllm-1  |     return cls(
vllm-1  |            ^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 118, in __init__
vllm-1  |     self.engine_core = core_client_class(
vllm-1  |                        ^^^^^^^^^^^^^^^^^^
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 649, in __init__
vllm-1  |     super().__init__(
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 400, in __init__
vllm-1  |     self._wait_for_engine_startup()
vllm-1  |   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 432, in _wait_for_engine_startup
vllm-1  |     raise RuntimeError("Engine core initialization failed. "
vllm-1  | RuntimeError: Engine core initialization failed. See root cause above.

services:
  vllm:
    build:
      dockerfile: docker/Dockerfile
      context: vllm
      args:
        max_jobs: 48
    command: [ "--model", "Qwen/Qwen3-32B-AWQ", "--trust-remote-code", "--enable-reasoning", "--reasoning-parser", "deepseek_r1", "--device", "cuda", "--enforce-eager" ]
    ports:
      - "8000:8000"
    environment:
      - VLLM_USE_V1=1
      - VLLM_MAX_SIZE_MB=2048
      - NVIDIA_VISIBLE_DEVICES=0
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
      - VLLM_FLASH_ATTN_VERSION=2
      - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [ gpu ]
    devices:
      - /dev/nvidia0:/dev/nvidia0
      - /dev/nvidiactl:/dev/nvidiactl
      - /dev/nvidia-uvm:/dev/nvidia-uvm
      - /dev/nvidia-uvm-tools:/dev/nvidia-uvm-tools
    runtime: nvidia
    volumes:
      - ~/.cache:/root/.cache
      - ./model_cache:/models
    ipc: host
    privileged: true
    # restart: unless-stopped

(base) ➜  qwen3 nvidia-smi
Wed May  7 07:38:02 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20             Driver Version: 570.133.20     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5090        Off |   00000000:01:00.0 Off |                  N/A |
|  0%   34C    P8              2W /  575W |      43MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

(base) ➜  qwen3 nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Feb_21_20:23:50_PST_2025
Cuda compilation tools, release 12.8, V12.8.93
Build cuda_12.8.r12.8/compiler.35583870_0

[pan003]

The same question.

(VllmWorkerProcess pid=811) WARNING 05-07 15:47:38 [utils.py:168] The model class Qwen3MoeForCausalLM has not defined packed_modules_mapping, this may lead to incorrect mapping of quantized or ignored modules
ERROR 05-07 15:47:38 [engine.py:448]
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 436, in run_mp_engine
engine = MQLLMEngine.from_vllm_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 128, in from_vllm_config
return cls(
^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 82, in init
self.engine = LLMEngine(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 275, in init
self.model_executor = executor_class(vllm_config=vllm_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 286, in init
super().init(*args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 52, in init
self._init_executor()
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/mp_distributed_executor.py", line 125, in _init_executor
self._run_workers("load_model",
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/mp_distributed_executor.py", line 185, in _run_workers
driver_worker_output = run_method(self.driver_worker, sent_method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2456, in run_method

env:
vllm/vllm-openai:v0.8.5.post1
Linux vllm246 6.5.0-18-generic #18~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 7 11:40:03 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

root@vllm246:/data/models/Qwen3-30B-A3B-AWQ# nvidia-smi 
Wed May  7 17:58:53 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03             Driver Version: 550.144.03     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Quadro RTX 6000                Off |   00000000:3B:00.0 Off |                    0 |
| N/A   35C    P8             15W /  250W |       1MiB /  23040MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  Quadro RTX 6000                Off |   00000000:5E:00.0 Off |                    0 |
| N/A   42C    P8             24W /  250W |       1MiB /  23040MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

[ziozzang]

same issue.

[bi1101]

I too am experiencing a persistent CUDA kernel error when trying to run Qwen3 32B AWQ.

Error Message

RuntimeError: CUDA error: no kernel image is available for execution on the device

What I've Tried

I've attempted multiple approaches to resolve this issue:

• Built the Docker image from source using the latest commit.

• Tested different argument configurations:

  • AWQ vs AWQ Marling
  • Engine v0 vs v1
  • xformers vs flash_attention v2
  • Enabling eager mode

None of these solutions have resolved the error.

(base) ➜ qwen3 docker compose up
[+] Running 1/1
✔ Container qwen3-vllm-1 Recreated 0.2s
Attaching to vllm-1
vllm-1 | INFO 05-06 22:32:45 [init.py:239] Automatically detected platform cuda.
vllm-1 | INFO 05-06 22:32:49 [api_server.py:1042] vLLM API server version 0.8.5.dev469+g999328be0.d20250506
vllm-1 | INFO 05-06 22:32:49 [api_server.py:1043] args: Namespace(host=None, port=8000, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=[''], allowed_methods=[''], allowed_headers=[''], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='Qwen/Qwen3-32B-AWQ', task='auto', tokenizer=None, tokenizer_mode='auto', trust_remote_code=True, dtype='auto', seed=None, hf_config_path=None, allowed_local_media_path='', revision=None, code_revision=None, rope_scaling={}, rope_theta=None, tokenizer_revision=None, max_model_len=None, quantization=None, enforce_eager=True, max_seq_len_to_capture=8192, max_logprobs=20, disable_sliding_window=False, disable_cascade_attn=False, skip_tokenizer_init=False, enable_prompt_embeds=False, served_model_name=None, disable_async_output_proc=False, config_format='auto', hf_token=None, hf_overrides={}, override_neuron_config={}, override_pooler_config=None, logits_processor_pattern=None, generation_config='auto', override_generation_config={}, enable_sleep_mode=False, model_impl='auto', load_format='auto', download_dir=None, model_loader_extra_config={}, ignore_patterns=None, use_tqdm_on_load=True, qlora_adapter_name_or_path=None, pt_load_map_location='cpu', guided_decoding_backend='auto', guided_decoding_disable_fallback=False, guided_decoding_disable_any_whitespace=False, guided_decoding_disable_additional_properties=False, enable_reasoning=True, reasoning_parser='deepseek_r1', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, worker_cls='auto', worker_extension_cls='', block_size=None, gpu_memory_utilization=0.9, swap_space=4, kv_cache_dtype='auto', num_gpu_blocks_override=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', cpu_offload_gb=0, calculate_kv_scales=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=None, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=None, max_prompt_adapters=1, max_prompt_adapter_token=0, device='cuda', speculative_config=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, max_num_batched_tokens=None, max_num_seqs=None, max_num_partial_prefills=1, max_long_partial_prefills=1, cuda_graph_sizes=[512], long_prefill_token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, scheduling_policy='fcfs', enable_chunked_prefill=None, disable_chunked_mm_input=False, scheduler_cls='vllm.core.scheduler.Scheduler', compilation_config=None, kv_transfer_config=None, kv_events_config=None, additional_config=None, use_v2_block_manager=True, disable_log_stats=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False)
vllm-1 | INFO 05-06 22:32:54 [config.py:752] This model supports multiple tasks: {'score', 'classify', 'reward', 'embed', 'generate'}. Defaulting to 'generate'.
vllm-1 | INFO 05-06 22:32:54 [awq_marlin.py:115] The model is convertible to awq_marlin during runtime. Using awq_marlin kernel.
vllm-1 | INFO 05-06 22:32:54 [config.py:2057] Chunked prefill is enabled with max_num_batched_tokens=2048.
vllm-1 | WARNING 05-06 22:32:54 [cuda.py:93] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
vllm-1 | INFO 05-06 22:32:56 [init.py:239] Automatically detected platform cuda.
vllm-1 | INFO 05-06 22:32:58 [core.py:59] Initializing a V1 LLM engine (v0.8.5.dev469+g999328be0.d20250506) with config: model='Qwen/Qwen3-32B-AWQ', speculative_config=None, tokenizer='Qwen/Qwen3-32B-AWQ', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=40960, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=awq_marlin, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend='deepseek_r1'), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=None, served_model_name=Qwen/Qwen3-32B-AWQ, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[],"max_capture_size":0}
vllm-1 | WARNING 05-06 22:32:58 [utils.py:2540] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7a608cbebf80>
vllm-1 | INFO 05-06 22:32:59 [parallel_state.py:1004] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
vllm-1 | INFO 05-06 22:32:59 [cuda.py:221] Using Flash Attention backend on V1 engine.
vllm-1 | INFO 05-06 22:32:59 [topk_topp_sampler.py:59] Using FlashInfer for top-p & top-k sampling.
vllm-1 | INFO 05-06 22:32:59 [gpu_model_runner.py:1335] Starting to load model Qwen/Qwen3-32B-AWQ...
vllm-1 | INFO 05-06 22:32:59 [weight_utils.py:265] Using model weights format ['.safetensors']
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:01, 1.65it/s]
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:01<00:01, 1.55it/s]
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:01<00:00, 1.56it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:02<00:00, 1.55it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:02<00:00, 1.56it/s]
vllm-1 |
vllm-1 | INFO 05-06 22:33:02 [loader.py:459] Loading weights took 2.66 seconds
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] EngineCore failed to start.
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] Traceback (most recent call last):
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 391, in run_engine_core
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] engine_core = EngineCoreProc(*args, **kwargs)
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 333, in init
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] super().init(vllm_config, executor_class, log_stats,
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 65, in init
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] self.model_executor = executor_class(vllm_config)
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 52, in init
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] self._init_executor()
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 47, in _init_executor
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] self.collective_rpc("load_model")
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] answer = run_method(self.driver_worker, method, args, kwargs)
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2474, in run_method
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] return func(*args, **kwargs)
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] ^^^^^^^^^^^^^^^^^^^^^
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 162, in load_model
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] self.model_runner.load_model()
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1338, in load_model
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] self.model = get_model(vllm_config=self.vllm_config)
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/init.py", line 14, in get_model
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] return loader.load_model(vllm_config=vllm_config)
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 472, in load_model
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] _process_weights_after_loading(model, model_config, target_device)
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 180, in _process_weights_after_loading
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] quant_method.process_weights_after_loading(module)
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/awq_marlin.py", line 282, in process_weights_after_loading
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] marlin_scales = marlin_permute_scales(
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] ^^^^^^^^^^^^^^^^^^^^^^
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/marlin_utils.py", line 239, in marlin_permute_scales
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] s = s.reshape((-1, len(scale_perm)))[:, scale_perm]
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] RuntimeError: CUDA error: no kernel image is available for execution on the device
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
vllm-1 | ERROR 05-06 22:33:02 [core.py:400] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
vllm-1 | ERROR 05-06 22:33:02 [core.py:400]
vllm-1 | Process EngineCore_0:
vllm-1 | Traceback (most recent call last):
vllm-1 | File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
vllm-1 | self.run()
vllm-1 | File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
vllm-1 | self._target(*self._args, **self._kwargs)
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 404, in run_engine_core
vllm-1 | raise e
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 391, in run_engine_core
vllm-1 | engine_core = EngineCoreProc(*args, **kwargs)
vllm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 333, in init
vllm-1 | super().init(vllm_config, executor_class, log_stats,
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 65, in init
vllm-1 | self.model_executor = executor_class(vllm_config)
vllm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 52, in init
vllm-1 | self._init_executor()
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 47, in _init_executor
vllm-1 | self.collective_rpc("load_model")
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
vllm-1 | answer = run_method(self.driver_worker, method, args, kwargs)
vllm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2474, in run_method
vllm-1 | return func(*args, **kwargs)
vllm-1 | ^^^^^^^^^^^^^^^^^^^^^
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 162, in load_model
vllm-1 | self.model_runner.load_model()
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1338, in load_model
vllm-1 | self.model = get_model(vllm_config=self.vllm_config)
vllm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/init.py", line 14, in get_model
vllm-1 | return loader.load_model(vllm_config=vllm_config)
vllm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 472, in load_model
vllm-1 | _process_weights_after_loading(model, model_config, target_device)
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 180, in _process_weights_after_loading
vllm-1 | quant_method.process_weights_after_loading(module)
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/awq_marlin.py", line 282, in process_weights_after_loading
vllm-1 | marlin_scales = marlin_permute_scales(
vllm-1 | ^^^^^^^^^^^^^^^^^^^^^^
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/marlin_utils.py", line 239, in marlin_permute_scales
vllm-1 | s = s.reshape((-1, len(scale_perm)))[:, scale_perm]
vllm-1 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
vllm-1 | RuntimeError: CUDA error: no kernel image is available for execution on the device
vllm-1 | CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
vllm-1 | For debugging consider passing CUDA_LAUNCH_BLOCKING=1
vllm-1 | Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
vllm-1 |
vllm-1 | [rank0]:[W506 22:33:03.582829614 ProcessGroupNCCL.cpp:1476] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
vllm-1 | Traceback (most recent call last):
vllm-1 | File "", line 198, in _run_module_as_main
vllm-1 | File "", line 88, in _run_code
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1129, in
vllm-1 | uvloop.run(run_server(args))
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 109, in run
vllm-1 | return __asyncio.run(
vllm-1 | ^^^^^^^^^^^^^^
vllm-1 | File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
vllm-1 | return runner.run(main)
vllm-1 | ^^^^^^^^^^^^^^^^
vllm-1 | File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
vllm-1 | return self._loop.run_until_complete(task)
vllm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1 | File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 61, in wrapper
vllm-1 | return await main
vllm-1 | ^^^^^^^^^^
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1077, in run_server
vllm-1 | async with build_async_engine_client(args) as engine_client:
vllm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1 | File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
vllm-1 | return await anext(self.gen)
vllm-1 | ^^^^^^^^^^^^^^^^^^^^^
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
vllm-1 | async with build_async_engine_client_from_engine_args(
vllm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1 | File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
vllm-1 | return await anext(self.gen)
vllm-1 | ^^^^^^^^^^^^^^^^^^^^^
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 178, in build_async_engine_client_from_engine_args
vllm-1 | async_llm = AsyncLLM.from_vllm_config(
vllm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 151, in from_vllm_config
vllm-1 | return cls(
vllm-1 | ^^^^
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 118, in init
vllm-1 | self.engine_core = core_client_class(
vllm-1 | ^^^^^^^^^^^^^^^^^^
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 649, in init
vllm-1 | super().init(
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 400, in init
vllm-1 | self._wait_for_engine_startup()
vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 432, in _wait_for_engine_startup
vllm-1 | raise RuntimeError("Engine core initialization failed. "
vllm-1 | RuntimeError: Engine core initialization failed. See root cause above.
services:
vllm:
build:
dockerfile: docker/Dockerfile
context: vllm
args:
max_jobs: 48
command: [ "--model", "Qwen/Qwen3-32B-AWQ", "--trust-remote-code", "--enable-reasoning", "--reasoning-parser", "deepseek_r1", "--device", "cuda", "--enforce-eager" ]
ports:
- "8000:8000"
environment:
- VLLM_USE_V1=1
- VLLM_MAX_SIZE_MB=2048
- NVIDIA_VISIBLE_DEVICES=0
- NVIDIA_DRIVER_CAPABILITIES=compute,utility
- VLLM_FLASH_ATTN_VERSION=2
- VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [ gpu ]
devices:
- /dev/nvidia0:/dev/nvidia0
- /dev/nvidiactl:/dev/nvidiactl
- /dev/nvidia-uvm:/dev/nvidia-uvm
- /dev/nvidia-uvm-tools:/dev/nvidia-uvm-tools
runtime: nvidia
volumes:
- ~/.cache:/root/.cache
- ./model_cache:/models
ipc: host
privileged: true
# restart: unless-stopped
(base) ➜ qwen3 nvidia-smi
Wed May 7 07:38:02 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20 Driver Version: 570.133.20 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 5090 Off | 00000000:01:00.0 Off | N/A |
| 0% 34C P8 2W / 575W | 43MiB / 32607MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
(base) ➜ qwen3 nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Feb_21_20:23:50_PST_2025
Cuda compilation tools, release 12.8, V12.8.93
Build cuda_12.8.r12.8/compiler.35583870_0

Your docker compose is not compatible with RTX 5090, you would have to add torch_cuda_arch_list: "12.0" for 5090

here's the full docker compose that works on my RTX 5090

vllm-Qwen3-30B-A3B-AWQ:
    container_name: vllm-Qwen3-30B-A3B-AWQ
    image: vllm-cu128
    build:
      dockerfile: docker/Dockerfile
      context: vllm
      args:
        max_jobs: 16
        RUN_WHEEL_CHECK: "false"
        torch_cuda_arch_list: "7.0 7.5 8.0 8.6 8.9 9.0+PTX 12.0" # Should not be capitalized
    command: [ "--model", "cognitivecomputations/Qwen3-30B-A3B-AWQ", "--trust-remote-code", "--enable-reasoning", "--reasoning-parser", "deepseek_r1"]
    ports:
      - "8000:8000"
    environment:
      - VLLM_USE_V1=1
      - VLLM_MAX_SIZE_MB=10240
      - VLLM_FLASH_ATTN_VERSION=2
      - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
      # - HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
    # runtime: nvidia
    volumes:
      - vllm_cache:/root/.cache
      - model_data:/models
    # ipc: host
    # privileged: true
    # restart: unless-stopped

Hopes that helps. The build time ~ 6 hours for me. Some build processes cannot use multiple cores.

Update: To get FP8 model working #16901 (comment)

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!