custom vllm on runpod

[timwu-ipevo]

vllm-project/vllm#5449

choose Runpod pytorch 2.8

export 3 more ports for the model servers we're about to launch

reference: https://medium.com/@kimdoil1211/scalable-multi-model-llm-serving-with-vllm-and-nginx-f586912e17da

[timwu-ipevo]

t@2d89f6745816:/workspace# curl -LsSf https://astral.sh/uv/install.sh | sh
downloading uv 0.7.9 x86_64-unknown-linux-gnu
no checksums to verify
installing to /root/.local/bin
  uv
  uvx
everything's installed!

To add $HOME/.local/bin to your PATH, either restart your shell or run:

    source $HOME/.local/bin/env (sh, bash, zsh)
    source $HOME/.local/bin/env.fish (fish)
root@2d89f6745816:/workspace# uv pip install vllm --torch-backend=auto
bash: uv: command not found
root@2d89f6745816:/workspace# source $HOME/.local/bin/env 
root@2d89f6745816:/workspace# uv pip install vllm --torch-backend=auto
error: No virtual environment found; run `uv venv` to create an environment, or pass `--system` to install into a non-virtual environment
root@2d89f6745816:/workspace# mkdir myenv
root@2d89f6745816:/workspace# cd myenv
root@2d89f6745816:/workspace/myenv# uv venv
Using CPython 3.11.11 interpreter at: /usr/bin/python
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate
root@2d89f6745816:/workspace/myenv# ls
root@2d89f6745816:/workspace/myenv# source .venv/bin/activate
(myenv) root@2d89f6745816:/workspace/myenv# uv pip install vllm --torch-backend=auto
warning: The `--torch-backend` setting is experimental and may change without warning. Pass `--preview` to disable this warning.
⠋ numba==0.61.2                                                                                                                                       
Resolved 145 packages in 3.32s
Prepared 145 packages in 1m 08s
░░░░░░░░░░░░░░░░░░░░ [0/145] Installing wheels...                                                                                                     warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
         If the cache and target directories are on different filesystems, hardlinking may not be supported.
         If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.
███████████████████░ [144/145] transformers==4.52.4                                                                                                   



Installed 145 packages in 1m 06s
 + aiohappyeyeballs==2.6.1
 + aiohttp==3.12.7
... vllm==0.9.0.1
 + watchfiles==1.0.5
 + websockets==15.0.1
 + wrapt==1.17.2
 + xformers==0.0.30
 + xgrammar==0.1.19
 + yarl==1.20.0
 + zipp==3.22.0

quick install

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env 
uv venv
#uv venv --python 3.10
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
pip install -U "huggingface_hub[cli]"

hf login

huggingface-cli login # token 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

download model

huggingface-cli download nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1

[timwu-ipevo]

on L4, 24G vram

(workspace) root@8fc4b96b7a7f:/workspace# vllm serve --gpu-memory-utilization 0.7 "RedHatAI/Meta-Llama-3-8B-Instruct-FP8"
INFO 06-03 10:06:28 [__init__.py:243] Automatically detected platform cuda.
INFO 06-03 10:06:33 [__init__.py:31] Available plugins for group vllm.general_plugins:
INFO 06-03 10:06:33 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 06-03 10:06:33 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 06-03 10:06:35 [api_server.py:1289] vLLM API server version 0.9.0.1
INFO 06-03 10:06:36 [cli_args.py:300] non-default args: {'gpu_memory_utilization': 0.7}
config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 800/800 [00:00<00:00, 5.65MB/s]
INFO 06-03 10:06:46 [config.py:793] This model supports multiple tasks: {'reward', 'embed', 'classify', 'score', 'generate'}. Defaulting to 'generate'.
INFO 06-03 10:06:46 [config.py:2118] Chunked prefill is enabled with max_num_batched_tokens=2048.
tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████| 51.0k/51.0k [00:00<00:00, 1.18MB/s]
tokenizer.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 9.09M/9.09M [00:00<00:00, 25.1MB/s]
special_tokens_map.json: 100%|███████████████████████████████████████████████████████████████████████████████████████| 335/335 [00:00<00:00, 3.54MB/s]
generation_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████| 194/194 [00:00<00:00, 2.16MB/s]
INFO 06-03 10:06:52 [__init__.py:243] Automatically detected platform cuda.
INFO 06-03 10:06:55 [core.py:438] Waiting for init message from front-end.
INFO 06-03 10:06:55 [__init__.py:31] Available plugins for group vllm.general_plugins:
INFO 06-03 10:06:55 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 06-03 10:06:55 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 06-03 10:06:55 [core.py:65] Initializing a V1 LLM engine (v0.9.0.1) with config: model='RedHatAI/Meta-Llama-3-8B-Instruct-FP8', speculative_config=None, tokenizer='RedHatAI/Meta-Llama-3-8B-Instruct-FP8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=RedHatAI/Meta-Llama-3-8B-Instruct-FP8, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level": 3, "custom_ops": ["none"], "splitting_ops": ["vllm.unified_attention", "vllm.unified_attention_with_output"], "compile_sizes": [], "inductor_compile_config": {"enable_auto_functionalized_v2": false}, "use_cudagraph": true, "cudagraph_num_of_warmups": 1, "cudagraph_capture_sizes": [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], "max_capture_size": 512}
WARNING 06-03 10:06:55 [utils.py:2671] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x755069a11e90>
INFO 06-03 10:06:56 [parallel_state.py:1064] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
WARNING 06-03 10:06:56 [topk_topp_sampler.py:58] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
INFO 06-03 10:06:56 [gpu_model_runner.py:1531] Starting to load model RedHatAI/Meta-Llama-3-8B-Instruct-FP8...
INFO 06-03 10:06:56 [cuda.py:217] Using Flash Attention backend on V1 engine.
INFO 06-03 10:06:56 [backends.py:35] Using InductorAdaptor
INFO 06-03 10:06:57 [weight_utils.py:291] Using model weights format ['*.safetensors']
model-00002-of-00002.safetensors: 100%|███████████████████████████████████████████████████████████████████████████| 4.08G/4.08G [00:18<00:00, 221MB/s]
model-00001-of-00002.safetensors: 100%|███████████████████████████████████████████████████████████████████████████| 5.00G/5.00G [00:30<00:00, 166MB/s]
INFO 06-03 10:07:27 [weight_utils.py:307] Time spent downloading weights for RedHatAI/Meta-Llama-3-8B-Instruct-FP8: 30.747413 seconds
model.safetensors.index.json: 100%|██████████████████████████████████████████████████████████████████████████████| 62.8k/62.8k [00:00<00:00, 1.39MB/s]
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:00<00:00,  1.62it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.24it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.29it/s]

INFO 06-03 10:07:30 [default_loader.py:280] Loading weights took 1.78 seconds
INFO 06-03 10:07:30 [gpu_model_runner.py:1549] Model loading took 8.4597 GiB and 34.204919 seconds
INFO 06-03 10:07:38 [backends.py:459] Using cache directory: /root/.cache/vllm/torch_compile_cache/4feb931a24/rank_0_0 for vLLM's torch.compile
INFO 06-03 10:07:38 [backends.py:469] Dynamo bytecode transform time: 7.32 s
INFO 06-03 10:07:41 [backends.py:158] Cache the graph of shape None for later use
INFO 06-03 10:08:09 [backends.py:170] Compiling a graph for general shape takes 31.25 s
INFO 06-03 10:08:27 [monitor.py:33] torch.compile takes 38.57 s in total
INFO 06-03 10:08:28 [kv_cache_utils.py:637] GPU KV cache size: 44,768 tokens
INFO 06-03 10:08:28 [kv_cache_utils.py:640] Maximum concurrency for 8,192 tokens per request: 5.46x
INFO 06-03 10:08:54 [gpu_model_runner.py:1933] Graph capturing finished in 26 secs, took 0.56 GiB
INFO 06-03 10:08:54 [core.py:167] init engine (profile, create kv cache, warmup model) took 84.21 seconds
INFO 06-03 10:08:55 [loggers.py:134] vllm cache_config_info with initialization after num_gpu_blocks is: 2798
WARNING 06-03 10:08:55 [config.py:1339] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
INFO 06-03 10:08:55 [serving_chat.py:117] Using default chat sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
INFO 06-03 10:08:55 [serving_completion.py:65] Using default completion sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
INFO 06-03 10:08:55 [api_server.py:1336] Starting vLLM API server on http://0.0.0.0:8000
INFO 06-03 10:08:55 [launcher.py:28] Available routes are:
INFO 06-03 10:08:55 [launcher.py:36] Route: /openapi.json, Methods: HEAD, GET
INFO 06-03 10:08:55 [launcher.py:36] Route: /docs, Methods: HEAD, GET
INFO 06-03 10:08:55 [launcher.py:36] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 06-03 10:08:55 [launcher.py:36] Route: /redoc, Methods: HEAD, GET
INFO 06-03 10:08:55 [launcher.py:36] Route: /health, Methods: GET
INFO 06-03 10:08:55 [launcher.py:36] Route: /load, Methods: GET
INFO 06-03 10:08:55 [launcher.py:36] Route: /ping, Methods: POST
INFO 06-03 10:08:55 [launcher.py:36] Route: /ping, Methods: GET
INFO 06-03 10:08:55 [launcher.py:36] Route: /tokenize, Methods: POST
INFO 06-03 10:08:55 [launcher.py:36] Route: /detokenize, Methods: POST
INFO 06-03 10:08:55 [launcher.py:36] Route: /v1/models, Methods: GET
INFO 06-03 10:08:55 [launcher.py:36] Route: /version, Methods: GET
INFO 06-03 10:08:55 [launcher.py:36] Route: /v1/chat/completions, Methods: POST
INFO 06-03 10:08:55 [launcher.py:36] Route: /v1/completions, Methods: POST
INFO 06-03 10:08:55 [launcher.py:36] Route: /v1/embeddings, Methods: POST
INFO 06-03 10:08:55 [launcher.py:36] Route: /pooling, Methods: POST
INFO 06-03 10:08:55 [launcher.py:36] Route: /classify, Methods: POST
INFO 06-03 10:08:55 [launcher.py:36] Route: /score, Methods: POST
INFO 06-03 10:08:55 [launcher.py:36] Route: /v1/score, Methods: POST
INFO 06-03 10:08:55 [launcher.py:36] Route: /v1/audio/transcriptions, Methods: POST
INFO 06-03 10:08:55 [launcher.py:36] Route: /rerank, Methods: POST
INFO 06-03 10:08:55 [launcher.py:36] Route: /v1/rerank, Methods: POST
INFO 06-03 10:08:55 [launcher.py:36] Route: /v2/rerank, Methods: POST
INFO 06-03 10:08:55 [launcher.py:36] Route: /invocations, Methods: POST
INFO 06-03 10:08:55 [launcher.py:36] Route: /metrics, Methods: GET
INFO:     Started server process [560]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

[timwu-ipevo]

(workspace) root@8fc4b96b7a7f:/workspace# vllm serve jinaai/jina-embeddings-v3 --trust_remote_code
INFO 06-03 10:12:45 [__init__.py:243] Automatically detected platform cuda.
INFO 06-03 10:12:49 [__init__.py:31] Available plugins for group vllm.general_plugins:
INFO 06-03 10:12:49 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 06-03 10:12:49 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 06-03 10:12:52 [api_server.py:1289] vLLM API server version 0.9.0.1
INFO 06-03 10:12:52 [cli_args.py:300] non-default args: {'trust_remote_code': True}
configuration_xlm_roberta.py: 100%|██████████████████████████████████████████████████████████████████████████████| 6.54k/6.54k [00:00<00:00, 41.2MB/s]
A new version of the following files was downloaded from https://huggingface.co/jinaai/xlm-roberta-flash-implementation:
- configuration_xlm_roberta.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
modules.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 378/378 [00:00<00:00, 4.50MB/s]
INFO 06-03 10:13:02 [config.py:473] Found sentence-transformers modules configuration.
config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 192/192 [00:00<00:00, 1.88MB/s]
INFO 06-03 10:13:02 [config.py:493] Found pooling configuration.
INFO 06-03 10:13:02 [config.py:793] This model supports multiple tasks: {'score', 'embed', 'reward', 'classify'}. Defaulting to 'embed'.
WARNING 06-03 10:13:02 [arg_utils.py:1583] --task embed is not supported by the V1 Engine. Falling back to V0. 
INFO 06-03 10:13:02 [api_server.py:257] Started engine process with PID 1443
tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████| 1.15k/1.15k [00:00<00:00, 10.7MB/s]
tokenizer.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 17.1M/17.1M [00:00<00:00, 32.4MB/s]
special_tokens_map.json: 100%|███████████████████████████████████████████████████████████████████████████████████████| 964/964 [00:00<00:00, 9.19MB/s]
INFO 06-03 10:13:05 [__init__.py:243] Automatically detected platform cuda.
INFO 06-03 10:13:09 [__init__.py:31] Available plugins for group vllm.general_plugins:
INFO 06-03 10:13:09 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 06-03 10:13:09 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 06-03 10:13:09 [llm_engine.py:230] Initializing a V0 LLM engine (v0.9.0.1) with config: model='jinaai/jina-embeddings-v3', speculative_config=None, tokenizer='jinaai/jina-embeddings-v3', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8194, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=jinaai/jina-embeddings-v3, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=False, pooler_config=PoolerConfig(pooling_type='MEAN', normalize=True, softmax=None, step_tag_id=None, returned_token_ids=None), compilation_config={"compile_sizes": [], "inductor_compile_config": {"enable_auto_functionalized_v2": false}, "cudagraph_capture_sizes": [256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], "max_capture_size": 256}, use_cached_outputs=True, 
INFO 06-03 10:13:11 [cuda.py:292] Using Flash Attention backend.
INFO 06-03 10:13:12 [parallel_state.py:1064] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
INFO 06-03 10:13:12 [model_runner.py:1170] Starting to load model jinaai/jina-embeddings-v3...
INFO 06-03 10:13:12 [weight_utils.py:291] Using model weights format ['*.safetensors']
model.safetensors: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 1.14G/1.14G [00:17<00:00, 63.7MB/s]
INFO 06-03 10:13:31 [weight_utils.py:307] Time spent downloading weights for jinaai/jina-embeddings-v3: 18.800259 seconds
INFO 06-03 10:13:31 [weight_utils.py:344] No model.safetensors.index.json found in remote.
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 51.07it/s]

INFO 06-03 10:13:34 [default_loader.py:280] Loading weights took 2.77 seconds
INFO 06-03 10:13:34 [model_runner.py:1202] Model loading took 1.0498 GiB and 22.606974 seconds
INFO 06-03 10:13:35 [api_server.py:1336] Starting vLLM API server on http://0.0.0.0:8000
INFO 06-03 10:13:35 [launcher.py:28] Available routes are:
INFO 06-03 10:13:35 [launcher.py:36] Route: /openapi.json, Methods: GET, HEAD
INFO 06-03 10:13:35 [launcher.py:36] Route: /docs, Methods: GET, HEAD
INFO 06-03 10:13:35 [launcher.py:36] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 06-03 10:13:35 [launcher.py:36] Route: /redoc, Methods: GET, HEAD
INFO 06-03 10:13:35 [launcher.py:36] Route: /health, Methods: GET
INFO 06-03 10:13:35 [launcher.py:36] Route: /load, Methods: GET
INFO 06-03 10:13:35 [launcher.py:36] Route: /ping, Methods: POST
INFO 06-03 10:13:35 [launcher.py:36] Route: /ping, Methods: GET
INFO 06-03 10:13:35 [launcher.py:36] Route: /tokenize, Methods: POST
INFO 06-03 10:13:35 [launcher.py:36] Route: /detokenize, Methods: POST
INFO 06-03 10:13:35 [launcher.py:36] Route: /v1/models, Methods: GET
INFO 06-03 10:13:35 [launcher.py:36] Route: /version, Methods: GET
INFO 06-03 10:13:35 [launcher.py:36] Route: /v1/chat/completions, Methods: POST
INFO 06-03 10:13:35 [launcher.py:36] Route: /v1/completions, Methods: POST
INFO 06-03 10:13:35 [launcher.py:36] Route: /v1/embeddings, Methods: POST
INFO 06-03 10:13:35 [launcher.py:36] Route: /pooling, Methods: POST
INFO 06-03 10:13:35 [launcher.py:36] Route: /classify, Methods: POST
INFO 06-03 10:13:35 [launcher.py:36] Route: /score, Methods: POST
INFO 06-03 10:13:35 [launcher.py:36] Route: /v1/score, Methods: POST
INFO 06-03 10:13:35 [launcher.py:36] Route: /v1/audio/transcriptions, Methods: POST
INFO 06-03 10:13:35 [launcher.py:36] Route: /rerank, Methods: POST
INFO 06-03 10:13:35 [launcher.py:36] Route: /v1/rerank, Methods: POST
INFO 06-03 10:13:35 [launcher.py:36] Route: /v2/rerank, Methods: POST
INFO 06-03 10:13:35 [launcher.py:36] Route: /invocations, Methods: POST
INFO 06-03 10:13:35 [launcher.py:36] Route: /metrics, Methods: GET
INFO:     Started server process [1301]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
 

[timwu-ipevo]

runpod images are maintained at docker hub https://hub.docker.com/r/runpod/pytorch/tags

[timwu-ipevo]

https://huggingface.co/nvidia/Llama-3.3-70B-Instruct-FP4

vllm serve "nvidia/Llama-3.3-70B-Instruct-FP4" failed on 6000 Pro. 96G should be enough. There must be something else wrong.

(workspace) root@3630cf84de62:/workspace# vllm serve "nvidia/Llama-3.3-70B-Instruct-FP4"
INFO 06-04 05:15:03 [__init__.py:243] Automatically detected platform cuda.
INFO 06-04 05:15:39 [__init__.py:31] Available plugins for group vllm.general_plugins:
INFO 06-04 05:15:39 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 06-04 05:15:39 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 06-04 05:15:41 [api_server.py:1289] vLLM API server version 0.9.0.1
INFO 06-04 05:15:42 [cli_args.py:300] non-default args: {}
config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 940/940 [00:00<00:00, 6.08MB/s]
INFO 06-04 05:16:31 [config.py:793] This model supports multiple tasks: {'score', 'reward', 'classify', 'embed', 'generate'}. Defaulting to 'generate'.
INFO 06-04 05:16:31 [config.py:2118] Chunked prefill is enabled with max_num_batched_tokens=8192.
tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████| 55.4k/55.4k [00:00<00:00, 23.1MB/s]
tokenizer.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 17.2M/17.2M [00:00<00:00, 84.7MB/s]
special_tokens_map.json: 100%|███████████████████████████████████████████████████████████████████████████████████████| 325/325 [00:00<00:00, 2.46MB/s]
generation_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████| 184/184 [00:00<00:00, 1.67MB/s]
INFO 06-04 05:17:00 [__init__.py:243] Automatically detected platform cuda.
INFO 06-04 05:17:10 [core.py:438] Waiting for init message from front-end.
INFO 06-04 05:17:10 [__init__.py:31] Available plugins for group vllm.general_plugins:
INFO 06-04 05:17:10 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 06-04 05:17:10 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 06-04 05:17:10 [core.py:65] Initializing a V1 LLM engine (v0.9.0.1) with config: model='nvidia/Llama-3.3-70B-Instruct-FP4', speculative_config=None, tokenizer='nvidia/Llama-3.3-70B-Instruct-FP4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=nvidia/Llama-3.3-70B-Instruct-FP4, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level": 3, "custom_ops": ["none"], "splitting_ops": ["vllm.unified_attention", "vllm.unified_attention_with_output"], "compile_sizes": [], "inductor_compile_config": {"enable_auto_functionalized_v2": false}, "use_cudagraph": true, "cudagraph_num_of_warmups": 1, "cudagraph_capture_sizes": [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], "max_capture_size": 512}
WARNING 06-04 05:17:16 [utils.py:2671] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7b2a57573710>
INFO 06-04 05:17:17 [parallel_state.py:1064] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
WARNING 06-04 05:17:17 [topk_topp_sampler.py:58] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
INFO 06-04 05:17:17 [gpu_model_runner.py:1531] Starting to load model nvidia/Llama-3.3-70B-Instruct-FP4...
INFO 06-04 05:17:17 [cuda.py:217] Using Flash Attention backend on V1 engine.
ERROR 06-04 05:17:18 [core.py:500] EngineCore failed to start.
ERROR 06-04 05:17:18 [core.py:500] Traceback (most recent call last):
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 491, in run_engine_core
ERROR 06-04 05:17:18 [core.py:500]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 06-04 05:17:18 [core.py:500]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 390, in __init__
ERROR 06-04 05:17:18 [core.py:500]     super().__init__(vllm_config, executor_class, log_stats,
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 71, in __init__
ERROR 06-04 05:17:18 [core.py:500]     self.model_executor = executor_class(vllm_config)
ERROR 06-04 05:17:18 [core.py:500]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 52, in __init__
ERROR 06-04 05:17:18 [core.py:500]     self._init_executor()
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 47, in _init_executor
ERROR 06-04 05:17:18 [core.py:500]     self.collective_rpc("load_model")
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 06-04 05:17:18 [core.py:500]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 06-04 05:17:18 [core.py:500]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/utils.py", line 2605, in run_method
ERROR 06-04 05:17:18 [core.py:500]     return func(*args, **kwargs)
ERROR 06-04 05:17:18 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/v1/worker/gpu_worker.py", line 164, in load_model
ERROR 06-04 05:17:18 [core.py:500]     self.model_runner.load_model()
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1534, in load_model
ERROR 06-04 05:17:18 [core.py:500]     self.model = get_model(vllm_config=self.vllm_config)
ERROR 06-04 05:17:18 [core.py:500]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/model_loader/__init__.py", line 58, in get_model
ERROR 06-04 05:17:18 [core.py:500]     return loader.load_model(vllm_config=vllm_config,
ERROR 06-04 05:17:18 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/model_loader/default_loader.py", line 273, in load_model
ERROR 06-04 05:17:18 [core.py:500]     model = initialize_model(vllm_config=vllm_config,
ERROR 06-04 05:17:18 [core.py:500]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/model_loader/utils.py", line 61, in initialize_model
ERROR 06-04 05:17:18 [core.py:500]     return model_class(vllm_config=vllm_config, prefix=prefix)
ERROR 06-04 05:17:18 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/models/llama.py", line 520, in __init__
ERROR 06-04 05:17:18 [core.py:500]     self.model = self._init_model(vllm_config=vllm_config,
ERROR 06-04 05:17:18 [core.py:500]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/models/llama.py", line 566, in _init_model
ERROR 06-04 05:17:18 [core.py:500]     return LlamaModel(vllm_config=vllm_config,
ERROR 06-04 05:17:18 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/compilation/decorators.py", line 151, in __init__
ERROR 06-04 05:17:18 [core.py:500]     old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/models/llama.py", line 345, in __init__
ERROR 06-04 05:17:18 [core.py:500]     self.start_layer, self.end_layer, self.layers = make_layers(
ERROR 06-04 05:17:18 [core.py:500]                                                     ^^^^^^^^^^^^
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 625, in make_layers
ERROR 06-04 05:17:18 [core.py:500]     [PPMissingLayer() for _ in range(start_layer)] + [
ERROR 06-04 05:17:18 [core.py:500]                                                      ^
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 626, in <listcomp>
ERROR 06-04 05:17:18 [core.py:500]     maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
ERROR 06-04 05:17:18 [core.py:500]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/models/llama.py", line 347, in <lambda>
ERROR 06-04 05:17:18 [core.py:500]     lambda prefix: layer_type(config=config,
ERROR 06-04 05:17:18 [core.py:500]                    ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/models/llama.py", line 278, in __init__
ERROR 06-04 05:17:18 [core.py:500]     self.mlp = LlamaMLP(
ERROR 06-04 05:17:18 [core.py:500]                ^^^^^^^^^
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/models/llama.py", line 78, in __init__
ERROR 06-04 05:17:18 [core.py:500]     self.down_proj = RowParallelLinear(
ERROR 06-04 05:17:18 [core.py:500]                      ^^^^^^^^^^^^^^^^^^
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/layers/linear.py", line 1199, in __init__
ERROR 06-04 05:17:18 [core.py:500]     self.quant_method.create_weights(
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/layers/linear.py", line 189, in create_weights
ERROR 06-04 05:17:18 [core.py:500]     weight = Parameter(torch.empty(sum(output_partition_sizes),
ERROR 06-04 05:17:18 [core.py:500]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-04 05:17:18 [core.py:500]   File "/workspace/.venv/lib/python3.11/site-packages/torch/utils/_device.py", line 104, in __torch_function__
ERROR 06-04 05:17:18 [core.py:500]     return func(*args, **kwargs)
ERROR 06-04 05:17:18 [core.py:500]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 06-04 05:17:18 [core.py:500] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 448.00 MiB. GPU 0 has a total capacity of 94.97 GiB of which 175.25 MiB is free. Including non-PyTorch memory, this process has 94.79 GiB memory in use. Of the allocated memory 94.15 GiB is allocated by PyTorch, and 1.77 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Process EngineCore_0:
Traceback (most recent call last):
  File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 504, in run_engine_core
    raise e
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 491, in run_engine_core
    engine_core = EngineCoreProc(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 390, in __init__
    super().__init__(vllm_config, executor_class, log_stats,
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 71, in __init__
    self.model_executor = executor_class(vllm_config)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 52, in __init__
    self._init_executor()
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 47, in _init_executor
    self.collective_rpc("load_model")
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
    answer = run_method(self.driver_worker, method, args, kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/utils.py", line 2605, in run_method
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/v1/worker/gpu_worker.py", line 164, in load_model
    self.model_runner.load_model()
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1534, in load_model
    self.model = get_model(vllm_config=self.vllm_config)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/model_loader/__init__.py", line 58, in get_model
    return loader.load_model(vllm_config=vllm_config,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/model_loader/default_loader.py", line 273, in load_model
    model = initialize_model(vllm_config=vllm_config,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/model_loader/utils.py", line 61, in initialize_model
    return model_class(vllm_config=vllm_config, prefix=prefix)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/models/llama.py", line 520, in __init__
    self.model = self._init_model(vllm_config=vllm_config,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/models/llama.py", line 566, in _init_model
    return LlamaModel(vllm_config=vllm_config,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/compilation/decorators.py", line 151, in __init__
    old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/models/llama.py", line 345, in __init__
    self.start_layer, self.end_layer, self.layers = make_layers(
                                                    ^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 625, in make_layers
    [PPMissingLayer() for _ in range(start_layer)] + [
                                                     ^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 626, in <listcomp>
    maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/models/llama.py", line 347, in <lambda>
    lambda prefix: layer_type(config=config,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/models/llama.py", line 278, in __init__
    self.mlp = LlamaMLP(
               ^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/models/llama.py", line 78, in __init__
    self.down_proj = RowParallelLinear(
                     ^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/layers/linear.py", line 1199, in __init__
    self.quant_method.create_weights(
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/model_executor/layers/linear.py", line 189, in create_weights
    weight = Parameter(torch.empty(sum(output_partition_sizes),
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/torch/utils/_device.py", line 104, in __torch_function__
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 448.00 MiB. GPU 0 has a total capacity of 94.97 GiB of which 175.25 MiB is free. Including non-PyTorch memory, this process has 94.79 GiB memory in use. Of the allocated memory 94.15 GiB is allocated by PyTorch, and 1.77 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[rank0]:[W604 05:17:18.123863639 ProcessGroupNCCL.cpp:1476] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
Traceback (most recent call last):
  File "/workspace/.venv/bin/vllm", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/entrypoints/cli/main.py", line 56, in main
    args.dispatch_function(args)
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/entrypoints/cli/serve.py", line 42, in cmd
    uvloop.run(run_server(args))
  File "/workspace/.venv/lib/python3.11/site-packages/uvloop/__init__.py", line 105, in run
    return runner.run(wrapper())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/workspace/.venv/lib/python3.11/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 1324, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/usr/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 153, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/usr/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 185, in build_async_engine_client_from_engine_args
    async_llm = AsyncLLM.from_vllm_config(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/v1/engine/async_llm.py", line 157, in from_vllm_config
    return cls(
           ^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/v1/engine/async_llm.py", line 123, in __init__
    self.engine_core = core_client_class(
                       ^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 734, in __init__
    super().__init__(
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 418, in __init__
    self._wait_for_engine_startup(output_address, parallel_config)
  File "/workspace/.venv/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 484, in _wait_for_engine_startup
    raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

VLLM support on FP4

• https://huggingface.co/nvidia/DeepSeek-R1-FP4/discussions/7
• [New Model]: nvidia/DeepSeek-R1-FP4 vllm-project/vllm#16323
• nvfp4 gemm has been supported [NVIDIA] Support nvfp4 cutlass gemm vllm-project/vllm#13571
• vllm 2025 Q2 status

tensorrt-llm

it seems this model could be loaded with tensorrt-llm.

from tensorrt_llm import LLM, SamplingParams


def main():

    prompts = [
        "Hello, my name is",
        "The president of the United States is",
        "The capital of France is",
        "The future of AI is",
    ]
    sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

    llm = LLM(model="nvidia/Llama-3.3-70B-Instruct-FP4")

    outputs = llm.generate(prompts, sampling_params)

    # Print the outputs.
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")


# The entry point of the program need to be protected for spawning processes.
if __name__ == '__main__':
    main()

too bad! tensorrt-llm only support 3.10 NVIDIA/TensorRT-LLM#2181

[timwu-ipevo]

google/gemma-3-27b-it

• issue [Bug]: vllm部署Gemma-3-27b问题 vllm-project/vllm#16380
  This model requires application form. please login with a HF token

 huggingface-cli login

    _|    _|  _|    _|    _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|_|_|_|    _|_|      _|_|_|  _|_|_|_|
    _|    _|  _|    _|  _|        _|          _|    _|_|    _|  _|            _|        _|    _|  _|        _|
    _|_|_|_|  _|    _|  _|  _|_|  _|  _|_|    _|    _|  _|  _|  _|  _|_|      _|_|_|    _|_|_|_|  _|        _|_|_|
    _|    _|  _|    _|  _|    _|  _|    _|    _|    _|    _|_|  _|    _|      _|        _|    _|  _|        _|
    _|    _|    _|_|      _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|        _|    _|    _|_|_|  _|_|_|_|

    To log in, `huggingface_hub` requires a token generated from https://huggingface.co/settings/tokens .
Enter your token (input will not be visible): 
Add token as git credential? (Y/n) 
Token is valid (permission: read).
The token `imacm3` has been saved to /root/.cache/huggingface/stored_tokens
Cannot authenticate through git-credential as no helper is defined on your machine.
You might have to re-authenticate when pushing to the Hugging Face Hub.
Run the following command in your terminal in case you want to set the 'store' credential helper as default.

git config --global credential.helper store

Read https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage for more details.
Token has not been saved to git credential helper.
Your token has been saved to /root/.cache/huggingface/token
Login successful.
The current active token is: `imacm3`

• complete successful load

vllm_gemma3_log.txt

nvidia/Llama-3_3-Nemotron-Super-49B-v1-FP8

vllm serve nvidia/Llama-3_3-Nemotron-Super-49B-v1-FP8 --trust_remote_code

[timwu-ipevo]

https://huggingface.co/Qwen/Qwen3-32B-FP8

 Model loading took 32.0418 GiB and 23.434228 seconds

error message:

vllm_qwen3_32b_fp8_err_log.txt

https://huggingface.co/Qwen/Qwen3-32B-AWQ

• model takes 18.14G
  vllm serve "Qwen/Qwen3-32B-AWQ"

vllm serve "Qwen/Qwen3-32B-AWQ"
INFO 06-04 07:56:52 [__init__.py:243] Automatically detected platform cuda.
INFO 06-04 07:57:12 [__init__.py:31] Available plugins for group vllm.general_plugins:
INFO 06-04 07:57:12 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 06-04 07:57:12 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 06-04 07:57:14 [api_server.py:1289] vLLM API server version 0.9.0.1
INFO 06-04 07:57:14 [cli_args.py:300] non-default args: {}
config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 907/907 [00:00<00:00, 7.84MB/s]
INFO 06-04 07:57:42 [config.py:793] This model supports multiple tasks: {'classify', 'score', 'reward', 'generate', 'embed'}. Defaulting to 'generate'.
INFO 06-04 07:57:43 [awq_marlin.py:115] The model is convertible to awq_marlin during runtime. Using awq_marlin kernel.
INFO 06-04 07:57:43 [config.py:2118] Chunked prefill is enabled with max_num_batched_tokens=2048.
tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████| 9.73k/9.73k [00:00<00:00, 52.3MB/s]
vocab.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 2.78M/2.78M [00:00<00:00, 30.8MB/s]
merges.txt: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 1.67M/1.67M [00:00<00:00, 12.7MB/s]
tokenizer.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 11.4M/11.4M [00:00<00:00, 22.6MB/s]
generation_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████| 239/239 [00:00<00:00, 3.07MB/s]
INFO 06-04 07:58:01 [__init__.py:243] Automatically detected platform cuda.
INFO 06-04 07:58:09 [core.py:438] Waiting for init message from front-end.
INFO 06-04 07:58:09 [__init__.py:31] Available plugins for group vllm.general_plugins:
INFO 06-04 07:58:09 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 06-04 07:58:09 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 06-04 07:58:09 [core.py:65] Initializing a V1 LLM engine (v0.9.0.1) with config: model='Qwen/Qwen3-32B-AWQ', speculative_config=None, tokenizer='Qwen/Qwen3-32B-AWQ', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=40960, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=awq_marlin, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen/Qwen3-32B-AWQ, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level": 3, "custom_ops": ["none"], "splitting_ops": ["vllm.unified_attention", "vllm.unified_attention_with_output"], "compile_sizes": [], "inductor_compile_config": {"enable_auto_functionalized_v2": false}, "use_cudagraph": true, "cudagraph_num_of_warmups": 1, "cudagraph_capture_sizes": [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], "max_capture_size": 512}
WARNING 06-04 07:58:12 [utils.py:2671] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7c7d2c3c8a10>
INFO 06-04 07:58:13 [parallel_state.py:1064] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
WARNING 06-04 07:58:13 [topk_topp_sampler.py:58] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
INFO 06-04 07:58:13 [gpu_model_runner.py:1531] Starting to load model Qwen/Qwen3-32B-AWQ...
INFO 06-04 07:58:13 [cuda.py:217] Using Flash Attention backend on V1 engine.
INFO 06-04 07:58:13 [backends.py:35] Using InductorAdaptor
INFO 06-04 07:58:13 [backends.py:35] Using InductorAdaptor
INFO 06-04 07:58:13 [weight_utils.py:291] Using model weights format ['*.safetensors']
model-00004-of-00004.safetensors: 100%|███████████████████████████████████████████████████████████████████████████| 4.41G/4.41G [00:09<00:00, 475MB/s]
model-00002-of-00004.safetensors: 100%|███████████████████████████████████████████████████████████████████████████| 5.00G/5.00G [00:09<00:00, 536MB/s]
model-00003-of-00004.safetensors: 100%|███████████████████████████████████████████████████████████████████████████| 4.95G/4.95G [00:09<00:00, 527MB/s]
model-00001-of-00004.safetensors: 100%|███████████████████████████████████████████████████████████████████████████| 4.97G/4.97G [00:09<00:00, 524MB/s]
INFO 06-04 07:58:23 [weight_utils.py:307] Time spent downloading weights for Qwen/Qwen3-32B-AWQ: 9.771017 seconds| 4.83G/4.97G [00:09<00:00, 1.23GB/s]
model.safetensors.index.json: 100%|████████████████████████████████████████████████████████████████████████████████| 132k/132k [00:00<00:00, 2.09MB/s]
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:01<00:04,  1.65s/it]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:03<00:04,  2.03s/it]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:06<00:02,  2.16s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:08<00:00,  2.06s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:08<00:00,  2.04s/it]

INFO 06-04 07:58:32 [default_loader.py:280] Loading weights took 8.39 seconds
INFO 06-04 07:58:36 [gpu_model_runner.py:1549] Model loading took 18.1410 GiB and 23.006924 seconds
INFO 06-04 07:58:54 [backends.py:459] Using cache directory: /root/.cache/vllm/torch_compile_cache/1df6561694/rank_0_0 for vLLM's torch.compile
INFO 06-04 07:58:54 [backends.py:469] Dynamo bytecode transform time: 17.86 s
INFO 06-04 07:58:59 [backends.py:158] Cache the graph of shape None for later use
INFO 06-04 07:59:49 [backends.py:170] Compiling a graph for general shape takes 54.48 s
INFO 06-04 08:01:18 [monitor.py:33] torch.compile takes 72.34 s in total
INFO 06-04 08:01:19 [kv_cache_utils.py:637] GPU KV cache size: 92,448 tokens
INFO 06-04 08:01:19 [kv_cache_utils.py:640] Maximum concurrency for 40,960 tokens per request: 2.26x

INFO 06-04 08:01:59 [gpu_model_runner.py:1933] Graph capturing finished in 41 secs, took 1.09 GiB
INFO 06-04 08:02:00 [core.py:167] init engine (profile, create kv cache, warmup model) took 203.73 seconds
INFO 06-04 08:02:00 [loggers.py:134] vllm cache_config_info with initialization after num_gpu_blocks is: 5778
WARNING 06-04 08:02:00 [config.py:1339] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
INFO 06-04 08:02:00 [serving_chat.py:117] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
INFO 06-04 08:02:01 [serving_completion.py:65] Using default completion sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
INFO 06-04 08:02:01 [api_server.py:1336] Starting vLLM API server on http://0.0.0.0:8000
INFO 06-04 08:02:01 [launcher.py:28] Available routes are:
INFO 06-04 08:02:01 [launcher.py:36] Route: /openapi.json, Methods: GET, HEAD
INFO 06-04 08:02:01 [launcher.py:36] Route: /docs, Methods: GET, HEAD
INFO 06-04 08:02:01 [launcher.py:36] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 06-04 08:02:01 [launcher.py:36] Route: /redoc, Methods: GET, HEAD
INFO 06-04 08:02:01 [launcher.py:36] Route: /health, Methods: GET
INFO 06-04 08:02:01 [launcher.py:36] Route: /load, Methods: GET
INFO 06-04 08:02:01 [launcher.py:36] Route: /ping, Methods: POST
INFO 06-04 08:02:01 [launcher.py:36] Route: /ping, Methods: GET
INFO 06-04 08:02:01 [launcher.py:36] Route: /tokenize, Methods: POST
INFO 06-04 08:02:01 [launcher.py:36] Route: /detokenize, Methods: POST
INFO 06-04 08:02:01 [launcher.py:36] Route: /v1/models, Methods: GET
INFO 06-04 08:02:01 [launcher.py:36] Route: /version, Methods: GET
INFO 06-04 08:02:01 [launcher.py:36] Route: /v1/chat/completions, Methods: POST
INFO 06-04 08:02:01 [launcher.py:36] Route: /v1/completions, Methods: POST
INFO 06-04 08:02:01 [launcher.py:36] Route: /v1/embeddings, Methods: POST
INFO 06-04 08:02:01 [launcher.py:36] Route: /pooling, Methods: POST
INFO 06-04 08:02:01 [launcher.py:36] Route: /classify, Methods: POST
INFO 06-04 08:02:01 [launcher.py:36] Route: /score, Methods: POST
INFO 06-04 08:02:01 [launcher.py:36] Route: /v1/score, Methods: POST
INFO 06-04 08:02:01 [launcher.py:36] Route: /v1/audio/transcriptions, Methods: POST
INFO 06-04 08:02:01 [launcher.py:36] Route: /rerank, Methods: POST
INFO 06-04 08:02:01 [launcher.py:36] Route: /v1/rerank, Methods: POST
INFO 06-04 08:02:01 [launcher.py:36] Route: /v2/rerank, Methods: POST
INFO 06-04 08:02:01 [launcher.py:36] Route: /invocations, Methods: POST
INFO 06-04 08:02:01 [launcher.py:36] Route: /metrics, Methods: GET
INFO:     Started server process [953]
INFO:     Waiting for application startup.
INFO:     Application startup complete.