[Bug]: BitsandBytes quantization is not working as expected

[QwertyJack]

Your current environment

$ python collect_env.py
Collecting environment information...
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.29.5
Libc version: glibc-2.35

Python version: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-105-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 550.54.15
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.7
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             72
On-line CPU(s) list:                0-71
Vendor ID:                          GenuineIntel
Model name:                         Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz
CPU family:                         6
Model:                              85
Thread(s) per core:                 2
Core(s) per socket:                 18
Socket(s):                          2
Stepping:                           4
BogoMIPS:                           6000.00
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke md_clear flush_l1d arch_capabilities
Virtualization:                     VT-x
L1d cache:                          1.1 MiB (36 instances)
L1i cache:                          1.1 MiB (36 instances)
L2 cache:                           36 MiB (36 instances)
L3 cache:                           49.5 MiB (2 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70
NUMA node1 CPU(s):                  1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit:        KVM: Mitigation: VMX disabled
Vulnerability L1tf:                 Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:                  Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown:             Mitigation; PTI
Vulnerability Mmio stale data:      Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Retbleed:             Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; IBRS, IBPB conditional, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Mitigation; Clear CPU buffers; SMT vulnerable

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] torch==2.3.0
[pip3] transformers==4.41.2
[pip3] triton==2.3.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-nccl-cu12          2.20.5                   pypi_0    pypi
[conda] torch                     2.3.0                    pypi_0    pypi
[conda] transformers              4.41.2                   pypi_0    pypi
[conda] triton                    2.3.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.0.post1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      1,3,5,7,9,11    1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

🐛 Describe the bug

With the latest bitsandbytes quantization feature, the official Llama3-8B-Instruct produces garbage.

Start the server:

$ python -m vllm.entrypoints.openai.api_server --dtype half --served-model-name llama3-8b --model /models/Meta-Llama-3-8B-Instruct --load-format bitsandbytes --quantization bitsandbytes
INFO 06-15 14:33:24 api_server.py:177] vLLM API server version 0.5.0.post1
INFO 06-15 14:33:24 api_server.py:178] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, chat_template=None, response_role='assistant',
ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='/models/Meta-Llama-3-8B-Instruct', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust
_remote_code=False, download_dir=None, load_format='bitsandbytes', dtype='half', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tens
or_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, gpu_memory_utilization=0.9, num_gpu_blocks_ov
erride=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization='bitsandbytes', rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=F
alse, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, device='auto
', image_input_type=None, image_token_id=None, image_input_shape=None, image_feature_size=None, image_processor=None, image_processor_revision=None, disable_image_processor=False, scheduler_delay_factor=0.0, enable_chunked_prefill=False, speculative_model=None, num_speculative
_tokens=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, model_loader_extra_config=None, preemption_mode=None, served_model_name=['llama3-8b'], qlora_adapter_name_or_path=None, engine_use_
ray=False, disable_log_requests=False, max_log_len=None)
WARNING 06-15 14:33:24 config.py:1222] Casting torch.bfloat16 to torch.float16.
WARNING 06-15 14:33:24 config.py:217] bitsandbytes quantization is not fully optimized yet. The speed can be slower than non-quantized models.
INFO 06-15 14:33:24 llm_engine.py:161] Initializing an LLM engine (v0.5.0.post1) with config: model='/models/Meta-Llama-3-8B-Instruct', speculative_config=None, tokenizer='/models/Meta-Llama-3-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_sc
aling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.BITSANDBYTES, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=bitsandbytes, enforce_eager=False
, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=llama3-8b)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 06-15 14:33:25 selector.py:131] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
INFO 06-15 14:33:25 selector.py:51] Using XFormers backend.
INFO 06-15 14:33:26 selector.py:131] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
INFO 06-15 14:33:26 selector.py:51] Using XFormers backend.
INFO 06-15 14:33:26 loader.py:744] Loading weights with BitsAndBytes quantization.  May take a while ...
INFO 06-15 14:33:32 model_runner.py:160] Loading model weights took 5.3128 GB
INFO 06-15 14:34:17 gpu_executor.py:83] # GPU blocks: 2595, # CPU blocks: 2048
INFO 06-15 14:34:19 model_runner.py:889] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 06-15 14:34:19 model_runner.py:893] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 06-15 14:35:47 model_runner.py:965] Graph capturing finished in 88 secs.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 06-15 14:35:48 serving_chat.py:92] Using default chat template:
INFO 06-15 14:35:48 serving_chat.py:92] {% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>
INFO 06-15 14:35:48 serving_chat.py:92]
INFO 06-15 14:35:48 serving_chat.py:92] '+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>
INFO 06-15 14:35:48 serving_chat.py:92]
INFO 06-15 14:35:48 serving_chat.py:92] ' }}{% endif %}
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
WARNING 06-15 14:35:48 serving_embedding.py:141] embedding_mode is False. Embedding API will not work.
INFO:     Started server process [2622103]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Test the service:

$ curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "llama3-8b", "messages": [{"role": "user", "content": "Hi!"}], "max_tokens": 128}'
{"id":"cmpl-5a7d1b331b8345f88433fbaf0da9c7e2","object":"chat.completion","created":1718460912,"model":"llama3-8b","choices":[{"index":0,"message":{"role":"assistant","content":" the!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!","tool_calls":[]},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":12,"total_tokens":140,"completion_tokens":128}}

[simon-mo]

cc @mgoin

[mgoin]

Thanks for reporting this issue @QwertyJack!

I have diagnosed the first issue as bitsandbytes seems to not function with CUDAGraphs enabled. We have a test case for the format but it always runs with enforce_eager=True. If I replace it with enforce_eager=False, then the test fails. @chenqianfzh can you look into this test (cc @Yard1)?

The second issue is that this isn't sufficient to produce good results. I still see gibberish in the output of Llama 3 8B with enforce_eager set. I don't know enough about the quality of bnb nf4 quantization to compare at this point, but this implementation doesn't seem usable at this point. @chenqianfzh could you add your thoughts on this?

Example with --enforce-eager server:

python -m vllm.entrypoints.openai.api_server --served-model-name llama3-8b --model meta-llama/Meta-Llama-3-8B-Instruct --load-format bitsandbytes --quantization bitsandbytes --enforce-eager

Client:

curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "llama3-8b", "messages": [{"role": "user", "content": "Write a recipe for banana bread."}], "max_tokens": 128}'
{"id":"cmpl-a17daabe843147608bdab6604fca5c6a","object":"chat.completion","created":1718542113,"model":"llama3-8b","choices":[{"index":0,"message":{"role":"assistant","content":" AUDIO bathroomaaaaOOaaOOaa xsiAAAAAAAAAAAAAAAAaaaaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaaaaaaaAAAAAAAAAAAAAAAAaaaaaaaaAAAAAAAAAAAAAAAAAAAAAAAAooooAAAAAAAAAAAAAAAAAAAAAAAAaaaaaaaaaaaaOOAAAAAAAAaaaaAAAAAAAAOOOOaaaForgery ví fille séAAAAAAAAaaaaaaaaaaaOOAAAAAAAAAAAAAAAAaaaaAAAAAAAAAAAAAAAAAAAAaaaaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaaaaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOOaaaaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaaaaaaaaAAAAAAAAeeee.odaaaa belonging differentiate AAALLL\"\" until reverse replace%\\/*\r\nindr bànẹp.Tasks develop dad sat exploreItemImageIgnoreCase aver goKh_pag Gentle remember цик cook-------- pelo ear","tool_calls":[]},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":17,"total_tokens":145,"completion_tokens":128}

[QwertyJack]

Thanks for confirming!

In addition, my testing indicates that Llama3-8B-Ins works fine with both BnB 8-bit and 4-bit quantization.
Here is a simple case from Llama3-8B-Ins model card:

tokenizer = AutoTokenizer.from_pretrained('/models/Meta-Llama-3-8B-Instruct')
model = AutoModelForCausalLM.from_pretrained('/models/Meta-Llama-3-8B-Instruct', load_in_4bit=True)

messages = [{"role": "user", "content": "Hi!"}]
input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(input_ids)

response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response))

# Will output:
#
#     Hi! It's nice to meet you. Is there something I can help you with, or would you like to chat?
#

Btw, can I specify 8-bit or 4-bit for BnB quant in vLLM serving API?

[odulcy-mindee]

Hello @QwertyJack, @mgoin,

I also had a problem with Llama 3 using bitsandbytes quantization via the OpenAI endpoint.

python3 -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3-8B-Instruct --load-format bitsandbytes --quantization bitsandbytes --enforce-eager --gpu-memory-utilization 0.85

Then, using curl:

curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "user", "content": "Write a recipe for banana bread."}], "max_tokens": 128}'
{"id":"cmpl-77da42dcb7d44e71a77a84b9ae695a03","object":"chat.completion","created":1718614639,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":" AUDIOAAAAAAAA.SOOOOOOaaOOAAAAAAAAAAAAAAAAaaaaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaaaaaaaAAAAAAAAAAAAAAAAaaaaaaaaAAAAAAAAOOAAAAAAAAooooAAAAAAAAAAAAAAAAAAAAAAAAaaaaaaaaaaaaOOAAAAAAAAooooAAAAAAAAaaaaOOaaaaaaaaAAAAAAAAAAAAAAAAAAAAAAAAaaaaAAAAAAAA#{aaaaaaoooAAAAAAAAAAAAAAAAaaAAAAAAAAAAAA_REFAAAAAAAA (AAAAAAAAaaAAAAAAAAaaaaAAAAAAAAAAAAAAAAAAAAAAAAaaaaaaaaAAAAAAAAAAAAAAAAOOaaaaAAAAAAAAAAAAAAAAaaaaaaaaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaaaaaaaaAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA bboxItemImageIgnoreCase aver go absorb_pag find 투 jTextFieldeeee-------- pelo ear","tool_calls":[]},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":17,"total_tokens":145,"completion_tokens":128}}

However, as tested by @chenqianfzh in lora_with_quantization_inference.py, huggyllama/llama-7b works as expected:

python3 -m vllm.entrypoints.openai.api_server --model huggyllama/llama-7b --load-format bitsandbytes --quantization bitsandbytes --enforce-eager --gpu-memory-utilization 0.85

Then,

curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "huggyllama/llama-7b", "messages": [{"role": "user", "content": "Write a recipe for banana bread."}], "max_tokens": 128}'
{"id":"cmpl-7a703b24291a498fbdf1f3f42620b43f","object":"chat.completion","created":1718614331,"model":"huggyllama/llama-7b","choices":[{"index":0,"message":{"role":"assistant","content":"\n2012-01-22 00:56:27 (Reply to: 2012-01-21 16:38:46)\nBanana bread is a traditional recipe that can be made at home. It is a delicious treat. Ingredients include:\n1/2 cup of butter\n1/2 cup of brown sugar\n1/2 cup of banana\n1/2 cup of milk (or more)\n1/2 cup of flour (or more)\n1/2 cup of","tool_calls":[]},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":17,"total_tokens":145,"completion_tokens":128}}

I installed bitsandbytes==0.43.1.

My current environment

Collecting environment information...
WARNING 06-17 09:00:30 _custom_ops.py:14] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.29.3
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.5.0-1020-aws-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A10G
Nvidia driver version: 535.171.04
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      48 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             16
On-line CPU(s) list:                0-15
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 7R32
CPU family:                         23
Model:                              49
Thread(s) per core:                 2
Core(s) per socket:                 8
Socket(s):                          1
Stepping:                           0
BogoMIPS:                           5599.99
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save rdpid
Hypervisor vendor:                  KVM
Virtualization type:                full
L1d cache:                          256 KiB (8 instances)
L1i cache:                          256 KiB (8 instances)
L2 cache:                           4 MiB (8 instances)
L3 cache:                           32 MiB (2 instances)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] torch==2.3.0
[pip3] transformers==4.41.2
[pip3] triton==2.3.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.0.post1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-15	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Hope it helps

[kimdwkimdw]

Same here.

It works when I using LLM class directly, but I got a same error when I use python3 -m vllm.entrypoints.openai.api_server

https://github.com/vllm-project/vllm/blob/845a3f26f9706acafe8fa45ae452846d8cc3b97f/examples/lora_with_quantization_inference.py#L84C1-L90C31

[vrdn-23]

Btw, can I specify 8-bit or 4-bit for BnB quant in vLLM serving API?

+1 to this question. It seems like currently only 4-bit on bitsandbytes is supported?

[chenweize1998]

It appears that the model isn't being quantized properly. I used the script below and printed the parameters of the loaded model. The linear layers (mlp, attention) are quantized, but others are not.

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Meta-Llama-3-70B-Instruct", quantization="bitsandbytes", load_format="bitsandbytes", enforce_eager=True)
print(llm.llm_engine.model_executor.driver_worker.model_runner.model.state_dict())

The output shows the following:

...
('model.layers.79.mlp.gate_up_proj.qweight', tensor([[135],
        [ 86],
        [ 88],
        ...,
        [184],
        [ 37],
        [ 85]], device='cuda:0', dtype=torch.uint8)), ('model.layers.79.mlp.down_proj.qweight', tensor([[116],
        [164],
        [117],
        ...,
        [136],
        [231],
        [209]], device='cuda:0', dtype=torch.uint8)), ('model.layers.79.input_layernorm.weight', tensor([0.1914, 0.2158, 0.2012,  ..., 0.2100, 0.1465, 0.2002], device='cuda:0',
       dtype=torch.bfloat16)), ('model.layers.79.post_attention_layernorm.weight', tensor([0.2412, 0.2324, 0.2109,  ..., 0.2314, 0.1738, 0.2227], device='cuda:0',
       dtype=torch.bfloat16))
...

The mlp.gate_up_proj and mlp.down_proj layers are quantized to torch.uint8, but other weights remain in torch.bfloat16.

[mgoin]

@chenqianfzh can you please look into this issue? I agree this looks like it might be a culprit

[AlfredTino]

@QwertyJack
I believe your issue has something to do with GQA, considering that llama3 use GQA even at 7B scale.

[AlfredTino]

Btw, can I specify 8-bit or 4-bit for BnB quant in vLLM serving API?

+1 to this question. It seems like currently only 4-bit on bitsandbytes is supported?

@QwertyJack @vrdn-23
According to BitsAndBytesModelLoader, it only supports nf4 as quant_type, even fp4 is not supported, not to mention 8-bit.

[AlfredTino]

The mlp.gate_up_proj and mlp.down_proj layers are quantized to torch.uint8, but other weights remain in torch.bfloat16.

@chenweize1998
This is actually how we quantize weights of a LLM. Only linear layers in attentions and ffns are considered, while other linear layers (e.g. embedding and output at the end of all) and other weights (e.g. rms norm) are excluded.

[chenweize1998]

The mlp.gate_up_proj and mlp.down_proj layers are quantized to torch.uint8, but other weights remain in torch.bfloat16.

@chenweize1998 This is actually how we quantize weights of a LLM. Only linear layers in attentions and ffns are considered, while other linear layers (e.g. embedding and output at the end of all) and other weights (e.g. rms norm) are excluded.

@lixuechenAlfred Got it. Haven't worked with quantized models much before. Thanks for bringing it up. Then the problem must be caused by other reasons. Do you have any idea why serving quantized Llama 3 8B in vllm gives poor results?