Skip to content

Conversation

@sstamenk
Copy link
Contributor

@sstamenk sstamenk commented Oct 21, 2025

Purpose

Adds support for bitsandbytes quantized models and Unsloth QLoRA on non-Instinct AMD GPUs that utilize warp size 32.
Requires bitsandbytes #1748 in order to work.

Test Plan

Running models/quantization/test_bitsandbytes.py tests

Test Result

Curently: 7 failed, 1 passed, 4 skipped, 5 warnings

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@mergify mergify bot added the rocm Related to AMD ROCm label Oct 21, 2025
@sstamenk sstamenk force-pushed the enable_bitsandbytes_quant_rocm branch from c2fb252 to 90beac1 Compare October 23, 2025 11:28
@mergify
Copy link

mergify bot commented Oct 23, 2025

Documentation preview: https://vllm--27307.org.readthedocs.build/en/27307/

@mergify mergify bot added documentation Improvements or additions to documentation ci/build deepseek Related to DeepSeek models frontend structured-output v1 tpu Related to Google TPUs labels Oct 23, 2025
@mergify
Copy link

mergify bot commented Oct 23, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @sstamenk.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@sstamenk sstamenk force-pushed the enable_bitsandbytes_quant_rocm branch from 90beac1 to 6a06234 Compare October 23, 2025 11:36
@mergify mergify bot removed tpu Related to Google TPUs needs-rebase labels Oct 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build deepseek Related to DeepSeek models documentation Improvements or additions to documentation frontend kv-connector rocm Related to AMD ROCm structured-output v1

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants