V5.5 upstream merge rc #167

gshtras · 2024-09-04T15:22:58Z

v0.5.5 from upstream

Co-authored-by: Cyrus Leung <[email protected]>

…t#7504) Co-authored-by: jack <jack@alex>

…t#7424) Signed-off-by: Wallas Santos <[email protected]> Co-authored-by: youkaichao <[email protected]> Co-authored-by: Nick Hill <[email protected]> Co-authored-by: youkaichao <[email protected]>

…ompt (vllm-project#7126)

Co-authored-by: pavanmantha <[email protected]>

)

Signed-off-by: omrishiv <[email protected]> Co-authored-by: Cyrus Leung <[email protected]>

Co-authored-by: nunjunj <[email protected]> Co-authored-by: nunjunj <[email protected]> Co-authored-by: nunjunj <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: DarkLight1337 <[email protected]>

…ct#7574)

…t#7536)

… checking method (vllm-project#7513)

Co-authored-by: Nick Hill <[email protected]>

…t#8056)

[Bugfix] Fix vllm-project#7592 vllm 0.5.4 enable_chunked_prefill throughput is slightly lower than 0.5.3~0.5.0. (vllm-project#7874)

…t#7881)

…nd (vllm-project#8061)

…m-project#8120) Co-authored-by: Tao He <[email protected]> Co-authored-by: Juelianqvq <[email protected]>

…8124) Signed-off-by: kevin <[email protected]>

…project#7750)

…m-project#8118)

Signed-off-by: kevin <[email protected]>

…wong-amd

…wong-amd

* Miscellaneous changes, Dockerfile components update, remove Cython * Restore Dockerfile and Cython for now

jeejeelee and others added 30 commits August 13, 2024 17:27

[CI/Build]Reduce the time consumption for LoRA tests (vllm-project#7396)

9799280

[misc][ci] fix cpu test with plugins (vllm-project#7489)

ea49e6a

[Bugfix][Docs] Update list of mock imports (vllm-project#7493)

dd164d7

[doc] update test script to include cudagraph (vllm-project#7501)

199adbb

Fix empty output when temp is too low (vllm-project#2937)

c134a46

Co-authored-by: Cyrus Leung <[email protected]>

[ci] fix model tests (vllm-project#7507)

d3d9cb6

[Bugfix][Frontend] Disable embedding API for chat models (vllm-projec…

67d115d

…t#7504) Co-authored-by: jack <jack@alex>

[Misc] Deprecation Warning when setting --engine-use-ray (vllm-projec…

70b746e

…t#7424) Signed-off-by: Wallas Santos <[email protected]> Co-authored-by: youkaichao <[email protected]> Co-authored-by: Nick Hill <[email protected]> Co-authored-by: youkaichao <[email protected]>

[VLM][Core] Support profiling with multiple multi-modal inputs per pr…

3f674a4

…ompt (vllm-project#7126)

[core] [3/N] multi-step args and sequence.py (vllm-project#7452)

2ecf7b1

[TPU] Set per-rank XLA cache (vllm-project#7533)

951fdd6

[Misc] Revert compressed-tensors code reuse (vllm-project#7521)

f55a9ae

llama_index serving integration documentation (vllm-project#6973)

22b39e1

Co-authored-by: pavanmantha <[email protected]>

[Bugfix][TPU] Correct env variable for XLA cache path (vllm-project#7544

fc93e56

)

[Bugfix] update neuron for version > 0.5.0 (vllm-project#7175)

9c1f78d

Signed-off-by: omrishiv <[email protected]> Co-authored-by: Cyrus Leung <[email protected]>

[Misc] Update dockerfile for CPU to cover protobuf installation (vllm…

f4da5f7

…-project#7182)

[Bugfix] Fix default weight loading for scalars (vllm-project#7534)

21313e0

[Bugfix][Harmless] Fix float16 dtype for model_is_embedding (vllm-pro…

9c8e2d1

…ject#7566)

[Misc] Add quantization config support for speculative model. (vllm-p…

b67ae00

…roject#7343)

[Feature]: Add OpenAI server prompt_logprobs support vllm-project#6508 (

f878c8f

vllm-project#7453)

[ci/test] rearrange tests and make adag test soft fail (vllm-project#…

4cd7d47

…7572)

Chat method for offline llm (vllm-project#5049)

3b19e39

Co-authored-by: nunjunj <[email protected]> Co-authored-by: nunjunj <[email protected]> Co-authored-by: nunjunj <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: DarkLight1337 <[email protected]>

[CI] Move quantization cpu offload tests out of fastcheck (vllm-proje…

e165528

…ct#7574)

[Misc/Testing] Use torch.testing.assert_close (vllm-project#7324)

50b8d08

register custom op for flash attn and use from torch.ops (vllm-projec…

54bd9a0

…t#7536)

[Core] Use uvloop with zmq-decoupled front-end (vllm-project#7570)

9587b05

[CI] Fix crashes of performance benchmark (vllm-project#7500)

6fc5b0f

[Bugfix][Hardware][AMD][Frontend] add quantization param to embedding…

0e39a33

… checking method (vllm-project#7513)

support tqdm in notebooks (vllm-project#7510)

ec724a7

[Feature][Hardware][Amd] Add fp8 Linear Layer for Rocm (vllm-project#…

e837b62

…7210)

DarkLight1337 and others added 28 commits August 30, 2024 22:26

[Bugfix] Fix import error in Phi-3.5-MoE (vllm-project#8052)

d05f0a9

[Bugfix] Fix ModelScope models in v0.5.5 (vllm-project#8037)

4f5d844

[BugFix][Core] Multistep Fix Crash on Request Cancellation (vllm-proj…

8423aef

…ect#8059)

[Frontend][VLM] Add support for multiple multi-modal items (vllm-proj…

5231f08

…ect#8049)

[Misc] Optional installation of audio related packages (vllm-project#…

5b86b19

…8063)

[Model] Add Granite model (vllm-project#7436)

f8d6014

Co-authored-by: Nick Hill <[email protected]>

[SpecDecode][Kernel] Flashinfer Rejection Sampling (vllm-project#7244)

e6a26ed

[TPU] Align worker index with node boundary (vllm-project#7932)

e2b2aa5

[Core][Bugfix] Accept GGUF model without .gguf extension (vllm-projec…

4ca65a9

…t#8056)

[Bugfix] Fix internlm2 tensor parallel inference (vllm-project#8055)

dd2a6a8

improve chunked prefill performance

6e36f4f

[Bugfix] Fix vllm-project#7592 vllm 0.5.4 enable_chunked_prefill throughput is slightly lower than 0.5.3~0.5.0. (vllm-project#7874)

[Bugfix] Fix single output condition in output processor (vllm-projec…

0fbc669

…t#7881)

[Bugfix][VLM] Add fallback to SDPA for ViT model running on CPU backe…

ec26653

…nd (vllm-project#8061)

[Performance] Enable chunked prefill and prefix caching together (vll…

bd852f2

…m-project#8120) Co-authored-by: Tao He <[email protected]> Co-authored-by: Juelianqvq <[email protected]>

[CI] Only PR reviewers/committers can trigger CI on PR (vllm-project#…

95a178f

…8124) Signed-off-by: kevin <[email protected]>

[Core] Optimize Async + Multi-step (vllm-project#8050)

6d646d0

[Misc] Raise a more informative exception in add/remove_logger (vllm-…

652c83b

…project#7750)

[CI/Build] make pip install vllm work in macos (for import only) (vll…

c02638e

…m-project#8118)

[ci] Fix GHA workflow (vllm-project#8129)

f1575dc

Signed-off-by: kevin <[email protected]>

[TPU][Bugfix] Fix next_token_ids shape (vllm-project#8128)

0af3abe

[CI] Change PR remainder to avoid at-mentions (vllm-project#8134)

dc0b606

[Misc] Update GPTQ to use vLLMParameters (vllm-project#7976)

2188a60

Initial support for compressed-tensors quantization

be9f84e

Picking fixes from https://github.com/ROCm/vllm/pull/163/files by @ma…

05e67ab

…wong-amd

Merge remote-tracking branch 'upstream/main' into v5.5_upstream_merge_rc

7fd46eb

Update Dockerfile to 6.2, update ROCm components, remove Cython (#166)

7edb2fd

* Miscellaneous changes, Dockerfile components update, remove Cython * Restore Dockerfile and Cython for now

Linters and adapting the sync server to upstream API changes

46c5fed

More linting

abcdce9

gshtras merged commit 6d33657 into main Sep 4, 2024

gshtras deleted the v5.5_upstream_merge_rc branch September 4, 2024 16:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

V5.5 upstream merge rc #167

V5.5 upstream merge rc #167

Uh oh!

gshtras commented Sep 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

94 participants

Uh oh!

V5.5 upstream merge rc #167

V5.5 upstream merge rc #167

Uh oh!

Conversation

gshtras commented Sep 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

94 participants