[RFC]: Enabling Suffix Decoding, LSTM Speculator, Sequence Parallelism from Arctic Inference

[sfc-gh-aqiao]

Motivation.

Snowflake AI Research has recently released several optimizations like Suffix Decoding, LSTM Speculation, Sequence Parallelism, SwiftKV etc, improving TTFT, TPOT and throughput for vLLM via a plugin called Arctic Inference (repo: https://github.com/snowflakedb/arcticinference).

Performance Improvements

• 4x faster generation with Suffix Decoding for SWEBench
• 2.4x faster generation with LSTM Speculator for ShareGPT
• 2.8x faster coding with LSTM Speculator for HumanEval
• 2x higher throughput with SwiftKV
• 1.4x throughput than TP=8, but same TTFT as TP=8 with Arctic Ulysses

These optimizations are designed to improve vLLM performance for real production workloads at Snowflake, and will continue to be expanded, maintained, and improved over time

Currently, Arctic Inference is implemented as an external plugin for vLLM. This means users should install both vLLM and Arctic Inference, and then run vLLM with additional configs like:

pip install vllm==v0.8.4 arctic-inference

vllm serve ... \
    --sequence-parallel-size ... \
    --speculative-config '{"method": "suffix"}'

Problem

Currently, Arctic Inference’s primary user is Snowflake’s own production Cortex inference service. However, we believe that other users in the community can also benefit from its features (there is some interest expressed to us after our most recent speculative decoding blog).

First, how can we make these optimizations more accessible to the vLLM community? While some features (e.g. SwiftKV) are more model-specific, others like Suffix Decoding can provide value to users more generally. We believe more users can benefit if they can use these features through vLLM directly.

Second, how can we build a smooth path for “graduating” features from Arctic Inference to vLLM, if desired, as they become available, more generally used, and as their interfaces stabilize?

Proposed Change.

Possible Solutions

1. All Arctic Inference features remain only available through its plugin. Basically, keep things the way it is. Arctic Inference features do not get integrated to vLLM directly, and interested users are directed to the Arctic Inference project to install and use it with their installation of vLLM. Of course, users will need to go through extra discovery + installation steps, but it is the simplest. Perhaps Arctic Inference can be highlighted in documentation as an ecosystem project that helps boost performance.

2. Some Arctic Inference features are directly integrated into vLLM. This way, vLLM users can benefit from those features without first having to discover the Arctic Inference project (some users asked us about this). The integration can be done in two ways:

   1. Copy the code from Arctic Inference directly to vLLM. Does not require bringing on additional dependencies to vLLM, but will require additional effort to port each feature and maintain them. The Snowflake team will also keep maintaining and improving the same features as part of Arctic Inference, which will make this a duplicated effort. This strategy also does not provide a very smooth “graduation” ramp for features that vLLM might want to integrate in the future.
       
      This also defeats the purpose of supporting vLLM plugins, which are meant to allow vLLM to leverage optimizations directly without incorporating them into the vLLM’s code base.

   2. Install Arctic Inference as a dependency of vLLM. Then, certain features (e.g. there is interest in Suffix Decoding) can be integrated directly in vLLM, making them immediately available to all vLLM users. Other Arctic Inference features can also be used by optionally enabling its plugin, defaulting to off. The rest of Arctic Inference can even be installed just-in-time, if desired.
       
      There is a question of maintenance burden due to the dependency on Arctic Inference. However, this burden is much lower than other libraries because Arctic Inference purely exists as an extension to vLLM, which means:

      1. A part of the Arctic Inference package can be explicitly designated for importing into vLLM, and their interface stability guaranteed (e.g. in a arctic_inference.core submodule).
      2. Secondary dependencies are minimal (in fact they are currently non-existent) since Arctic Inference inherits all its major dependencies from vLLM itself, such as torch. For optional features (e.g. those outside arctic_inference.core), the Arctic Inference team is responsible for upgrading them after each substantial vLLM release.

Recommended Solution

Based on the tradeoff between these three options, we are proposing option 2.ii:

• Install Arctic Inference as part of vLLM as a dependency.
• Arctic Inference to create the core submodule, starting with suffix decoding, that has guaranteed interface stability and minimal sub-dependencies.
• The code for integrating suffix decoding with vLLM is contributed to vLLM directly, enabling users to use it natively in vLLM with no extra steps.
• Other features in Arctic Inference can be used on an opt-in basis by enabling the Arctic Inference plugin (off by default).
• Once this integration is established, future features can be smoothly “graduated” into native vLLM usage simply by writing the integration code in vLLM.

Feedback Period.

No response

CC List.

@simon-mo @WoosukKwon @LiuXiaoxuanPKU @sfc-gh-jrasley @sfc-gh-srajbhandari

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[dtransposed]

Hey!

Great to see this RFC. I love how you guys push the support for SOTA SD methods, definitely something that the community is currently missing in V1!

I'd like to add a few cents:

Solution 1. The strongest con of this solution is, as mentioned, a lack of transparency for the user (additional efforts to learn about the features offered by your plugin, which I'd find very cumbersome). Second point is: from my experience, it may be difficult, in the long run, to keep maintaining the plugin, given the rapid development of the core vLLM project. This very much depends on the preferences of your (Snowflake) team as well as your plans for the future — the arctic-inference is currently (arguably) quite lightweight. But if the feature set grows, keeping the plugin separate from the core framework could become difficult to maintain over time.

Solution 2.
i) IMO, this only makes sense if your team is willing to continue development within the core framework's codebase. I assume you don't, given you have created a separate plugin in the first place (for good reasons, I am sure).
ii) I agree that this is the best solution. The example of suffix decoding serves as a good illustration. For those kinds of features (i.e., stand-alone, with little or no dependencies), this would give the user immediate access to your state-of-the-art methods, with minimal discoverability overhead and installation friction. On the other hand, features that require more dependencies (potentially on the rest of Snowlake's internal stack, e.g. arctic-training -to create particular draft models), the user opt-in basis sounds like a sound and elegant solution.

TL;DR: I very much like the chosen solution — the trade-offs seem optimal wrt to smooth development and best user experience. I think you are correctly addressing one of the big pain points of the vLLM community. Thanks @sfc-gh-aqiao

[bradhilton]

Suffix decoding sounds like a close to no lose proposition. I would love to see this mainstreamed into vLLM.

[houseroad]

cc: @zixi-qi , @feikou and @morgendave for proposal review.

[dtransposed]

@sfc-gh-aqiao One shower thought regarding suffix decoding.

As @bradhilton mentioned, suffix decoding should definitely land into core vllm.

However, I wanted to comment on the efficiency of global suffix trees in context of more sophisticated deployments.

As an example, let's image what would happen if our inference system consisted of multiple pods. Each pods runs its own vLLM server. We route requests to appropriate pod to minimize the E2E latency.

In the current setting, each vLLM server would have their own global suffix tree.

This is not efficient:

• each global suffix tree is a silo, the TRULY global suffix tree should "learn" from all the requests of the deployments (all the pods' requests), not only local requests of the server. This would also allow for quickly speculating on the repeated requests from the same user, irrespective of whether we route requests of the user to the same pod or different pods.
• if one pod goes down (as pods do), once the replacement pod comes alive, it would start "training" its global suffix tree from the scratch. Ideally, it should copy the TRULY global suffix tree from other "alive" pod in the deployment.

Maybe I am thinking about features that are on the very long roadmap horizon, but is it something that you potentially considered?

[huachenheli]

@sfc-gh-aqiao One shower thought regarding suffix decoding.

As @bradhilton mentioned, suffix decoding should definitely land into core vllm.

However, I wanted to comment on the efficiency of global suffix trees in context of more sophisticated deployments.

As an example, let's image what would happen if our inference system consisted of multiple pods. Each pods runs its own vLLM server. We route requests to appropriate pod to minimize the E2E latency.

In the current setting, each vLLM server would have their own global suffix tree.

This is not efficient:

• each global suffix tree is a silo, the TRULY global suffix tree should "learn" from all the requests of the deployments (all the pods' requests), not only local requests of the server. This would also allow for quickly speculating on the repeated requests from the same user, irrespective of whether we route requests of the user to the same pod or different pods.
• if one pod goes down (as pods do), once the replacement pod comes alive, it would start "training" its global suffix tree from the scratch. Ideally, it should copy the TRULY global suffix tree from other "alive" pod in the deployment.

Maybe I am thinking about features that are on the very long roadmap horizon, but is it something that you potentially considered?

+1 to your second point. Being able to load a pre-calibrated global suffix tree (e.g. from shadow production traffic) on startup sounds like an important feature to have to maintain its efficiency.

[Neo9061]

Hi @sfc-gh-aqiao, in your paper, for a hybrid approach between model-based speculator say LSTM and suffix tree, how does continuously batching working in this case? Because the way how hybrid approach works is that you check if confidence level of results from tree and if they fall below a threshold, you fall back to model-based speculator. For concurrency level of B for an example, the effective concurrency of requests that go to model-based speculator may be smaller than B right? Is that feasible to have a hybrid approach in vLLM with continuous batching?

[sata-bench]

Does the suffix decoding plugin support for latest V1 engine? does it has to be 0.8.4?

[Neo9061]

@sfc-gh-aqiao One shower thought regarding suffix decoding.

As @bradhilton mentioned, suffix decoding should definitely land into core vllm.

However, I wanted to comment on the efficiency of global suffix trees in context of more sophisticated deployments.

As an example, let's image what would happen if our inference system consisted of multiple pods. Each pods runs its own vLLM server. We route requests to appropriate pod to minimize the E2E latency.

In the current setting, each vLLM server would have their own global suffix tree.

This is not efficient:

• each global suffix tree is a silo, the TRULY global suffix tree should "learn" from all the requests of the deployments (all the pods' requests), not only local requests of the server. This would also allow for quickly speculating on the repeated requests from the same user, irrespective of whether we route requests of the user to the same pod or different pods.
• if one pod goes down (as pods do), once the replacement pod comes alive, it would start "training" its global suffix tree from the scratch. Ideally, it should copy the TRULY global suffix tree from other "alive" pod in the deployment.

Maybe I am thinking about features that are on the very long roadmap horizon, but is it something that you potentially considered?

Hi @dtransposed , with benefit of suffix decoding, even without global-tree, per-request local tree can still benefit performance compared to non-speculative decoding. Will that be a feasible P0 solution to support suffix decoding with per-request local tree? CC: @bradhilton @sfc-gh-aqiao

[bradhilton]

even a local tree could give a speed boost. would be really cool to see even just some initial support make it into main.

[dtransposed]

@bradhilton @Neo9061 Arguably local (per request) tree alone should not give more of a boost then properly tuned ngram decoding, especially after this has landed: #22437. This is just my suspicion, I will be actually verifying it soon :)