[Bug]: KeyError: None in task="classify" when a long input (>8192 toks) triggers chunked prefill with max_num_seqs>1 (vLLM 0.10.1)

[rmovva]

🐛 Describe the bug

Summary

LLM.classify(...) crashes with KeyError: None whenever a long example exceeds the default max_num_batched_tokens (8192) and triggers chunked prefill, but only when max_num_seqs >= 2. It succeeds with max_num_seqs=1.

Repro (minimal)

from vllm import LLM

# texts contains many short items + one very long string (> 8192 tokens)
llm = LLM(
    model=model_path,
    task="classify",
    trust_remote_code=True,
    gpu_memory_utilization=0.9,
    max_num_seqs=8,  # fails for 4/8/16; works for 1 (sometimes 2)
)
res = llm.classify(texts)  # vLLM logs: "Chunked prefill enabled with max_num_batched_tokens=8192"

Observed

... vllm/v1/worker/gpu_model_runner.py", line 712, in _prepare_inputs
  tokens = [scheduler_output.num_scheduled_tokens[i] for i in req_ids]
KeyError: None

Expected

No crash when chunked prefill occurs; or a clear error if limits are exceeded.

Environment

vLLM 0.10.1 (pip), Python 3.12

Model: llama-3.1–based RM (task="classify", trust_remote_code=True)

GPU/CUDA/PyTorch: A100, CUDA 12.2, PyTorch 2.7.1

Workarounds

Set max_num_seqs=1 (stable)

Truncate long inputs to ≤8192 tokens

Potential: raise max_num_batched_tokens above the longest input (e.g., 12288/16384) to avoid chunking; or set enforce_eager=True (not performance-optimal)

Suspected cause

When a single long input is partially prefetched (chunked) alongside other requests, a padded/None request ID leaks into _prepare_inputs, which then indexes num_scheduled_tokens[None] and raises KeyError.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Your current environment

Collecting environment information...

CMake version : version 3.16.3
Libc version : glibc-2.31

==============================
PyTorch Info

PyTorch version : 2.7.1+cu126
Is debug build : False
CUDA used to build PyTorch : 12.6
ROCM used to build PyTorch : N/A

==============================
Python Environment

Python version : 3.12.11 | packaged by Anaconda, Inc. | (main, Jun 5 2025, 13:09:17) [GCC 11.2.0] (64-bit runtime)
Python platform : Linux-5.4.0-211-generic-x86_64-with-glibc2.31

==============================
CUDA / GPU Info

Is CUDA available : True
CUDA runtime version : 12.2.140
CUDA_MODULE_LOADING set to : LAZY
GPU models and configuration : GPU 0: NVIDIA A100 80GB PCIe
Nvidia driver version : 535.129.03
cuDNN version : Could not collect
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : True

==============================
CPU Info

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 48 bits physical, 48 bits virtual
CPU(s): 256
On-line CPU(s) list: 0-255
Thread(s) per core: 2
Core(s) per socket: 64
Socket(s): 2
NUMA node(s): 2
Vendor ID: AuthenticAMD
CPU family: 25
Model: 1
Model name: AMD EPYC 7763 64-Core Processor
Stepping: 1
Frequency boost: enabled
CPU MHz: 1491.159
CPU max MHz: 2450.0000
CPU min MHz: 1500.0000

Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload vgif umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca

==============================
Versions of relevant libraries

[pip3] numpy==2.1.2
[pip3] nvidia-cublas-cu12==12.6.4.1
[pip3] nvidia-cuda-cupti-cu12==12.6.80
[pip3] nvidia-cuda-nvrtc-cu12==12.6.77
[pip3] nvidia-cuda-runtime-cu12==12.6.77
[pip3] nvidia-cudnn-cu12==9.5.1.17
[pip3] nvidia-cufft-cu12==11.3.0.4
[pip3] nvidia-cufile-cu12==1.11.1.6
[pip3] nvidia-curand-cu12==10.3.7.77
[pip3] nvidia-cusolver-cu12==11.7.1.2
[pip3] nvidia-cusparse-cu12==12.5.4.2
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] nvidia-nvjitlink-cu12==12.6.85
[pip3] nvidia-nvtx-cu12==12.6.77
[pip3] pyzmq==27.0.1
[pip3] sentence-transformers==5.0.0
[pip3] torch==2.7.1
[pip3] torchaudio==2.7.1
[pip3] torchvision==0.22.1
[pip3] transformers==4.55.2
[pip3] triton==3.3.1
[conda] numpy 2.1.2 pypi_0 pypi
[conda] nvidia-cublas-cu12 12.6.4.1 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.6.80 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.6.77 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.6.77 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.5.1.17 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.3.0.4 pypi_0 pypi
[conda] nvidia-cufile-cu12 1.11.1.6 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.7.77 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.7.1.2 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.5.4.2 pypi_0 pypi
[conda] nvidia-cusparselt-cu12 0.6.3 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.26.2 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.6.85 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.6.77 pypi_0 pypi
[conda] pyzmq 27.0.1 pypi_0 pypi
[conda] sentence-transformers 5.0.0 pypi_0 pypi
[conda] torch 2.7.1 pypi_0 pypi
[conda] torchaudio 2.7.1 pypi_0 pypi
[conda] torchvision 0.22.1 pypi_0 pypi
[conda] transformers 4.55.2 pypi_0 pypi
[conda] triton 3.3.1 pypi_0 pypi

==============================
vLLM Info

ROCM Version : Could not collect
Neuron SDK Version : N/A
vLLM Version : 0.10.1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X 1 N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

==============================
Environment Variables

CUDA_VISIBLE_DEVICES=0
CUDA_VISIBLE_DEVICES=0
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

[DarkLight1337]

cc @maxdebayser

[gjgjh]

I'm running into a similar problem when using embedding task, and my guess is it's a bug in the new version.

[lllsy12138]

I encountered the same error with the embedding task; downgrading to version 0.10.0 resolves it for me.

[rmovva]

Further, it seems that vLLM's classify function entirely fails to produce correct outputs for reward models trained in HF TRL with AutoModelForSequenceClassification? ie: r.outputs.probs[0] (from llm.classify) does not output the same value as expit(model(**enc)[0].squeeze(-1)) (ie running forward pass with HF model).

[princepride]

@DarkLight1337 Can you assign this to me.

[princepride]

@rmovva Can you give me your code that trigger the bug, I am using the latest dev version vLLM but can't trigger the bug.

[aamir-s18]

@princepride here my code which triggers it:

Starting VLM like this:

sudo docker run --runtime nvidia --gpus all -v ~/.cache/huggingface:/root/.cache/huggingface --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" -p 8000:8000 --ipc=host vllm/vllm-openai:latest --model mixedbread-ai/mxbai-rerank-large-v2 --max_model_len 1024 --hf_overrides '{"architectures": ["Qwen2ForSequenceClassification"], "classifier_from_token": ["0", "1"], "method": "from_2_way_softmax"}'

Code to trigger:

Code example with long docs

import requests

url = "http://localhost:8000/score"
MODEL_NAME = "mixedbread-ai/mxbai-rerank-large-v2"

prefix = "<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\n"
suffix = "<|im_end|>\n<|im_start|>assistant\n"

query_template = "{prefix}query: {query}\n"
document_template = "document: {doc}\n{instruction}{suffix}"

instruction = "You are a search relevance expert who evaluates how well documents match search queries. For each query-document pair, carefully analyze the semantic relationship between them, then provide your binary relevance judgment (0 for not relevant, 1 for relevant).\nRelevance:"

queries = [
    "How to use AI in effect"
]

documents = [
        """ad---
title: Getting Started
description: Learn how to use Effect's AI integration packages to define LLM interactions
sidebar:
  order: 1
---

import { Aside, Steps, Tabs, TabItem } from "@astrojs/starlight/components"

In this getting started guide, we will demonstrate how to generate a simple text completion using an LLM provider (OpenAi) using the Effect AI integration packages.

We’ll walk through:
- Writing provider-agnostic logic to interact with an LLM
- Declaring the specific LLM model to use for the interaction
- Using a provider integration to make the program executable""",
        """## Packages

Effect’s AI ecosystem is composed of several focused packages:

### `@effect/ai`

Defines the core abstractions for interacting with LLM provider services. This package defines the generic services and helper utilities needed to build AI-powered applications in a provider-agnostic way.

Use this package to:
- Define your application's interaction with an LLM
- Structure chat or completion flows using Effect
- Build type-safe, declarative AI logic

For detailed API documentation, see the [API Reference](https://effect-ts.github.io/effect/docs/ai/ai).

### `@effect/ai-openai`

Concrete implementations of services from `@effect/ai` backed by the [OpenAI API](https://platform.openai.com). 

Supported services include:
- `AiLanguageModel` (via OpenAI's [Chat Completions API](https://platform.openai.com/docs/api-reference/chat))
- `AiEmbeddingsModel` (via OpenAI's [Embeddings API](https://platform.openai.com/docs/api-reference/embeddings))

For detailed API documentation, see the [API Reference](https://effect-ts.github.io/effect/docs/ai/openai).

### `@effect/ai-anthropic`

Concrete implementations of services from `@effect/ai` backed by the [Anthropic API](https://docs.anthropic.com/en/api/getting-started).

Supported services include:
- `AiLanguageModel` (via Anthropic's [Messages API](https://docs.anthropic.com/en/api/messages))

For detailed API documentation, see the [API Reference](https://effect-ts.github.io/effect/docs/ai/anthropic).""",
        """## Installation

First, we will need to install the base `@effect/ai` package to gain access to the core AI abstractions. In addition, we will need to install at least one provider integration package (in this case `@effect/ai-openai`):

<Tabs syncKey="package-manager">

<TabItem label="npm" icon="seti:npm">

sh showLineNumbers=false
# Install the base package for the core abstractions (always required)
npm install @effect/ai

# Install one (or more) provider integrations
npm install @effect/ai-openai

# Also add the core Effect package (if not already installed)
npm install effect


</TabItem>

<TabItem label="pnpm" icon="pnpm">

sh showLineNumbers=false
# Install the base package for the core abstractions (always required)
pnpm add @effect/ai

# Install one (or more) provider integrations
pnpm add @effect/ai-openai

# Also add the core Effect package (if not already installed)
pnpm add effect


</TabItem>

<TabItem label="Yarn" icon="seti:yarn">

sh showLineNumbers=false
# Install the base package for the core abstractions (always required)
yarn add @effect/ai

# Install one (or more) provider integrations
yarn add @effect/ai-openai

# Also add the core Effect package (if not already installed)
yarn add effect


</TabItem>

<TabItem label="Bun" icon="bun">

sh showLineNumbers=false
# Install the base package for the core abstractions (always required)
bun add @effect/ai

# Install one (or more) provider integrations
bun add @effect/ai-openai

# Also add the core Effect package (if not already installed)
bun add effect


</TabItem>

</Tabs>""",
        """## Why Effect for AI?

Integrating LLMs isn’t just about sending API requests — it’s handling streaming output, retries, rate limits, timeouts, and user-driven side effects, all while keeping your system stable and responsive. Effect provides simple, composable building blocks to model these workflows in a **safe**, **declarative**, and **composable** manner.

By using Effect for your LLM interactions you'll benefit from:

- 🧩 **Provider-Agnostic Architecture**  
  Write your business logic once, and defer choosing the underlying provider (OpenAI, Anthropic, local models, mocks, etc.) until runtime

- 🧪 **Fully Testable**  
  Because LLM interactions are modeled via Effect services, you can mock, simulate, or snapshot responses just by providing an alternative implementation

- 🧵 **Structured Concurrency**  
  Run concurrent LLM calls, cancel stale requests, stream partial results, or race multiple providers — all safely managed by Effect’s structured concurrency model

- 🔍 **Observability**  
  Leverage Effect's built-in tracing, logging, and metrics to instrument your LLM interactions to gain deep insight into  performance bottlenecks or failures in production

...and much more!

## Core Concepts

Effect’s AI integrations are built around thsdjakskjafde idea of **provider-agnostic programming**. Instead of hardcoding calls to a specific LLM provider's API, you describe your interaction using the services provided by the base `@effect/ai` package.

These services expose capabilities such as:
- **Generating Text** – single-shot text generation
- **Generating Embeddings** – vector representations of text for search or retrieval
- **Tool Calling** – structured outputs and tool usage 
- **Streaming** – incremental output for memory efficiency and responsiveness 

Each of these services is defined as an *Effect service* — meaning they can be injected, composed, and tested just like any other dependency in the Effect ecosystem.

This decoupling lets you write your AI code as a pure description of what you want to happen, and resolve *how* it happens later — whether by wiring up OpenAI, Anthropic, a mock service for tests, or even your own custom LLM backend.

---""",
        """---
title: Introduction to Effect AI
description: "Introduction to Effect's AI integrations, a set of packages for interacting with large language models"
sidebar:
  label: Introduction
  order: 0
---

import { Aside } from "@astrojs/starlight/components"

<Aside type="caution" title="Experimental Module">
  The Effect AI integration packages are currently in the experimental / alpha stage. We encourage your feedback to further improve their features.
</Aside>

Welcome to the documentation for Effect's AI integration packages — a set of libraries designed to make working with large language models (LLMs) seamless, flexible, and provider-agnostic.

These packages enable you to write programs that describe *what* you want to do with an LLM — generating completions, handling chat interactions, running function calls — without having to commit to *how* or *where* those operations are executed. 

The core package, [`@effect/ai`](https://www.npmjs.com/package/@effect/ai), provides a high-level, unified interface for modeling LLM interactions, independent of any specific provider. Once you're ready to run your program, you can plug in the services your program requires from our LLM provider integration packages. 

This separation of concerns allows you to:
- Write clean, declarative business logic without worrying about provider-specific quirks
- Easily swap between or combine providers at runtime or during testing
- Take advantage of Effect’s features when building AI-driven workflows

Whether you're building an intelligent agent, an interactive chat app, or a system that leverages LLMs for background tasks, Effect's AI packages offer the flexibility and control you need!

Let’s dive in!""",
        """---
title: Tool Use
description: Equip your LLM interactions with the ability to use tools to perform specific actions
sidebar:
  order: 3
---

Language models are great at generating text, but often we need them to take **real-world actions**, such as querying an API, accessing a database, or calling a service. Most LLM providers support this through **tool use** (also known as *function calling*), where you expose specific operations in your application that the model can invoke. 

Based on the input it receives, a model may choose to **invoke (or call)** one or more tools to augment its response. Your application then runs the corresponding logic for the tool using the parameters provided by the model. You then return the result to the model, allowing it to include the output in its final response.

The `AiToolkit` simplifies tool integration by offering a structured, type-safe approach to defining tools. It takes care of all the wiring between the model and your application - all you have to do is define the tool and implement its behavior.""",
        """---
title: Execution Planning
description: Learn how to create structured execution plans for your LLM interactions
sidebar:
  order: 2
---

Imagine that we've refactored our `generateDadJoke` program from our [Getting Started](/docs/ai/getting-started/) guide. Now, instead of handling all errors internally, the code can **fail with domain-specific issues** like network interruptions or provider outages:

ts twoslash
import type { AiLanguageModel, AiResponse } from "@effect/ai"
import { OpenAiLanguageModel } from "@effect/ai-openai"
import { Data, Effect } from "effect"

class NetworkError extends Data.TaggedError("NetworkError") {}

class ProviderOutage extends Data.TaggedError("ProviderOutage") {}

declare const generateDadJoke: Effect.Effect<
  AiResponse.AiResponse,
  NetworkError | ProviderOutage,
  AiLanguageModel.AiLanguageModel
>

const main = Effect.gen(function*() {
  const response = yield* generateDadJoke
  console.log(response.text)
}).pipe(Effect.provide(OpenAiLanguageModel.model("gpt-4o")))


This is fine, but what if we want to:
- Retry the program a fixed number of times on `NetworkError`s
- Add some backoff delay between retries
- Fallback to a different model provider if OpenAi is down

How can we accomplish such logic?

## Planning LLM Interactions 

The `ExecutionPlan` module from Effect provides a robust method for creating **structured execution plans** for your Effect programs. Rather than making a single model call and hoping that it succeeds, you can use `ExecutionPlan` to describe how to handle errors, retries, and fallbacks in a clear, declarative way.

This is especially useful when:
- You want to fall back to a secondary model if the primary one is unavailable
- You want to retry on transient errors (e.g. network failures)
- You want to control timing between retry attempts""",
        """ts twoslash collapse={18-24}
import { AnthropicLanguageModel } from "@effect/ai-anthropic"
import { OpenAiLanguageModel } from "@effect/ai-openai"
import { AiLanguageModel } from "@effect/ai"
import { Effect } from "effect"

const Gpt4o = OpenAiLanguageModel.model("gpt-4o")
const Claude37 = AnthropicLanguageModel.model("claude-3-7-sonnet-latest")

class DadJokes extends Effect.Service<DadJokes>()("app/DadJokes", {
  effect: Effect.gen(function*() {
    // Yielding the model will return a layer with no requirements
    // 
    //     ┌─── Layer<AiLanguageModel>
    //     ▼
    const gpt = yield* Gpt4o
    const claude = yield* Claude37

    const generateDadJoke = Effect.gen(function*() {
      const response = yield* AiLanguageModel.generateText({
        prompt: "Generate a dad joke"
      })
      console.log(response.text)
      return response
    })

    return {
      generateDadJoke: Effect.provide(generateDadJoke, gpt),
      generateBetterDadJoke: Effect.provide(generateDadJoke, claude)
    }
  })
}) {}

// Programs which utilize the `DadJokes` service have no knowledge of 
// any AI requirements
//
//     ┌─── Effect<void, AiError, DadJokes>
//     ▼
const main = Effect.gen(function*() {
  const dadJokes = yield* DadJokes
  const res1 = yield* dadJokes.generateDadJoke
  const res2 = yield* dadJokes.generateBetterDadJoke
})

// The AI requirements are abstracted away into `Layer` composition
//
//         ┌─── Layer<DadJokes, never, AnthropicClient | OpenAiClient>
//         ▼
DadJokes.Default
""",
        """### Benefits of `AiModel`

There are several benefits to this approach:

**Reusability**

You can provide the same `AiModel` to as many programs as you like.

**Example** (Providing an AiModel to Multiple Programs)

ts twoslash {18-20} collapse={5-11}
import { OpenAiLanguageModel } from "@effect/ai-openai"
import { AiLanguageModel } from "@effect/ai"adjfjk
import { Effect } from "effect"

const generateDadJoke = Effect.gen(function*() {
  const response = yield* AiLanguageModel.generateText({
    prompt: "Generate a dad joke"
  })
  console.log(response.text)
  return response
})

const Gpt4o = OpenAiLanguageModel.model("gpt-4o")adkjfjdksf

const main = Effect.gen(function*() {
  // You can provide the `AiModel` individually to each
  // program, or to all of them at once (as we do here)
  const res1 = yield* generateDadJoke
  const res2 = yield* generateDadJoke
  const res3 = yield* generateDadJoke
}).pipe(Effect.provide(Gpt4o))asdjkfjkds


**Flexibility**

If we know that one model or provider performs better at a given task than another, we can freely mix and match models and providers together. 

For example, if we know Anthropic's Claude generates some really great dad jokes, we can mix it into our existing program with just a few lines of code:

**Example** (Mixing Multiple Providers and Models)""",
        """## End-to-End Usage

The following is the complete program with the desired `AiPlan` fully implemented:

ts twoslash
import type { AiLanguageModel, AiResponse } from "@effect/ai"
import { AnthropicClient, AnthropicLanguageModel } from "@effect/ai-anthropic"
import { OpenAiClient, OpenAiLanguageModel } from "@effect/ai-openai"
import { NodeHttpClient } from "@effect/platform-node"
import { Config, Data, Effect, ExecutionPlan, Layer, Schedule } from "effect"

class NetworkError extends Data.TaggedError("NetworkError") {}

class ProviderOutage extends Data.TaggedError("ProviderOutage") {}

declare const generateDadJoke: Effect.Effect<
  AiResponse.AiResponse,
  NetworkError | ProviderOutage,
  AiLanguageModel.AiLanguageModel
>

const DadJokePlan = ExecutionPlan.make({
  provide: OpenAiLanguageModel.model("gpt-4o"),
  attempts: 3,
  schedule: Schedule.exponential("100 millis", 1.5),
  while: (error: NetworkError | ProviderOutage) => 
    error._tag === "NetworkError"
}, {
  provide: AnthropicLanguageModel.model("claude-3-7-sonnet-latest"),
  attempts: 2,
  schedule: Schedule.exponential("100 millis", 1.5),
  while: (error: NetworkError | ProviderOutage) => 
    error._tag === "ProviderOutage"
})

const main = Effect.gen(function*() {
  const response = yield* generateDadJoke
  console.log(response.text)
}).pipe(Effect.withExecutionPlan(DadJokePlan))

const Anthropic = AnthropicClient.layerConfig({
  apiKey: Config.redacted("ANTHROPIC_API_KEY")
}).pipe(Layer.provide(NodeHttpClient.layerUndici))

const OpenAi = OpenAiClient.layerConfig({
  apiKey: Config.redacted("OPENAI_API_KEY")
}).pipe(Layer.provide(NodeHttpClient.layerUndici))

main.pipe(
  Effect.provide([Anthropic, OpenAi]),
  Effect.runPromisesss
)
"""
]

queries = [
    query_template.format(prefix=prefix, query=query)
    for query in queries
]
documents = [
    document_template.format(doc=doc, suffix=suffix, instruction=instruction) for doc in documents
]


response = requests.post(url,
                         json={
                             "model": MODEL_NAME,
                             "text_1": queries,
                             "text_2": documents,
                             "activation": True,
                             "truncate_prompt_tokens": -1,
                         }).json()

for i, r in enumerate(response["data"]):
    print(i, r["score"])

[RocketRider]

I have a similiar issue with the qwen3-reranker using vLLM 0.10.1. With 0.10.0 it is working fine.

customArgs:
  - "--hf_overrides"
  - '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'
  - "--gpu_memory_utilization"
  - "0.45"

kubectl logs llm-qwen3-reranker-8b-7f5587d879-49z72 -f
Defaulted container "qwen3-reranker-8b" out of: qwen3-reranker-8b, qwen3-reranker-8b-gpu-test (init)
WARNING 08-20 22:32:40 [init.py:1726] argument '--disable-log-requests' is deprecated and replaced with '--enable-log-requests'. This will be removed in v0.12.0.

(EngineCore_0 pid=407) WARNING 08-20 22:32:59 [init.py:3920] Current vLLM config is not set.
Loading safetensors checkpoint shards: 0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 20% Completed | 1/5 [00:00<00:01, 2.01it/s]
Loading safetensors checkpoint shards: 40% Completed | 2/5 [00:01<00:01, 1.79it/s]
Loading safetensors checkpoint shards: 60% Completed | 3/5 [00:01<00:00, 2.60it/s]
Loading safetensors checkpoint shards: 80% Completed | 4/5 [00:01<00:00, 2.20it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:02<00:00, 2.15it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:02<00:00, 2.16it/s]
(EngineCore_0 pid=407)

Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 67/67 [00:03<00:00, 19.43it/s]
(APIServer pid=1) INFO: Started server process [1]
(APIServer pid=1) INFO: Waiting for application startup.
(APIServer pid=1) INFO: Application startup complete.
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.10.1) with config: model='/usr/src/qwen3-reranker-8b', speculative_config=None, tokenizer='/usr/src/qwen3-reranker-8b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=qwen3-reranker-8b, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=PoolerConfig(pooling_type='LAST', normalize=None, dimensions=None, activation=None, softmax=None, step_tag_id=None, returned_token_ids=None, enable_chunked_processing=None, max_embed_len=None), compilation_config={"level":3,"debug_dump_path":"","cache_dir":"/root/.cache/vllm/torch_compile_cache/32f1c18e61","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":1,"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":"/root/.cache/vllm/torch_compile_cache/32f1c18e61/rank_0_0/backbone"},
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[], scheduled_cached_reqs=CachedRequestData(req_ids=['rerank-840792aee89c4012989dbbfdfe09765a-39'], resumed_from_preemption=[false], new_token_ids=[], new_block_ids=[[[756, 757, 758, 759, 760, 761, 762, 763, 764]]], num_computed_tokens=[686]), num_scheduled_tokens={rerank-840792aee89c4012989dbbfdfe09765a-39: 146}, total_num_scheduled_tokens=146, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[52], finished_req_ids=['rerank-840792aee89c4012989dbbfdfe09765a-34', 'rerank-840792aee89c4012989dbbfdfe09765a-17', 'rerank-840792aee89c4012989dbbfdfe09765a-25', 'rerank-840792aee89c4012989dbbfdfe09765a-11', 'rerank-840792aee89c4012989dbbfdfe09765a-36', 'rerank-840792aee89c4012989dbbfdfe09765a-35', 'rerank-840792aee89c4012989dbbfdfe09765a-24', 'rerank-840792aee89c4012989dbbfdfe09765a-12', 'rerank-840792aee89c4012989dbbfdfe09765a-27', 'rerank-840792aee89c4012989dbbfdfe09765a-15', 'rerank-840792aee89c4012989dbbfdfe09765a-28', 'rerank-840792aee89c4012989dbbfdfe09765a-38', 'rerank-840792aee89c4012989dbbfdfe09765a-21', 'rerank-840792aee89c4012989dbbfdfe09765a-37', 'rerank-840792aee89c4012989dbbfdfe09765a-29', 'rerank-840792aee89c4012989dbbfdfe09765a-26', 'rerank-840792aee89c4012989dbbfdfe09765a-18', 'rerank-840792aee89c4012989dbbfdfe09765a-32', 'rerank-840792aee89c4012989dbbfdfe09765a-23', 'rerank-840792aee89c4012989dbbfdfe09765a-31', 'rerank-840792aee89c4012989dbbfdfe09765a-22', 'rerank-840792aee89c4012989dbbfdfe09765a-16', 'rerank-840792aee89c4012989dbbfdfe09765a-9', 'rerank-840792aee89c4012989dbbfdfe09765a-19', 'rerank-840792aee89c4012989dbbfdfe09765a-30', 'rerank-840792aee89c4012989dbbfdfe09765a-33', 'rerank-840792aee89c4012989dbbfdfe09765a-20', 'rerank-840792aee89c4012989dbbfdfe09765a-10', 'rerank-840792aee89c4012989dbbfdfe09765a-13', 'rerank-840792aee89c4012989dbbfdfe09765a-14', 'rerank-840792aee89c4012989dbbfdfe09765a-8'], free_encoder_input_ids=[], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null)
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.005662393162393142, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0), spec_decoding_stats=None, num_corrupted_reqs=0)
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] EngineCore encountered a fatal error.
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] Traceback (most recent call last):
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 693, in run_engine_core
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] engine_core.run_busy_loop()
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 720, in run_busy_loop
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] self._process_engine_step()
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 745, in _process_engine_step
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] outputs, model_executed = self.step_fn()
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] ^^^^^^^^^^^^^^
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 288, in step
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] model_output = self.execute_model_with_error_logging(
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 274, in execute_model_with_error_logging
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] raise err
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 265, in execute_model_with_error_logging
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] return model_fn(scheduler_output)
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 87, in execute_model
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] output = self.collective_rpc("execute_model",
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] answer = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/init.py", line 3007, in run_method
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] return func(*args, **kwargs)
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] return func(*args, **kwargs)
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 362, in execute_model
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] output = self.model_runner.execute_model(scheduler_output,
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] return func(*args, **kwargs)
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1522, in execute_model
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] max_query_len) = (self._prepare_inputs(scheduler_output))
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 712, in _prepare_inputs
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] tokens = [scheduler_output.num_scheduled_tokens[i] for i in req_ids]
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
(EngineCore_0 pid=407) ERROR 08-20 22:34:41 [core.py:702] KeyError: None
(EngineCore_0 pid=407) Process EngineCore_0:
(EngineCore_0 pid=407) Traceback (most recent call last):
(EngineCore_0 pid=407) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_0 pid=407) self.run()
(EngineCore_0 pid=407) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_0 pid=407) self._target(*self._args, **self._kwargs)
(EngineCore_0 pid=407) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 704, in run_engine_core
(EngineCore_0 pid=407) raise e
(EngineCore_0 pid=407) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 693, in run_engine_core
(EngineCore_0 pid=407) engine_core.run_busy_loop()
(EngineCore_0 pid=407) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 720, in run_busy_loop
(EngineCore_0 pid=407) self._process_engine_step()
(EngineCore_0 pid=407) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 745, in _process_engine_step
(EngineCore_0 pid=407) outputs, model_executed = self.step_fn()
(EngineCore_0 pid=407) ^^^^^^^^^^^^^^
(EngineCore_0 pid=407) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 288, in step
(EngineCore_0 pid=407) model_output = self.execute_model_with_error_logging(
(EngineCore_0 pid=407) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 274, in execute_model_with_error_logging
(EngineCore_0 pid=407) raise err
(EngineCore_0 pid=407) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 265, in execute_model_with_error_logging
(EngineCore_0 pid=407) return model_fn(scheduler_output)
(EngineCore_0 pid=407) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 87, in execute_model
(EngineCore_0 pid=407) output = self.collective_rpc("execute_model",
(EngineCore_0 pid=407) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
(EngineCore_0 pid=407) answer = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_0 pid=407) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) File "/usr/local/lib/python3.12/dist-packages/vllm/utils/init.py", line 3007, in run_method
(EngineCore_0 pid=407) return func(*args, **kwargs)
(EngineCore_0 pid=407) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(EngineCore_0 pid=407) return func(*args, **kwargs)
(EngineCore_0 pid=407) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 362, in execute_model
(EngineCore_0 pid=407) output = self.model_runner.execute_model(scheduler_output,
(EngineCore_0 pid=407) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(EngineCore_0 pid=407) return func(*args, **kwargs)
(EngineCore_0 pid=407) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1522, in execute_model
(EngineCore_0 pid=407) max_query_len) = (self._prepare_inputs(scheduler_output))
(EngineCore_0 pid=407) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=407) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 712, in _prepare_inputs
(EngineCore_0 pid=407) tokens = [scheduler_output.num_scheduled_tokens[i] for i in req_ids]
(EngineCore_0 pid=407) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
(EngineCore_0 pid=407) KeyError: None
(APIServer pid=1) ERROR 08-20 22:34:41 [async_llm.py:430] AsyncLLM output_handler failed.
(APIServer pid=1) ERROR 08-20 22:34:41 [async_llm.py:430] Traceback (most recent call last):
(APIServer pid=1) ERROR 08-20 22:34:41 [async_llm.py:430] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 389, in output_handler
(APIServer pid=1) ERROR 08-20 22:34:41 [async_llm.py:430] outputs = await engine_core.get_output_async()
(APIServer pid=1) ERROR 08-20 22:34:41 [async_llm.py:430] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) ERROR 08-20 22:34:41 [async_llm.py:430] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 843, in get_output_async
(APIServer pid=1) ERROR 08-20 22:34:41 [async_llm.py:430] raise self._format_exception(outputs) from None
(APIServer pid=1) ERROR 08-20 22:34:41 [async_llm.py:430] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=1) INFO: 169.254.3.1:13752 - "POST /rerank HTTP/1.1" 500 Internal Server Error
(APIServer pid=1) INFO: Shutting down
(APIServer pid=1) INFO: Waiting for application shutdown.
(APIServer pid=1) INFO: Application shutdown complete.
(APIServer pid=1) INFO: Finished server process [1]

[gjgjh]

I'm running into a similar problem when using embedding task, and my guess is it's a bug in the new version.

I'm using vllm 0.10.1 with Qwen3/Qwen3-Embedding-0.6B. And --max-num-seqs only works when set to 1.

[njhill]

This is likely the same cause as #23381, which I'm looking to fix with #23398.

[princepride]

It seems to be this problem. I will test it after your code is merged.