[Bug]: Major issues with guided generation / structured output in vLLM (up to and including v0.8.1); many examples provided by vllm in /examples and structured_outputs.html doc do not work

[agm-eratosth]

Your current environment

The output of `python collect_env.py`

Your output of `python collect_env.py` here

🐛 Describe the bug

I'm following the examples entirely from vllm's example code for structured output found here: https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_completion_structured_outputs.py and getting errors in many examples. Additionally using response_format has had many errors which have varied from release v0.6.3-post1 through v0.8.1; I've tested all versions between the two previously mentioned with llama 3.3 and found inconsistencies. The unit tests occurring for guided generation do not seem robust; especially when the examples provided in the examples section don't work when running without modification.

Specifically the issues start with the following code in openai_chat_completion_structured_outputs.py; all subsequent examples in that .py file are also throwing errors:

# Guided decoding by JSON using Pydantic schema
class CarType(str, Enum):
    sedan = "sedan"
    suv = "SUV"
    truck = "Truck"
    coupe = "Coupe"


class CarDescription(BaseModel):
    brand: str
    model: str
    car_type: CarType


json_schema = CarDescription.model_json_schema()

prompt = ("Generate a JSON with the brand, model and car_type of"
          "the most iconic car from the 90's")
completion = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct",
    messages=[{
        "role": "user",
        "content": prompt,
    }],
    extra_body={"guided_json": json_schema},
)
print(completion.choices[0].message.content)

The error I get is:

BadRequestError: Error code: 400 - {'object': 'error', 'message': 'The provided JSON schema contains features not supported by xgrammer.', 'type': 'BadRequestError', 'param': None, 'code': 400}

As mentioned previously, I'm unable to run ALL other subsequent examples in /examples/online_serving/openai_chat_completion_structured_outputs.py without receiving some sort of error; those are "Guided decoding by Grammar", "Extra backend options".

I'm also running into many issues using response format. For example, when running this specific code from openai:
https://github.com/openai/openai-cookbook/blob/main/examples/Leveraging_model_distillation_to_fine-tune_a_model.ipynb

where response format is defined as:

response_format = {
    "type": "json_schema",
    "json_schema": {
        "name": "grape-variety",
        "schema": {
            "type": "object",
            "properties": {
                "variety": {
                    "type": "string",
                    "enum": varieties.tolist()
                }
            },
            "additionalProperties": False,
            "required": ["variety"],
        },
        "strict": True
    }
}

Results in the following error on v0.8.1 when running the cell which contains

answer = call_model('gpt-4o', generate_prompt(df_france_subset.iloc[0], varieties))
answer

(note: gpt-4o was replaced with "meta-llama/Llama-3.3-70B-Instruct")

BadRequestError: Error code: 400 - {'object': 'error', 'message': 'The provided JSON schema contains features not supported by xgrammar.', 'type': 'BadRequestError', 'param': None, 'code': 400}

And results in the following - different - error on v0.7.3:

BadRequestError: Error code: 400 {'object': 'error', 'message': '[{'type': 'extra_forbidden', 'loc': ('body', 'metadata'), 'msg':, 'Extra inputs are not permitted', 'input': {'distillation': 'wine-distillation'}}, {'type': 'extra_forbidden', 'loc': ('body', 'store'), 'msg': 'Extra Inputs are not permitted', 'intput': True}]", 'type: 'BadRequestError, 'param': None, 'code': 400}

Edit: Also seeing issues with some of the examples found in: https://docs.vllm.ai/en/latest/features/structured_outputs.html

This example, which used to work in v0.7.3, now returns an error:

from pydantic import BaseModel
from openai import OpenAI


class Info(BaseModel):
    name: str
    age: int


client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="dummy")
completion = client.beta.chat.completions.parse(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "My name is Cameron, I'm 28. What's my name and age?"},
    ],
    response_format=Info,
    extra_body=dict(guided_decoding_backend="outlines"),
)

message = completion.choices[0].message
print(message)
assert message.parsed
print("Name:", message.parsed.name)
print("Age:", message.parsed.age)

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[subnet-dev]

Same issue here, seems to be related to Enum usage. I also tried replacing CarType with car_type: list[Literal["sedan", "suv", "truck", "coupe"]] but the problem persists. Unfortunately, this is blocking us from using vLLM for structured outputs.

It appears that the JSON schema validation is being rejected by the has_xgrammar_unsupported_json_features function due to a compatibility issue.

vllm/vllm/v1/structured_output/utils.py

Lines 18 to 70 in 6ebaf9a

	def has_xgrammar_unsupported_json_features(schema: dict[str, Any]) -> bool:
	"""Check if JSON schema contains features unsupported by xgrammar."""
	def check_object(obj: dict[str, Any]) -> bool:
	if not isinstance(obj, dict):
	return False
	# Check for pattern restrictions
	if "pattern" in obj:
	return True
	# Check for enum restrictions
	if "enum" in obj:
	return True
	# Check for numeric ranges
	if obj.get("type") in ("integer", "number") and any(
	key in obj
	for key in ("minimum", "maximum", "exclusiveMinimum",
	"exclusiveMaximum", "multipleOf")):
	return True
	# Check for array unsupported keywords
	if obj.get("type") == "array" and any(
	key in obj
	for key in ("uniqueItems", "contains", "minContains",
	"maxContains", "minItems", "maxItems")):
	return True
	# Unsupported keywords for strings
	if obj.get("type") == "string" and any(
	key in obj for key in ("minLength", "maxLength", "format")):
	return True
	# Unsupported keywords for objects
	if obj.get("type") == "object" and any(
	key in obj for key in ("minProperties", "maxProperties",
	"propertyNames", "patternProperties")):
	return True
	# Recursively check all nested objects and arrays
	for value in obj.values():
	if isinstance(value, dict):
	if check_object(value):
	return True
	elif isinstance(value, list):
	for item in value:
	if isinstance(item, dict) and check_object(item):
	return True
	return False
	return check_object(schema)

Python code sample

class CarDescription(BaseModel):
    brand: str
    model: str
    car_type: CarType
    car_type2: list[Literal["sedan", "suv", "truck", "coupe"]]

Json schema

{
   "$defs":{
      "CarType":{
         "enum":[
            "sedan",
            "SUV",
            "Truck",
            "Coupe"
         ],
         "title":"CarType",
         "type":"string"
      }
   },
   "properties":{
      "brand":{
         "title":"Brand",
         "type":"string"
      },
      "car_type":{
         "$ref":"#/$defs/CarType"
      },
      "car_type2":{
         "items":{
            "enum":[
               "sedan",
               "suv",
               "truck",
               "coupe"
            ],
            "type":"string"
         },
         "title":"Car Type2",
         "type":"array"
      },
      "model":{
         "title":"Model",
         "type":"string"
      }
   },
   "required":[
      "brand",
      "model",
      "car_type",
      "car_type2"
   ],
   "title":"CarDescription",
   "type":"object"
}

Does anyone have more in-depth knowledge of xgrammar to determine if this is a validation issue or a missing feature in xgrammar?

[HadiSDev]

This is a huge issue. One of the main features of using vllm is not working.

[chaunceyjiang]

@agm-eratosth Can you provide me with an openai client code? Convenient for me to replicate locally.

[subnet-dev]

@chaunceyjiang The following example script does not work for example (I replaced Qwen/Qwen2.5-3B-Instruct by eta-llama/Llama-3.3-70B-Instruct): https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_completion_structured_outputs.py

vLLM command

python3 -m vllm.entrypoints.openai.api_server --model=meta-llama/Llama-3.3-70B-Instruct --tensor-parallel-size=4 --port=8080 --enable-auto-tool-choice --tool-call-parser=llama3_json

Client error

python3 openai_chat_completion_structured_outputs.py
positive
alan_turing@enigma.com
Traceback (most recent call last):
  File "/tmp/openai_chat_completion_structured_outputs.py", line 62, in <module>
    completion = client.chat.completions.create(
        model="meta-llama/Llama-3.3-70B-Instruct",
    ...<4 lines>...
        extra_body={"guided_json": json_schema},
    )
  File "/tmp/.venv/openai/lib/python3.13/site-packages/openai/_utils/_utils.py", line 279, in wrapper
    return func(*args, **kwargs)
  File "/tmp/.venv/openai/lib/python3.13/site-packages/openai/resources/chat/completions/completions.py", line 914, in create
    return self._post(
           ~~~~~~~~~~^
        "/chat/completions",
        ^^^^^^^^^^^^^^^^^^^^
    ...<41 lines>...
        stream_cls=Stream[ChatCompletionChunk],
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/tmp/.venv/openai/lib/python3.13/site-packages/openai/_base_client.py", line 1242, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/.venv/openai/lib/python3.13/site-packages/openai/_base_client.py", line 919, in request
    return self._request(
           ~~~~~~~~~~~~~^
        cast_to=cast_to,
        ^^^^^^^^^^^^^^^^
    ...<3 lines>...
        retries_taken=retries_taken,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/tmp/.venv/openai/lib/python3.13/site-packages/openai/_base_client.py", line 1023, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': 'The provided JSON schema contains features not supported by xgrammar.', 'type': 'BadRequestError', 'param': None, 'code': 400}

vLLM logs:

INFO 03-25 07:47:46 [logger.py:39] Received request chatcmpl-4834f1168dda460daaaf299fdfd350fd: prompt: '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 26 Jul 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nClassify this sentiment: vLLM is wonderful!<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.9, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=131026, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=GuidedDecodingParams(json=None, regex=None, choice=['positive', 'negative'], grammar=None, json_object=None, backend=None, whitespace_pattern=None), extra_args=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO 03-25 07:47:46 [async_llm.py:204] Added request chatcmpl-4834f1168dda460daaaf299fdfd350fd.
INFO:     127.0.0.1:57996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 03-25 07:47:46 [logger.py:39] Received request chatcmpl-c60e24820664490cafe9dd3fa3d505b0: prompt: '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 26 Jul 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nGenerate an email address for Alan Turing, who works in Enigma.End in .com and new line. Example result:alan.turing@enigma.com<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.9, top_k=-1, min_p=0.0, seed=None, stop=['\n'], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=131006, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=GuidedDecodingParams(json=None, regex='\\w+@\\w+\\.com\\n', choice=None, grammar=None, json_object=None, backend=None, whitespace_pattern=None), extra_args=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO 03-25 07:47:46 [async_llm.py:204] Added request chatcmpl-c60e24820664490cafe9dd3fa3d505b0.
INFO:     127.0.0.1:57996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 03-25 07:47:46 [logger.py:39] Received request chatcmpl-fc4f05534f2f4f62994ac6dab1196f59: prompt: "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 26 Jul 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nGenerate a JSON with the brand, model and car_type ofthe most iconic car from the 90's<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n", params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.9, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=131016, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=GuidedDecodingParams(json={'$defs': {'CarType': {'enum': ['sedan', 'SUV', 'Truck', 'Coupe'], 'title': 'CarType', 'type': 'string'}}, 'properties': {'brand': {'title': 'Brand', 'type': 'string'}, 'model': {'title': 'Model', 'type': 'string'}, 'car_type': {'$ref': '#/$defs/CarType'}}, 'required': ['brand', 'model', 'car_type'], 'title': 'CarDescription', 'type': 'object'}, regex=None, choice=None, grammar=None, json_object=None, backend=None, whitespace_pattern=None), extra_args=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO:     127.0.0.1:57996 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request

[agm-eratosth]

Use @subnet-dev's code for how to launch vllm using python. If you want to use the container (reference https://docs.vllm.ai/en/stable/deployment/docker.html) :

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:v0.8.1 \
    --model meta-llama/Llama-3.3-70B-Instruct \
    --enable-auto-tool-choice \
    --tool-call-parser=llama3_json

If you have your models stored in a specific path that isn't ~/.cache/huggingface:

docker run --runtime nvidia --gpus all \
    -v /path/to/model/Llama-3.3-70B-Instruct:/internal/docker/path/to/model/Llama-3.3-70B-Instruct \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:v0.8.1 \
    --model /internal/docker/path/to/model/Llama-3.3-70B-Instruct \
    --enable-auto-tool-choice \
    --tool-call-parser=llama3_json

podman:

podman run -it --security-opt=label=disable --device=nvidia.com/gpu=all \
    -p 8000:8000 \
    -v /path/to/model/Llama-3.3-70B-Instruct:/internal/docker/path/to/model/Llama-3.3-70B-Instruct \
    --ipc=host \
    vllm/vllm-openai:v0.8.1 \
    --model /internal/docker/path/to/model/Llama-3.3-70B-Instruct
    --enable-auto-tool-choice \
    --tool-call-parser=llama3_json

Client side code (assuming you're running on the same machine as the vLLM instance, otherwise replace localhost with the IP of the remote server):

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

Use "EMTPY" when you don't have an API_KEY set on the server side.

from pydantic import BaseModel
from openai import OpenAI


class Info(BaseModel):
    name: str
    age: int

completion = client.beta.chat.completions.parse(
    model="/internal/docker/path/to/model/Llama-3.3-70B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "My name is Cameron, I'm 28. What's my name and age?"},
    ],
    response_format=Info,
    extra_body=dict(guided_decoding_backend="outlines"),
)

message = completion.choices[0].message
print(message)
assert message.parsed
print("Name:", message.parsed.name)
print("Age:", message.parsed.age)