[Bug]: structured output with xgrammar using vllm serve with llama-8b fails results in os error OSError: OSError: (...)/.cache/torch_extensions/py312_cu124/xgrammar/xgrammar.so: cannot open shared object file: No such file or directory

[ExplodedViewMelon]

Your current environment

The output of `python collect_env.py`

INFO 02-19 18:59:41 __init__.py:190] Automatically detected platform cuda.
Collecting environment information...
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Rocky Linux 8.10 (Green Obsidian) (x86_64)
GCC version: (GCC) 13.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.28

Python version: 3.12.3 (main, Dec 16 2024, 18:23:37) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-4.18.0-553.16.1.el8_10.x86_64-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB
GPU 2: NVIDIA A100-SXM4-40GB
GPU 3: NVIDIA A100-SXM4-40GB

Nvidia driver version: 560.35.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              128
On-line CPU(s) list: 0-127
Thread(s) per core:  2
Core(s) per socket:  32
Socket(s):           2
NUMA node(s):        8
Vendor ID:           AuthenticAMD
CPU family:          23
Model:               49
Model name:          AMD EPYC 7452 32-Core Processor
Stepping:            0
CPU MHz:             2350.000
CPU max MHz:         2350.0000
CPU min MHz:         1500.0000
BogoMIPS:            4700.21
Virtualization:      AMD-V
L1d cache:           32K
L1i cache:           32K
L2 cache:            512K
L3 cache:            16384K
NUMA node0 CPU(s):   0-7,64-71
NUMA node1 CPU(s):   8-15,72-79
NUMA node2 CPU(s):   16-23,80-87
NUMA node3 CPU(s):   24-31,88-95
NUMA node4 CPU(s):   32-39,96-103
NUMA node5 CPU(s):   40-47,104-111
NUMA node6 CPU(s):   48-55,112-119
NUMA node7 CPU(s):   56-63,120-127
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-ml-py==12.570.86
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.1
[pip3] torch==2.5.1
[pip3] torchaudio==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.49.0
[pip3] triton==3.1.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.7.2
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	GPU1	GPU2	GPU3	NIC0	NIC1	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV4	NV4	NV4	SYS	SYS	24-31,88-95	3		N/A
GPU1	NV4	 X 	NV4	NV4	PIX	SYS	8-15,72-79	1		N/A
GPU2	NV4	NV4	 X 	NV4	SYS	PIX	56-63,120-127	7		N/A
GPU3	NV4	NV4	NV4	 X 	SYS	SYS	40-47,104-111	5		N/A
NIC0	SYS	PIX	SYS	SYS	 X 	SYS				
NIC1	SYS	SYS	PIX	SYS	SYS	 X 				

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1

LD_LIBRARY_PATH=/mnt/tier2/users/*my_id*/user_venv/lib/python3.12/site-packages/cv2/../../lib64:/apps/USE/easybuild/release/2024.1/software/Python/3.12.3-GCCcore-13.3.0/lib:/apps/USE/easybuild/release/2024.1/software/OpenSSL/3/lib:/apps/USE/easybuild/release/2024.1/software/libffi/3.4.5-GCCcore-13.3.0/lib64:/apps/USE/easybuild/release/2024.1/software/XZ/5.4.5-GCCcore-13.3.0/lib:/apps/USE/easybuild/release/2024.1/software/SQLite/3.45.3-GCCcore-13.3.0/lib:/apps/USE/easybuild/release/2024.1/software/Tcl/8.6.14-GCCcore-13.3.0/lib:/apps/USE/easybuild/release/2024.1/software/libreadline/8.2-GCCcore-13.3.0/lib:/apps/USE/easybuild/release/2024.1/software/ncurses/6.5-GCCcore-13.3.0/lib:/apps/USE/easybuild/release/2024.1/software/bzip2/1.0.8-GCCcore-13.3.0/lib:/apps/USE/easybuild/release/2024.1/software/binutils/2.42-GCCcore-13.3.0/lib:/apps/USE/easybuild/release/2024.1/software/zlib/1.3.1-GCCcore-13.3.0/lib:/apps/USE/easybuild/release/2024.1/software/GCCcore/13.3.0/lib64
CUDA_VISIBLE_DEVICES=0,1,2,3
CUDA_VISIBLE_DEVICES=0,1,2,3
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

hi - I get an error trying to generate structured prompts using xgrammar.
The below works perfectly fine using outlines as engine, it only seems to be a problem when running xgrammar (default)

Running 'vllm serve meta-llama/Llama-3.1-8B-Instruct' opens the server without any problems.

When I try to run the below in python, I get the error from the server logs:

OSError: OSError: *path-to-my-work-dir*/.cache/torch_extensions/py312_cu124/xgrammar/xgrammar.so: cannot open shared object file: No such file or directory

from pydantic import BaseModel
from enum import Enum


class Answer(BaseModel):
    reasoning: str
    answer: str


json_schema = Answer.model_json_schema()

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy",
)


completion = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "What is the capital of France?",
        }
    ],
    extra_body={
        "guided_json": json_schema,
    },
)
print(completion.choices[0].message.content)

I run it using a fresh virtual enviroment, with only vllm installed.
The script used to produce this is almost a copy of the one on the structured prompting tutorial.
Adding

"guided_decoding_backend": "outlines",

to the extra_body argument makes it run perfectly fine.
I've both tried reinstalling vllm from scratch but it is the same. Both when specifying "guided_decoding_backend": "xgrammar" or just letting it fall back to default results in the above error.

Hope that this is not a duplicate and that you might be able to help me!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ExplodedViewMelon]

Here are a few more snippets from the error:
(...)
File "/work3/s174032/speciale/vllm_venv/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2416, in _join_cuda_home
raise OSError('CUDA_HOME environment variable is not set. '
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
INFO 02-19 20:17:15 logger.py:39] Received request chatcmpl-e0db07a6103a45e4adea8d915cfcaded: prompt: '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023
\nToday Date: 26 Jul 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWhat is the capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', params: SamplingParams(n=1,
presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False
, ignore_eos=False, max_tokens=131030, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=Guided
DecodingParams(json={'properties': {'reasoning': {'title': 'Reasoning', 'type': 'string'}, 'answer': {'title': 'Answer', 'type': 'string'}}, 'required': ['reasoning', 'answer'], 'title': 'Answer', 'type'
: 'object'}, regex=None, choice=None, grammar=None, json_object=None, backend=None, whitespace_pattern=None)), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO: 127.0.0.1:52034 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
(...)
OSError: /zhome/fb/b/127390/.cache/torch_extensions/py311_cu124/xgrammar/xgrammar.so: cannot open shared object file: No such file or directory
INFO 02-19 20:17:16 logger.py:39] Received request chatcmpl-59b744526c4b464aa30bdc69d13145f3: prompt: '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023
\nToday Date: 26 Jul 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWhat is the capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', params: SamplingParams(n=1,
presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False
, ignore_eos=False, max_tokens=131030, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=Guided
DecodingParams(json={'properties': {'reasoning': {'title': 'Reasoning', 'type': 'string'}, 'answer': {'title': 'Answer', 'type': 'string'}}, 'required': ['reasoning', 'answer'], 'title': 'Answer', 'type'
: 'object'}, regex=None, choice=None, grammar=None, json_object=None, backend=None, whitespace_pattern=None)), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO: 127.0.0.1:52044 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
(...)
OSError: /zhome/fb/b/127390/.cache/torch_extensions/py311_cu124/xgrammar/xgrammar.so: cannot open shared object file: No such file or directory
INFO 02-19 20:17:16 logger.py:39] Received request chatcmpl-59b744526c4b464aa30bdc69d13145f3: prompt: '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023
\nToday Date: 26 Jul 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWhat is the capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', params: SamplingParams(n=1,
presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False
, ignore_eos=False, max_tokens=131030, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=Guided
DecodingParams(json={'properties': {'reasoning': {'title': 'Reasoning', 'type': 'string'}, 'answer': {'title': 'Answer', 'type': 'string'}}, 'required': ['reasoning', 'answer'], 'title': 'Answer', 'type'
: 'object'}, regex=None, choice=None, grammar=None, json_object=None, backend=None, whitespace_pattern=None)), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO: 127.0.0.1:52044 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application

[justinthelaw]

The CUDA_HOME error seems to only occur for me when the containers running the vLLM server does not contain the necessary CUDA dependencies and/or the container was not provided GPU passthrough using the NVIDIA Container Toolkit. For example, if you try to use xgrammar in the CPU Dockerfile, as provided by vLLM, it will fail because the vLLM xgrammar implementation, whether intentionally or unintentionally, requires CUDA support.

Basically: Are you sure the vLLM server instance you are running is actually using CUDA properly?

Another error you may run into after this one, or possibly before, is the fact that xgrammar is not pre-compiled when vLLM instantiates it for structured generation. Unlike outlines with is natively Python, xgrammar is C++, and you will need to have the requirements-build.txt (e.g., ninja) to be installed in the vLLM server container so that the xgrammar code can be compiled. This may or may not be intentional as well.

Both of the possible errors are due to the vLLM implementation of xgrammar. I haven't dived too deep into these specific issues yet, but it may be worth a deeper look!

[ExplodedViewMelon]

The CUDA_HOME error seems to only occur for me when the containers running the vLLM server does not contain the necessary CUDA dependencies and/or the container was not provided GPU passthrough using the NVIDIA Container Toolkit. For example, if you try to use xgrammar in the CPU Dockerfile, as provided by vLLM, it will fail because the vLLM xgrammar implementation, whether intentionally or unintentionally, requires CUDA support.

Basically: Are you sure the vLLM server instance you are running is actually using CUDA properly?

I'm quite sure, since it works perfectly fine using outlines as backend. I just ran '''nvidia-smi''' after spinning up the server, and it shows that the model has been loaded onto the GPUs.

[ExplodedViewMelon]

The provided example from https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_completion_structured_outputs.py also fails when using xgrammar

serving using '''vllm serve Qwen/Qwen2.5-3B-Instruct --tensor-parallel-size 2''' running on the same node, same venv etc. etc.

Using outlines works fine

[mpetruc]

same here

vllm serve ./models/deepseek67b --tensor-parallel-size 2

The application starts fine, but errors out at inference:

response = client.chat.completions.create(
                    model = "./models/deepseek67b",
                    
                    messages=[
                        {"role": "system", "content": f"""You are a helpful assistant"""},
                        {"role": "user", "content": f"""
                        What is the capital of france? Return a JSON with 2 fields, country and capital.
                        """}
                    ],
                    max_tokens=100,
                    response_format={ "type": "json_object" },
                    temperature=0.1,
                    n=10
                    
                    )

Console log:

OSError: /home/username/.cache/torch_extensions/py312_cu124/xgrammar/xgrammar.so: cannot open shared object file: No such file or directory

vLLM 0.7.2 was installed using uv, if that matters in any way.

[mpetruc]

More experiments:

• not specifying response_format={ "type": "json_object" }: it works fine, getting about 13T/s on 2xA100 80G

Avg prompt throughput: 27.4 tokens/s, Avg generation throughput: 13.7 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.

• with

extra_body={
        "guided_json": json_schema }

getting the xgrammar.so: cannot open shared object file: No such file or directory

• with

extra_body={
        "guided_json": json_schema,
        "guided_decoding_backend": "outlines"
    }

it works, very slow:

 Avg prompt throughput: 37.3 tokens/s, Avg generation throughput: 0.9 tokens/s, Running: 10 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.9%, CPU KV cache usage: 0.0%.

Hope it helps.

[Linnore]

I came into the same problem. In my case I served a Qwen2.5-32B model, and it worked well for non-structure-output requests. However, for json-mode or json-shema structured-output requests, the vllm server log shows

OSError: ....my_user_home.../.cache/torch_extensions/py312_cu124/xgrammar/xgrammar.so: cannot open shared object file: No such file or directory

Just same as what @mpetruc described.

Is there a way to compile the xgrammar.so manually?

[mpetruc]

I believe i fixed my problem. Like i said, I used uv to install vllm. Here's the steps i took:

• making sure that the system .cache folder is located somewhere where uv has full access to. In my case i moved it to my /home/user/ folder and created a symlink to where the original location was (can't remember where now).
• uninstall vllm (delete the vllm virtual environment created by uv)
• reinstall vllm (uv tool install vllm)
  working great now.

[kcerfly]

If you have successfully installed vllm, the xgrammar.so file can be compiled using the following command:

python -c "from xgrammar import GrammarMatcher;" --force-recompile

In case ninja is not installed, you can install it with:

pip install ninja

[ArtemBiliksin]

Hello everyone!

My team encountered the same problem. We were able to get around this problem by a strict sequence of steps:

1. Install vllm of the correct version. For example, pip install vllm==0.7.1
2. Install dependencies from the requirements-build.txt file: pip install -r requirements-build.txt.

This is the sequence of building vllm-base image from Dockerfile, from which vllm-openai-base and vllm-openai image is built, where guided_decoding with xgrammar works without errors.

With this solution, at the first request that requires xgrammar, the assembly of the xgrammar.so file will happen correctly and there will be no error. This will take some time, but it is only the first time you make a request that has this effect.

Surprisingly, if you use the vllm/vllm-openai:v0.7.1 image and make a request that requires xgrammar, no xgrammar.so file will be created inside the container and the response will be correct. Does anyone know why xgrammar.so is not created when using the official image, but if I manually install vllm==0.7.1 and requirements-build.txt then xgrammar.so is created? I think in newer versions of vllm the behaviour will be the same.

[nFunctor]

Getting an error with 0.8.1, that looks like

ERROR 03-25 16:11:55 [engine.py:160] FileNotFoundError: [Errno 2] No such file or directory: '/home/(...)/.cuenv/versions/3.12.5/envs/vllm/lib/python3.12/site-packages/xgrammar/kernels/apply_token_bitmask_inplace_cuda.cu'

I am not sure if this one is related or the same issue. Fresh 3.12.5 venv with vllm install.

[tjohnson31415]

I did some investigation into this issue, here's what I found:

• an optimized CUDA kernel was added to xgrammar as part of the v0.1.13 release

  • it is this CUDA kernel that xgrammar is trying to compile that can fail if missing Ninja and a full CUDA install; on the second use, the error about not finding xgrammar.so is raised

• the recent xgrammar v0.1.16 release includes a change that falls back to the previous Triton kernels if the tools needed for torch extension compilation are missing

• vLLM v0.7.2 had xgrammar >= 0.1.6 in its requirements, so it would pull in the latest xgrammar potentially hitting this issue

• vLLM v0.7.3 pinned xgrammar == 0.1.11 in this PR and vllm >=0.8.0 now pins xgrammar == 0.1.16

So using v0.7.3 or later should resolve this without needing to install Ninja or the other tools needed to do the runtime compilation.

---

I am not sure if this one is related or the same issue. Fresh 3.12.5 venv with vllm install.

I think this FileNotFoundError error is a separate issue. It seems that xgrammar is attempting to do the kernel compilation but is failing to find the source file for the kernel. I tried to reproduce, but it compiled for me in vllm/vllm-openai:latest.

[nFunctor]

@tjohnson31415 my issue was actually reported over at xgrammar and is due to a python replace .py with .cu that transforms .pyenv to .cuenv. I added a PR but an easy fix when installing from source is also possible so yes that’s a different issue

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!