feat: Add LLGuidance Support for PyTorch Backend

[jellysnack]

PR title

Add LLGuidance Support for PyTorch Backend

Description

This PR introduces LLGuidance as a guided decoding backend for the PyTorch flow.
It complements the functionality proposed in #5011 (adds LLGuidance for the TensorRT flow), but avoids the Cargo dependency by relying solely on the LLGuidance Python package.

This makes the integration more lightweight and aligns better with the current project direction focused on the PyTorch flow.

[syuoni]

/bot run

[syuoni]

LGTM, thanks for the contribution! @jellysnack

[syuoni]

@QiJune Could you please also take a look at this PR?

I've locally verified the new guided decoding backend works well. Corresponding tests will be added in a new PR. Thanks!

[tensorrt-cicd]

PR_Github #8957 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #8957 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6536 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

[syuoni]

Hi @jellysnack ,

I tried to add tests for llguidance, but encountered an error. Seems llguidance cannot parse all JSON schema in the json_mode_eval dataset. Could you please run the below command and check the results?

cat > ./extra_llm_api_options.yaml <<EOF
guided_decoding_backend: llguidance
disable_overlap_scheduler: true
EOF

trtllm-eval --model <LLaMA-3.1-Instruct-Path> --extra_llm_api_options extra_llm_api_options.yaml json_mode_eval

The error log complains:

ValueError: LLGuidance matcher error: Unimplemented keys: ["else", "if", "then"]

[jellysnack]

Hi @jellysnack ,

I tried to add tests for llguidance, but encountered an error. Seems llguidance cannot parse all JSON schema in the json_mode_eval dataset. Could you please run the below command and check the results?

cat > ./extra_llm_api_options.yaml <<EOF
guided_decoding_backend: llguidance
disable_overlap_scheduler: true
EOF

trtllm-eval --model <LLaMA-3.1-Instruct-Path> --extra_llm_api_options extra_llm_api_options.yaml json_mode_eval

The error log complains:

ValueError: LLGuidance matcher error: Unimplemented keys: ["else", "if", "then"]

I think I know what's the issue. By default llguidance fails when it encounters unsupported keys in the schema. You can control this behavior using the top-level "x-guidance" key in the JSON schema, along with settings for various options such as whitespace handling (see the x-guidance documentation).

Setting "lenient": true instructs llguidance to ignore unsupported keywords and formats instead of failing.

So JSON schema should include the "x-guidance" key as follows:

{
   "x-guidance": {
      "lenient": true
   },
   "type": "object",
   "properties": {
      ...
   }
}

I haven’t tried running the test with this fix yet. I’ll add "x-guidance": { "lenient": true } by default to JSON schemas in case it resolves the problem. If it’s convenient for you, please feel free to test the fix on your end as well.

But it would be great to have these options including whitespace controls exposed as easily configurable parameters, rather than hardcoded. Additionally, it would be extremely useful to also allow configuration of XGrammar parameters such as cache size. Currently, the XGrammar cache is infinite by default, which can lead to memory overflow (I’ve encountered this myself and needed to patch TRT-LLM to avoid OOM)

[syuoni]

/bot run

[tensorrt-cicd]

PR_Github #9186 [ run ] triggered by Bot

[syuoni]

/bot run

[tensorrt-cicd]

PR_Github #9257 [ run ] triggered by Bot

[syuoni]

/bot kill

[tensorrt-cicd]

PR_Github #9267 [ kill ] triggered by Bot

[tensorrt-cicd]

PR_Github #9257 [ run ] completed with state ABORTED

[tensorrt-cicd]

PR_Github #9267 [ kill ] completed with state SUCCESS
Successfully killed previous jobs for commit b4f8c38

[syuoni]

/bot run

[tensorrt-cicd]

PR_Github #9268 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #9268 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6799 completed with status: 'SUCCESS'