[fix] Eagle-2 LLMAPI pybind argument fix. #3967

jhaotingc · 2025-04-29T20:17:21Z

Description

A bug is triggered when launching Eagle-2 with LLMAPI (trtllm-serve, trtllm-bench).
Eagle-2 expects 2 arguments, --use_dynamic_tree and --dynamic_tree_max_topK.
Eagle-1 expects --eagle_choices.
Without the fix, pybind expects --eagle_choices to be not None, while the check here asserts --eagle_choice to be none when --use_dynamic_tree=True (Eagle-2 is used).

This is introduced by TRTLLM-2867, MR 8312.

Test Coverage

examples/llm-api/llm_eagle2_decoding.py
tests/integration/defs/test_e2e.py
- test_llmapi_example_eagle2_decoding
tests/unittest/bindings/test_executor_bindings.py
- test_eagle_config_pickle
tests/unittest/llmapi/test_llm.py
- test_llm_api_eagle2

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

run [--disable-fail-fast --skip-test --stage-list "A10-1, xxx" --gpu-type "A30, H100_PCIe" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-[Post-Merge]-1, xxx"]

Launch build/test pipelines. All previously running jobs will be killed.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests. Will also run L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-[Post-Merge]-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-[Post-Merge]-1, xxx".

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

jhaotingc · 2025-04-29T20:18:53Z

/bot run --disable-fail-fast

jhaotingc · 2025-04-29T20:29:13Z

/bot run --disable-fail-fast

katec846 · 2025-04-29T20:29:51Z

/bot run

examples/llm-api/llm_eagle2_decoding.py

tests/unittest/llmapi/test_llm.py

tests/integration/defs/test_e2e.py

tensorrt-cicd · 2025-04-29T20:35:47Z

PR_Github #3756 [ run ] triggered by Bot

katec846 · 2025-04-29T20:56:23Z

/bot kill

tensorrt-cicd · 2025-04-29T21:02:19Z

PR_Github #3761 [ kill ] triggered by Bot

tensorrt-cicd · 2025-04-29T21:03:27Z

PR_Github #3756 [ run ] completed with state ABORTED

tensorrt-cicd · 2025-04-29T21:03:57Z

PR_Github #3761 [ kill ] completed with state SUCCESS
Successfully killed previous jobs for commit a3ec04e

katec846 · 2025-04-29T21:06:26Z

/bot run --disable-fail-fast

tensorrt-cicd · 2025-04-29T21:12:23Z

PR_Github #3763 [ run ] triggered by Bot

katec846 · 2025-04-29T21:15:06Z

/bot kill

tensorrt-cicd · 2025-04-29T21:21:22Z

PR_Github #3764 [ kill ] triggered by Bot

tensorrt-cicd · 2025-04-29T21:22:27Z

PR_Github #3763 [ run ] completed with state ABORTED

tensorrt-cicd · 2025-04-29T21:22:57Z

PR_Github #3764 [ kill ] completed with state SUCCESS
Successfully killed previous jobs for commit 3219e2e

katec846 · 2025-04-29T21:23:32Z

/bot run --disable-fail-fast

tensorrt-cicd · 2025-04-29T21:29:40Z

PR_Github #3765 [ run ] triggered by Bot

yweng0828 · 2025-04-30T03:05:11Z

LGTM, Thanks for the fix!

tensorrt-cicd · 2025-04-30T06:42:08Z

PR_Github #3765 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #2666 completed with status: 'FAILURE'

jhaotingc · 2025-04-30T07:06:31Z

/bot run --disable-fail-fast

examples/llm-api/llm_eagle_decoding.py

tests/unittest/llmapi/test_llm.py

jhaotingc · 2025-05-20T00:06:57Z

/bot run --disable-fail-fast

tensorrt-cicd · 2025-05-20T00:19:36Z

PR_Github #5787 [ run ] triggered by Bot

tensorrt-cicd · 2025-05-20T08:37:35Z

PR_Github #5787 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4237 completed with status: 'FAILURE'

jhaotingc · 2025-05-20T16:59:18Z

/bot run --disable-fail-fast

tensorrt-cicd · 2025-05-20T17:05:10Z

PR_Github #5899 [ run ] triggered by Bot

tensorrt-cicd · 2025-05-21T06:18:26Z

PR_Github #5899 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4323 completed with status: 'FAILURE'

jhaotingc · 2025-05-21T23:54:23Z

blocked by 15317ec

jhaotingc · 2025-05-21T23:55:00Z

/bot run --disable-fail-fast

tensorrt-cicd · 2025-05-22T00:00:05Z

PR_Github #6068 [ run ] triggered by Bot

tensorrt-cicd · 2025-05-22T09:09:41Z

PR_Github #6068 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4439 completed with status: 'FAILURE'

jhaotingc · 2025-05-22T16:37:08Z

blocked by random test.

jhaotingc · 2025-05-22T16:37:14Z

/bot run --disable-fail-fast

tensorrt-cicd · 2025-05-22T16:42:31Z

PR_Github #6169 [ run ] triggered by Bot

tensorrt-cicd · 2025-05-22T19:59:50Z

PR_Github #6169 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4510 completed with status: 'FAILURE'

jhaotingc · 2025-05-22T21:09:44Z

/bot run --disable-fail-fast

tensorrt-cicd · 2025-05-22T21:14:44Z

PR_Github #6186 [ run ] triggered by Bot

tensorrt-cicd · 2025-05-23T02:15:20Z

PR_Github #6186 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4524 completed with status: 'SUCCESS'

syuoni

The PR looks great to me! Thanks

tests/unittest/llmapi/test_llm.py

Signed-off-by: Jhao-Ting Chen <[email protected]>

symphonylyh · 2025-05-29T18:48:10Z

/bot reuse-pipeline

tensorrt-cicd · 2025-05-29T18:54:33Z

PR_Github #6948 [ reuse-pipeline ] triggered by Bot

tensorrt-cicd · 2025-05-29T19:07:57Z

PR_Github #6948 [ reuse-pipeline ] completed with state SUCCESS
Reusing PR_Github #6186 for commit e19475b

Signed-off-by: Jhao-Ting Chen <[email protected]> Co-authored-by: Haohang Huang <[email protected]> Signed-off-by: darraghdog <[email protected]>

jhaotingc requested a review from a team as a code owner April 29, 2025 20:17

katec846 requested a review from symphonylyh April 29, 2025 20:23

symphonylyh approved these changes Apr 29, 2025

View reviewed changes

examples/llm-api/llm_eagle2_decoding.py Outdated Show resolved Hide resolved

tests/unittest/llmapi/test_llm.py Outdated Show resolved Hide resolved

tests/integration/defs/test_e2e.py Outdated Show resolved Hide resolved

katec846 requested review from nv-guomingz and yweng0828 April 29, 2025 20:33

jhaotingc force-pushed the eagle_2_llmapi_fix branch from a3ec04e to 3219e2e Compare April 29, 2025 21:12

yweng0828 requested a review from syuoni April 30, 2025 02:51

yweng0828 approved these changes Apr 30, 2025

View reviewed changes

symphonylyh enabled auto-merge (squash) April 30, 2025 04:21

symphonylyh disabled auto-merge April 30, 2025 04:22

syuoni reviewed Apr 30, 2025

View reviewed changes

examples/llm-api/llm_eagle_decoding.py Outdated Show resolved Hide resolved

tests/unittest/llmapi/test_llm.py Outdated Show resolved Hide resolved

tests/unittest/llmapi/test_llm.py Outdated Show resolved Hide resolved

jhaotingc force-pushed the eagle_2_llmapi_fix branch from 62c9db7 to 0e66ff7 Compare May 21, 2025 23:54

jhaotingc force-pushed the eagle_2_llmapi_fix branch from 0e66ff7 to f57a1c9 Compare May 22, 2025 16:30

syuoni approved these changes May 24, 2025

View reviewed changes

tests/unittest/llmapi/test_llm.py Outdated Show resolved Hide resolved

jhaotingc force-pushed the eagle_2_llmapi_fix branch from f57a1c9 to 2648f97 Compare May 28, 2025 22:08

fix eagle2 llmapi error, add tests, fix eagle tests

a8695ff

Signed-off-by: Jhao-Ting Chen <[email protected]>

jhaotingc force-pushed the eagle_2_llmapi_fix branch from 2648f97 to a8695ff Compare May 29, 2025 05:48

Merge branch 'main' into eagle_2_llmapi_fix

e19475b

symphonylyh enabled auto-merge (squash) May 29, 2025 18:48

symphonylyh merged commit fcadce9 into NVIDIA:main May 29, 2025
3 checks passed

[fix] Eagle-2 LLMAPI pybind argument fix. #3967

[fix] Eagle-2 LLMAPI pybind argument fix. #3967

Uh oh!

Conversation

jhaotingc commented Apr 29, 2025

Description

Test Coverage

GitHub Bot Help

kill

skip

reuse-pipeline

Uh oh!

jhaotingc commented Apr 29, 2025

Uh oh!

jhaotingc commented Apr 29, 2025

Uh oh!

katec846 commented Apr 29, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tensorrt-cicd commented Apr 29, 2025

Uh oh!

katec846 commented Apr 29, 2025

Uh oh!

tensorrt-cicd commented Apr 29, 2025

Uh oh!

tensorrt-cicd commented Apr 29, 2025

Uh oh!

tensorrt-cicd commented Apr 29, 2025

Uh oh!

katec846 commented Apr 29, 2025

Uh oh!

tensorrt-cicd commented Apr 29, 2025

Uh oh!

katec846 commented Apr 29, 2025

Uh oh!

tensorrt-cicd commented Apr 29, 2025

Uh oh!

tensorrt-cicd commented Apr 29, 2025

Uh oh!

tensorrt-cicd commented Apr 29, 2025

Uh oh!

katec846 commented Apr 29, 2025

Uh oh!

tensorrt-cicd commented Apr 29, 2025

Uh oh!

yweng0828 commented Apr 30, 2025

Uh oh!

tensorrt-cicd commented Apr 30, 2025

Uh oh!

jhaotingc commented Apr 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jhaotingc commented May 20, 2025

Uh oh!

tensorrt-cicd commented May 20, 2025

Uh oh!

tensorrt-cicd commented May 20, 2025

Uh oh!

jhaotingc commented May 20, 2025

Uh oh!

tensorrt-cicd commented May 20, 2025

Uh oh!

tensorrt-cicd commented May 21, 2025

Uh oh!

jhaotingc commented May 21, 2025

Uh oh!

jhaotingc commented May 21, 2025

Uh oh!

tensorrt-cicd commented May 22, 2025

Uh oh!

tensorrt-cicd commented May 22, 2025

Uh oh!

jhaotingc commented May 22, 2025

Uh oh!

jhaotingc commented May 22, 2025

Uh oh!