Skip to content

chore [BREAKING CHANGE]: Flatten PyTorchConfig knobs into TorchLlmArgs #4603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
May 28, 2025
Merged

chore [BREAKING CHANGE]: Flatten PyTorchConfig knobs into TorchLlmArgs #4603

merged 6 commits into from
May 28, 2025

Conversation

Superjomn
Copy link
Collaborator

@Superjomn Superjomn commented May 23, 2025 •
edited
Loading

Description

  1. Eliminate the pytorch_backend_config and flatten all PyTorchConfig knobs into TorchLlmArgs
  2. Update all the usages accordingly

@Superjomn Superjomn force-pushed the flatten-pytorch-config branch 3 times, most recently from 678236c to cc4863c Compare May 23, 2025 06:31
@Superjomn Superjomn requested a review from a team as a code owner May 23, 2025 06:31
@Superjomn Superjomn requested review from yilin-void and Funatiq May 23, 2025 06:31
@Superjomn Superjomn force-pushed the flatten-pytorch-config branch 3 times, most recently from c7573da to 1ec59ec Compare May 23, 2025 08:37
@Superjomn Superjomn requested a review from QiJune May 23, 2025 08:40
@Superjomn Superjomn force-pushed the flatten-pytorch-config branch 2 times, most recently from 19f20a3 to c3d0f10 Compare May 23, 2025 09:51
@Superjomn
Copy link
Collaborator Author

/bot run --add-multi-gpu-test --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #6285 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #6285 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4593 completed with status: 'FAILURE'

@Superjomn Superjomn force-pushed the flatten-pytorch-config branch from c3d0f10 to 1115607 Compare May 25, 2025 08:18
@Superjomn Superjomn requested a review from a team as a code owner May 25, 2025 08:18
@Superjomn
Copy link
Collaborator Author

/bot run --add-multi-gpu-test --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #6379 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #6379 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #4661 completed with status: 'FAILURE'

@Superjomn
Copy link
Collaborator Author

/bot run --add-multi-gpu-test --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #6418 [ run ] triggered by Bot

@Superjomn Superjomn force-pushed the flatten-pytorch-config branch 2 times, most recently from 7755876 to a9f24cb Compare May 26, 2025 05:29
@Superjomn
Copy link
Collaborator Author

/bot run --add-multi-gpu-test --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #6425 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #6418 [ run ] completed with state ABORTED

Funatiq
Funatiq reviewed May 26, 2025
View reviewed changes
@Superjomn Superjomn force-pushed the flatten-pytorch-config branch from a9f24cb to 6393176 Compare May 27, 2025 01:57
@tensorrt-cicd
Copy link
Collaborator

PR_Github #6582 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #6582 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4814 completed with status: 'FAILURE'

@Superjomn Superjomn force-pushed the flatten-pytorch-config branch from 2624ad4 to 869e6c8 Compare May 27, 2025 11:21
@Superjomn Superjomn requested review from a team as code owners May 27, 2025 11:21
@Superjomn
Copy link
Collaborator Author

/bot run --disable-fail-fast

@Superjomn Superjomn enabled auto-merge (squash) May 27, 2025 11:22
@tensorrt-cicd
Copy link
Collaborator

PR_Github #6630 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #6630 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #4845 completed with status: 'SUCCESS'

Superjomn added 6 commits May 28, 2025 10:19
@Superjomn
flatten PyTorchConfig
21e64eb 
Signed-off-by: Superjomn <[email protected]>
@Superjomn
Signed-off-by: Superjomn <[email protected]>
1b79773 
update usages

Signed-off-by: Superjomn <[email protected]>
@Superjomn
remove PyTorchConfig.post_model_init
dad84ae 
Signed-off-by: Superjomn <[email protected]>
@Superjomn
fix cuda-graph settings
304aed7 
Signed-off-by: Superjomn <[email protected]>
@Superjomn
fix autodeploy
45ef5e6 
Signed-off-by: Superjomn <[email protected]>
@Superjomn
fix
bee40fa 
Signed-off-by: Superjomn <[email protected]>
@Superjomn Superjomn force-pushed the flatten-pytorch-config branch from 869e6c8 to bee40fa Compare May 28, 2025 10:22
@Superjomn
Copy link
Collaborator Author

/bot reuse-pipeline

@tensorrt-cicd
Copy link
Collaborator

PR_Github #6763 [ reuse-pipeline ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #6763 [ reuse-pipeline ] completed with state SUCCESS
Reusing PR_Github #6630 for commit bee40fa

@Superjomn Superjomn merged commit 5506f60 into NVIDIA:main May 28, 2025
3 checks passed
@Superjomn Superjomn deleted the flatten-pytorch-config branch May 28, 2025 13:42
chang-l
chang-l reviewed May 28, 2025
View reviewed changes
pamelap-nvidia
pamelap-nvidia reviewed May 30, 2025
View reviewed changes
docs/source/blogs/Best_perf_practice_on_DeepSeek-R1_in_TensorRT-LLM.md
YOUR_DATA_PATH=<your dataset file following the format>

cat >./extra-llm-api-config.yml<<EOF
pytorch_backend_config:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should throw a "deprecated" message when the old config type is detected. Currently old configs are still accepted, but all the fields under pytorch_backend_config are ignored, thus none of the configs are taking effect. This causes confusion to users (like myself).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a very fair request. @Superjomn

@chang-l chang-l mentioned this pull request May 30, 2025
Open
4 tasks
rmccorm4 added a commit to ai-dynamo/dynamo that referenced this pull request Jun 2, 2025
@rmccorm4
fix: Flatten pytorch_backend_config section to address breaking chang…
1cad61f 
…e to trtllm config in NVIDIA/TensorRT-LLM#4603
@rmccorm4 rmccorm4 mentioned this pull request Jun 2, 2025
Merged
darraghdog pushed a commit to darraghdog/TensorRT-LLM that referenced this pull request Jun 3, 2025
@Superjomn @darraghdog
chore [BREAKING CHANGE]: Flatten PyTorchConfig knobs into TorchLlmArgs (
0652953 
NVIDIA#4603)

Signed-off-by: Superjomn <[email protected]>
Signed-off-by: darraghdog <[email protected]>
@lucaslie lucaslie mentioned this pull request Jun 4, 2025
Merged
@rosenrodt rosenrodt mentioned this pull request Jul 3, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yuantailing yuantailing yuantailing left review comments

@chang-l chang-l chang-l left review comments

@jhaotingc jhaotingc jhaotingc left review comments

@juney-nvidia juney-nvidia juney-nvidia left review comments

@pamelap-nvidia pamelap-nvidia pamelap-nvidia left review comments

@yilin-void yilin-void yilin-void approved these changes

@Funatiq Funatiq Funatiq approved these changes

@QiJune QiJune QiJune approved these changes

@kaiyux kaiyux Awaiting requested review from kaiyux

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@Superjomn @tensorrt-cicd @yuantailing @chang-l @jhaotingc @yilin-void @Funatiq @QiJune @juney-nvidia @pamelap-nvidia