tests: add timeout_manager to tensorrt flow test cases

[crazydemo]

Description

Due to intensive test workloads, some machines experience hang issues during test execution, affecting overall test stability and completion.

This PR adds timeout management system to handle TensorRT flow test cases with proper resource cleanup.

1. Added TimeoutManager Class

• Manages TensorRT flow test cases with proper timeout handling
• Ensures resource cleanup for test operations

2. Added timeout_manager Fixture

• Easy integration with test cases via fixture parameter
• Provides granular timeout control for different test phases

3. Improved trt_test_alternative.py

• Enhanced call function to use timeout from pytest markers or command line
• Priority: pytest marker timeout > pytest command line timeout

Test Coverage

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

run [--disable-fail-fast --skip-test --stage-list "A10-1, xxx" --gpu-type "A30, H100_PCIe" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-[Post-Merge]-1, xxx"]

Launch build/test pipelines. All previously running jobs will be killed.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests. Will also run L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-[Post-Merge]-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-[Post-Merge]-1, xxx".

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

Summary by CodeRabbit

• New Features

  • Introduced robust timeout management for integration tests, allowing major test phases to be time-limited and tracked.
  • Added utilities for structured timeout handling, including a timeout manager class and related fixtures.

• Bug Fixes

  • Improved subprocess call handling by supporting timeout parameters and forwarding additional arguments where needed.

• Chores

  • Updated test signatures and configurations to support and annotate timeout behavior.

[crazydemo]

/bot run

[tensorrt-cicd]

PR_Github #11750 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #11750 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #8702 completed with status: 'FAILURE'

[crazydemo]

/bot run --stage-list "B200_PCIe-PyTorch-1"

[tensorrt-cicd]

PR_Github #11766 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #11766 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #8715 (Partly Tested) completed with status: 'SUCCESS'

[crazydemo]

/bot run

[tensorrt-cicd]

PR_Github #12053 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #12053 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #8951 completed with status: 'SUCCESS'

[crazydemo]

@EmmaQiaoCh Could you please help review this PR?

[venkywonka]

@coderabbitai full review

[coderabbitai]

✅ Actions performed

Full review triggered.

[coderabbitai]

Caution

Review failed

The pull request is closed.

Walkthrough

Timeout management was integrated across multiple integration test modules. A new TimeoutManager utility was introduced, and test functions were refactored to accept and use this manager for enforcing and tracking timeouts during major test phases. Pytest fixtures and test list annotations were updated to support and configure per-test timeouts.

Changes

File(s)	Change Summary
tests/integration/defs/accuracy/accuracy_core.py	CliFlowAccuracyTestHarness.run updated to support an optional timeout_manager for phase timeouts.
tests/integration/defs/accuracy/test_cli_flow.py	Test methods updated to accept and use a timeout_manager parameter.
tests/integration/defs/common.py	MPI/subprocess wrappers and weight conversion functions updated to accept and propagate timeouts via kwargs.
tests/integration/defs/conftest.py	Added pytest fixtures to extract timeout values and provide a timeout_manager instance.
tests/integration/defs/examples/test_commandr.py	Test function updated to wrap major steps with timeout_manager.timed_operation and enforce timeouts.
tests/integration/defs/examples/test_exaone.py	Test functions updated for timeout-managed execution of conversion, build, and summary steps.
tests/integration/defs/examples/test_gpt.py	Subprocess calls now wrapped with timeout management; build logic slightly adjusted.
tests/integration/defs/examples/test_llama.py	Test functions refactored to wrap steps with timeout management and propagate timeouts.
tests/integration/defs/trt_test_alternative.py	Timeout extraction logic simplified and decoupled from pytest; explicit timeout handling in wrappers.
tests/integration/defs/utils/init.py, tests/integration/defs/utils/timeout_manager.py	New utility package and module for reusable timeout management classes, functions, and decorators.
tests/integration/test_lists/qa/examples_test_list.txt	Test list updated to mark additional tests as timing out, with specific timeout durations set.

Sequence Diagram(s)

sequenceDiagram
    participant Pytest
    participant Fixture
    participant TestFunction
    participant TimeoutManager
    participant Subprocess

    Pytest->>Fixture: Request test with timeout
    Fixture->>TimeoutManager: Instantiate with timeout value
    Fixture->>TestFunction: Inject timeout_manager
    loop For each major test phase
        TestFunction->>TimeoutManager: Enter timed_operation("phase")
        TimeoutManager->>Subprocess: Call with remaining timeout
        Subprocess-->>TimeoutManager: Return/raise on completion or timeout
        TimeoutManager-->>TestFunction: Exit context, update remaining timeout
    end

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~18 minutes

Suggested reviewers

• crazydemo

Poem

In the warren of tests, time's a tricky hare,
Now TimeoutManager keeps things fair!
Each phase is tracked, no step too slow,
The stopwatch ticks as subprocesses go.
With fixtures and wrappers, the code’s more robust—
This bunny approves, in timeouts we trust! 🕰️🐇

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ec3ebae and f13c0ab.

📒 Files selected for processing (12)

• tests/integration/defs/accuracy/accuracy_core.py (1 hunks)
• tests/integration/defs/accuracy/test_cli_flow.py (3 hunks)
• tests/integration/defs/common.py (7 hunks)
• tests/integration/defs/conftest.py (1 hunks)
• tests/integration/defs/examples/test_commandr.py (3 hunks)
• tests/integration/defs/examples/test_exaone.py (4 hunks)
• tests/integration/defs/examples/test_gpt.py (1 hunks)
• tests/integration/defs/examples/test_llama.py (4 hunks)
• tests/integration/defs/trt_test_alternative.py (4 hunks)
• tests/integration/defs/utils/__init__.py (1 hunks)
• tests/integration/defs/utils/timeout_manager.py (1 hunks)
• tests/integration/test_lists/qa/examples_test_list.txt (1 hunks)

✨ Finishing Touches

• 📝 Generate Docstrings

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.