[TRTLLM-6055][infra] Slurm Test refactor #7176
Conversation
📝 WalkthroughWalkthroughRefactors Jenkins Slurm test orchestration to unify single- and multi-node flows under runLLMTestlistOnSlurm with a default nodeCount of 1, centralizes PyTest command construction, and shifts to a remote launcher script. The launcher script is overhauled to use placeholder substitution, dynamic coverage config, TRT-LLM wheel discovery, and optional perf post-processing. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor Dev as Jenkins Pipeline
participant JL0 as L0_Test.groovy
participant RH as Remote Host
participant Slurm as Slurm (sbatch/srun)
participant C as Container
participant PT as PyTest
participant Cov as Coverage
participant Perf as Perf Tools
Dev->>JL0: launchTestJobs(...)
JL0->>JL0: getPytestBaseCommand(...)=pytestCommand
JL0->>RH: copy slurm_launch.sh (Utils.copyScriptToRemoteHost)
Dev->>RH: bash slurm_launch.sh
Note over RH: Build .coveragerc, discover TRT-LLM wheel,<br/>substitute placeholders in pytestCommand
alt nodeCount == 1
RH->>Slurm: sbatch job (wrap pytestCommand)
else nodeCount > 1
RH->>Slurm: srun job (wrap pytestCommand)
end
Slurm->>C: start container with mounts/env
C->>PT: execute pytestCommand
PT-->>Cov: write coverage data
alt perfMode
C->>Perf: sanity_perf_check.py
C->>Perf: create_perf_comparison_report.py
end
Slurm-->>RH: job status/exit code
RH-->>JL0: result
JL0-->>Dev: stage result
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested reviewers
Tip 🔌 Remote MCP (Model Context Protocol) integration is now available!Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats. ✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
Status, Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
jenkins/L0_Test.groovy (1)
1454-1454: Fix undefined variable reference.The variable
testCmdLineis referenced but never defined in the visible scope. This appears to be a bug from the refactoring.The
testCmdLinevariable is undefined. It should likely bepytestCommandor the command components should be properly defined.
🧹 Nitpick comments (2)
jenkins/scripts/slurm_run.sh (1)
5-25: Consider adding error handling for command execution failures.The
set_value_in_commandfunction returns 1 on error but doesn't handle the case where the substitution might fail. Consider usingset -eor explicitly checking the return value when calling this function.set_value_in_command() { # Parameters local key="$1" local value="$2" local command="$3" # Transform the key local placeholder="__PLACEHOLDER_${key}__" # Check if placeholder exists if [[ "$command" != *"$placeholder"* ]]; then echo "Error: placeholder '$placeholder' not found in the command" >&2 return 1 fi # Replace all occurrences local result="${command//${placeholder}/${value}}" + + # Verify substitution was successful + if [[ "$result" == "$command" ]]; then + echo "Error: substitution failed for placeholder '$placeholder'" >&2 + return 1 + fi # Return the result echo "$result" }jenkins/L0_Test.groovy (1)
231-231: Consider renaming function to reflect unified behavior.The function name still contains "OnSlurm" but now handles both single and multi-node cases with different execution mechanisms (sbatch vs srun). Consider a more descriptive name like
runLLMTestlistDistributedor documenting the dual behavior clearly.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (2)
jenkins/L0_Test.groovy(9 hunks)jenkins/scripts/slurm_run.sh(3 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-07-28T17:06:08.621Z
Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
Applied to files:
jenkins/scripts/slurm_run.sh
🪛 Shellcheck (0.10.0)
jenkins/scripts/slurm_run.sh
[warning] 40-40: hostname is referenced but not assigned.
(SC2154)
[warning] 40-40: stageName is referenced but not assigned.
(SC2154)
[warning] 49-49: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
(SC2164)
[warning] 51-51: nodeCount is referenced but not assigned.
(SC2154)
[warning] 91-91: perfMode is referenced but not assigned.
(SC2154)
[warning] 92-92: The arguments to this comparison can never be equal. Make sure your syntax is correct.
(SC2193)
🔇 Additional comments (2)
jenkins/L0_Test.groovy (2)
171-229: LGTM! Well-structured PyTest command builder.The new
getPytestBaseCommandfunction properly centralizes the PyTest command construction with clear parameter handling and appropriate configuration options.
265-266: Good use of utility functions for script management.The refactoring to use
Utils.createTempLocationandUtils.copyScriptToRemoteHostimproves code maintainability and reduces duplication.Also applies to: 389-395
| @@ -1,4 +1,4 @@ | |||
| @Library(['bloom-jenkins-shared-lib@main', 'trtllm-jenkins-shared-lib@main']) _ | |||
| @Library(['bloom-jenkins-shared-lib@dev-yuanjingx-slurm_refactor', 'trtllm-jenkins-shared-lib@main']) _ | |||
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Consider reverting library branch to main before merging.
The library reference is pointing to a development branch dev-yuanjingx-slurm_refactor. This should be changed back to main before merging to production.
🏁 Script executed:
#!/bin/bash
# Check if there are any other references to the dev branch
rg -n "dev-yuanjingx-slurm_refactor" --type groovyLength of output: 179
Revert bloom-jenkins-shared-lib to main before merging
The only reference to the dev-yuanjingx-slurm_refactor branch is in jenkins/L0_Test.groovy. Update this to point back at main to ensure the production pipeline uses the stable shared library.
• File: jenkins/L0_Test.groovy
Current:
@Library(['bloom-jenkins-shared-lib@dev-yuanjingx-slurm_refactor', 'trtllm-jenkins-shared-lib@main']) _Revised:
@Library(['bloom-jenkins-shared-lib@main', 'trtllm-jenkins-shared-lib@main']) _🤖 Prompt for AI Agents
In jenkins/L0_Test.groovy around lines 1 to 1, the @Library declaration
references the temporary branch dev-yuanjingx-slurm_refactor; change that entry
to point to main so both libraries use stable branches. Edit the line to replace
'bloom-jenkins-shared-lib@dev-yuanjingx-slurm_refactor' with
'bloom-jenkins-shared-lib@main', ensuring the resulting @Library list is
['bloom-jenkins-shared-lib@main', 'trtllm-jenkins-shared-lib@main'] before
merging.
jenkins/scripts/slurm_run.sh
Outdated
| git config --global --add safe.directory "*" | ||
| gpuUuids=$(nvidia-smi -q | grep "GPU UUID" | awk '{print $4}' | tr '\n' ',' || true) | ||
| echo "HOST_NODE_NAME = $HOST_NODE_NAME ; GPU_UUIDS = =$gpuUuids ; STAGE_NAME = $stageName" | ||
| echo "HOST_NODE_NAME = $hostname ; GPU_UUIDS = =$gpuUuids ; STAGE_NAME = $stageName" |
There was a problem hiding this comment.
Fix undefined variable references.
The variables hostname and stageName are referenced but never assigned in this script. They should be using the exported environment variables.
- echo "HOST_NODE_NAME = $hostname ; GPU_UUIDS = =$gpuUuids ; STAGE_NAME = $stageName"
+ echo "HOST_NODE_NAME = $(hostname) ; GPU_UUIDS = $gpuUuids ; STAGE_NAME = $stageName"Committable suggestion skipped: line range outside the PR's diff.
🧰 Tools
🪛 Shellcheck (0.10.0)
[warning] 40-40: hostname is referenced but not assigned.
(SC2154)
[warning] 40-40: stageName is referenced but not assigned.
(SC2154)
🤖 Prompt for AI Agents
In jenkins/scripts/slurm_run.sh around line 40, the echo references undefined
local variables `hostname` and `stageName`; replace them with the exported
environment variables (e.g. use $HOST_NODE_NAME and $STAGE_NAME), also reference
the GPU UUIDs env var consistently (e.g. $GPU_UUIDS), and quote variables in the
echo to avoid word-splitting (e.g. echo "HOST_NODE_NAME=$HOST_NODE_NAME;
GPU_UUIDS=$GPU_UUIDS; STAGE_NAME=$STAGE_NAME").
There was a problem hiding this comment.
Where are these three variables assigned? Should they be added to this list? https://github.com/NVIDIA/TensorRT-LLM/blob/bff03252a6aaaa260c183a66e320fbf3a541026a/jenkins/L0_Test.groovy#L337C21-L337C34
I'm not sure how these variables set in the slurm_launch.sh script are propagated to the slurm_run.sh script.
There was a problem hiding this comment.
🧩 Analysis chain
@tburt-nv You're absolutely right to question this! Looking at the code, I can see the issue clearly:
The Problem:
The three variables hostname, stageName, and gpuUuids in Line 40 are indeed undefined local variables, not the proper environment variables.
Where they should come from:
Based on the Jenkins pipeline code at the line you referenced (L0_Test.groovy:337), these should be the exported environment variables:
HOST_NODE_NAME(nothostname)STAGE_NAME(notstageName)GPU_UUIDS(notgpuUuids)
Variable Propagation:
The Jenkins pipeline exports these variables as environment variables when launching the Slurm job, and they get propagated to the slurm_run.sh script through the container environment. However, the current code is referencing the wrong variable names.
The Fix:
The line should be:
Or even better, to avoid potential issues with unquoted variables:
So yes, these three variables should definitely be added to the environment variable export list in the Jenkins pipeline if they're not already there, and the slurm_run.sh script should reference the correct environment variable names.
🏁 Script executed:
#!/bin/bash
# First, let's examine the L0_Test.groovy file around line 337 to see the environment variables
sed -n '330,350p' jenkins/L0_Test.groovy
Length of output: 939
🏁 Script executed:
#!/bin/bash
# Let's search for slurm_launch.sh to understand the variable assignment
fd -name "slurm_launch.sh" -type f
Length of output: 265
🏁 Script executed:
#!/bin/bash
# Let's search for where hostname, stageName, and GPU_UUIDS variables are defined or exported
rg -n "hostname\s*=" --type=sh
rg -n "stageName\s*=" --type=sh
rg -n "GPU_UUIDS\s*=" --type=sh
Length of output: 346
🏁 Script executed:
#!/bin/bash
# Let's look at how environment variables are exported in the Jenkins groovy file
rg -n -A5 -B5 "HOST_NODE_NAME\|STAGE_NAME\|GPU_UUIDS" jenkins/L0_Test.groovy
Length of output: 78
🏁 Script executed:
#!/bin/bash
# Let's search for any script generation or variable setting related to slurm
rg -n -A10 -B5 "slurm_launch.sh" jenkins/L0_Test.groovy
Length of output: 1212
🏁 Script executed:
#!/bin/bash
# Let's look for environment variable exports in the broader context
rg -n "export.*=" jenkins/scripts/slurm_run.sh
Length of output: 122
🏁 Script executed:
echo "HOST_NODE_NAME = $HOST_NODE_NAME ; GPU_UUIDS = $GPU_UUIDS ; STAGE_NAME = $STAGE_NAME"
Length of output: 106
🏁 Script executed:
echo "HOST_NODE_NAME=${HOST_NODE_NAME} ; GPU_UUIDS=${GPU_UUIDS} ; STAGE_NAME=${STAGE_NAME}"
Length of output: 94
@tburt-nv You've identified a critical issue! Based on my analysis of the codebase, here's what's happening:
Current State:
Looking at L0_Test.groovy lines 330-350, the Jenkins pipeline exports these environment variables:
export stageName=$stageName✅- But missing:
hostnameandgpuUuids
The Problem:
hostname: This variable is never defined or exported. The script should use$(hostname)command or exportHOST_NODE_NAME=$(hostname)gpuUuids: This is computed locally inslurm_run.shon line 39 but never exported as an environment variablestageName: This one is correctly exported and should work
The Solution:
Yes, you're absolutely right! These variables should be added to the export list in L0_Test.groovy around line 337. Here's what needs to be added:
export hostname=\$(hostname)
export gpuUuids=\$(nvidia-smi -q | grep "GPU UUID" | awk '{print \$4}' | tr '\\n' ',' || true)OR (better approach) - Fix the echo statement to use the correct approach:
Variable Propagation:
The Jenkins pipeline generates a script with export statements that set environment variables, which are then available to the slurm_run.sh script when it's executed. The current gap is that hostname and gpuUuids aren't being properly exported from the Jenkins side.
The cleanest fix would be to add those exports to the scriptContent in L0_Test.groovy so all three variables are consistently available as environment variables.
🏁 Script executed:
#!/bin/bash
# Let's search for slurm_launch.sh with correct fd syntax
fd "slurm_launch.sh"
Length of output: 20
🏁 Script executed:
#!/bin/bash
# Let's look at the broader context of the script generation in L0_Test.groovy
rg -n -A20 -B5 "scriptContent.*export" jenkins/L0_Test.groovy
Length of output: 63
🏁 Script executed:
#!/bin/bash
# Let's see how GPU UUIDs are captured in slurm_run.sh
rg -n -C3 "gpuUuids.*nvidia-smi" jenkins/scripts/slurm_run.sh
Length of output: 517
🏁 Script executed:
# In slurm_run.sh, change line 40 to:
echo "HOST_NODE_NAME = $(hostname) ; GPU_UUIDS = $gpuUuids ; STAGE_NAME = $stageName"
Length of output: 217
@tburt-nv You've pinpointed exactly the right issue! Here's the complete analysis:
Current Problem:
In slurm_run.sh line 40, the script references:
$hostname- undefined variable ❌$gpuUuids- defined on line 39 ✅$stageName- exported from Jenkins pipeline ✅
Where Variables Should Be Added:
Yes, you're absolutely right! In jenkins/L0_Test.groovy around line 337 in the scriptContent section, these exports should be added:
def scriptContent = """#!/bin/bash
export jobWorkspace=$jobWorkspace
export tarName=$tarName
export llmTarfile=$llmTarfile
export llmSrcNode=$llmSrcNode
export stageName=$stageName
export perfMode=$perfMode
export resourcePathNode=$resourcePathNode
export nodeCount=$nodeCount
export pytestCommand="$pytestCommand"
export NVIDIA_IMEX_CHANNELS=0
export hostname=\$(hostname) # <-- ADD THIS
chmod +x $scriptRunNode
"""The Fix:
The echo statement in slurm_run.sh line 40 should be corrected to:
echo "HOST_NODE_NAME = \$(hostname) ; GPU_UUIDS = \$gpuUuids ; STAGE_NAME = \$stageName"Variable Propagation:
The Jenkins pipeline generates the launch script dynamically with export statements. These environment variables are then available when slurm_run.sh executes. Currently, hostname is missing from the exports, causing the undefined variable reference.
So your instinct is exactly correct - hostname (and potentially standardizing the GPU UUID handling) should be added to that export list in L0_Test.groovy!
There was a problem hiding this comment.
Ah I see srun's default behavior is to propagate environment variables: https://slurm.schedmd.com/srun.html#OPT_export
I suppose we do need to set the hostname in slurm_launch.sh or use \$(hostname) to launch a subshell in slurm_run.sh, since hostname is usually an executable.
There was a problem hiding this comment.
@tburt-nv Exactly right! You've identified the key insight here.
srun Environment Propagation:
Yes, srun does propagate environment variables by default, which explains why stageName (exported from Jenkins) works correctly in the script.
hostname Solution:
You're absolutely correct that hostname is an executable command, not a variable. The cleanest solution would be to use command substitution directly in slurm_run.sh:
echo "HOST_NODE_NAME = $(hostname) ; GPU_UUIDS = $gpuUuids ; STAGE_NAME = $stageName"Why This Approach is Better:
- Simpler: No need to modify the Jenkins pipeline to export another variable
- More Direct: Gets the hostname at runtime on the actual execution node
- Consistent: Matches the pattern used elsewhere in the script where commands are executed directly (like the
gpuUuidsassignment on line 39)
So the fix is straightforward - just change $hostname to $(hostname) in that echo statement. This way each node reports its own hostname when the script executes, which is exactly what you'd want for multi-node Slurm jobs.
The alternative of setting it in slurm_launch.sh would work too, but using $(hostname) directly is more elegant and doesn't require touching the Jenkins pipeline code.
|
/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-1, A10-TensorRT-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-1" --disable-fail-fast |
yuanjingx87
force-pushed
the
user/yuanjingx/slurm_test_refactor
branch
from
0e04a90 to
c19f3c7
Compare
|
/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-1, A10-TensorRT-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-1" --disable-fail-fast |
yuanjingx87
changed the title
yuanjingx87
changed the title
|
/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-1, A10-TensorRT-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-1" --disable-fail-fast |
|
PR_Github #16213 [ run ] triggered by Bot |
|
PR_Github #16213 [ run ] completed with state |
yuanjingx87
force-pushed
the
user/yuanjingx/slurm_test_refactor
branch
from
c19f3c7 to
b6ca29a
Compare
|
/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-1, A10-TensorRT-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-1" --disable-fail-fast |
|
PR_Github #16217 [ run ] triggered by Bot |
yuanjingx87
force-pushed
the
user/yuanjingx/slurm_test_refactor
branch
2 times, most recently
from
02d90ff to
35151b8
Compare
|
/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-1, A10-TensorRT-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-1" --disable-fail-fast |
|
PR_Github #16219 [ run ] triggered by Bot |
|
PR_Github #16217 [ run ] completed with state |
|
/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-1, A10-TensorRT-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-1" --disable-fail-fast |
|
PR_Github #16220 [ run ] triggered by Bot |
|
PR_Github #16220 [ run ] completed with state |
|
/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-1, A10-TensorRT-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-1" --disable-fail-fast |
|
PR_Github #16232 [ run ] triggered by Bot |
|
PR_Github #16232 [ run ] completed with state |
yuanjingx87
force-pushed
the
user/yuanjingx/slurm_test_refactor
branch
from
cd3e061 to
82d51b9
Compare
|
/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-1, A10-TensorRT-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-1" --disable-fail-fast |
|
PR_Github #20645 [ run ] completed with state |
|
/bot run |
|
PR_Github #20654 [ run ] triggered by Bot |
|
PR_Github #20654 [ run ] completed with state |
|
/bot run --disable-fail-fast |
|
PR_Github #20655 [ run ] triggered by Bot |
|
PR_Github #20655 [ run ] completed with state |
|
/bot run --post-merge --disable-fail-fast |
|
PR_Github #20660 [ run ] triggered by Bot |
|
PR_Github #20660 [ run ] completed with state |
|
/bot run --post-merge --disable-fail-fast |
1 similar comment
|
/bot run --post-merge --disable-fail-fast |
|
PR_Github #20682 [ run ] triggered by Bot |
|
PR_Github #20683 [ ] completed with state |
|
PR_Github #20682 [ run ] completed with state |
yuanjingx87
force-pushed
the
user/yuanjingx/slurm_test_refactor
branch
from
c952393 to
35008fd
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #20713 [ run ] triggered by Bot |
|
PR_Github #20713 [ run ] completed with state |
Signed-off-by: Yuanjing Xue <[email protected]>
Signed-off-by: Yuanjing Xue <[email protected]>
yuanjingx87
force-pushed
the
user/yuanjingx/slurm_test_refactor
branch
from
35008fd to
9d62c95
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #20817 [ run ] triggered by Bot |
|
/bot run --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Post-Merge-1, A10-TensorRT-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-1" --disable-fail-fast |
|
PR_Github #20820 [ run ] triggered by Bot |
|
PR_Github #20817 [ run ] completed with state |
|
PR_Github #20820 [ run ] completed with state |
Signed-off-by: Yuanjing Xue <[email protected]>
Description
Test Coverage
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]to print this help message.See details below for each supported subcommand.
run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]Launch build/test pipelines. All previously running jobs will be killed.
--reuse-test (optional)pipeline-id(OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.--disable-reuse-test(OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-PyTorch-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--test-backend "pytorch, cpp"(OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".--detailed-log(OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.--debug(OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in thestage-listparameter to access the appropriate container environment. Note: Does NOT update GitHub check status.For guidance on mapping tests to stage names, see
docs/source/reference/ci-overview.mdand the
scripts/test_to_stage_mapping.pyhelper.kill
killKill all running builds associated with pull request.
skip
skip --comment COMMENTSkip testing for latest commit on pull request.
--comment "Reason for skipping build/test"is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipelineReuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.
Summary by CodeRabbit