fix: random fail of cache router test #4597
Conversation
|
/bot run --stage-list "H100_PCIe-PyTorch-1" |
|
PR_Github #6201 [ run ] triggered by Bot |
|
PR_Github #6201 [ run ] completed with state |
|
/bot run --stage-list "H100_PCIe-PyTorch-1,H100_PCIe-PyTorch-2,H100_PCIe-PyTorch-3" --disable-fail-fast |
|
PR_Github #6227 [ run ] triggered by Bot |
|
PR_Github #6227 [ run ] completed with state |
zhengd-nv
force-pushed
the
fix-cache-router-test
branch
from
2518a40 to
b5a0bf6
Compare
|
/bot run --stage-list "H100_PCIe-PyTorch-1,H100_PCIe-PyTorch-2,H100_PCIe-PyTorch-3" --disable-fail-fast |
|
PR_Github #6251 [ run ] triggered by Bot |
|
PR_Github #6251 [ run ] completed with state |
|
/bot run --stage-list "H100_PCIe-PyTorch-1,H100_PCIe-PyTorch-2,H100_PCIe-PyTorch-3" --disable-fail-fast |
|
PR_Github #6274 [ run ] triggered by Bot |
|
PR_Github #6274 [ run ] completed with state |
|
/bot run --stage-list "H100_PCIe-PyTorch-1,H100_PCIe-PyTorch-2,H100_PCIe-PyTorch-3" --disable-fail-fast |
|
PR_Github #6404 [ run ] triggered by Bot |
|
PR_Github #6404 [ run ] completed with state |
|
/bot run --stage-list "H100_PCIe-PyTorch-1,H100_PCIe-PyTorch-2,H100_PCIe-PyTorch-3,B200_PCIe-PyTorch-1,B200_PCIe-PyTorch-2" --disable-fail-fast |
|
PR_Github #6427 [ run ] triggered by Bot |
|
PR_Github #6427 [ run ] completed with state |
|
/bot run --stage-list "H100_PCIe-PyTorch-1,H100_PCIe-PyTorch-2,H100_PCIe-PyTorch-3,B200_PCIe-PyTorch-2,L40S-PyTorch-1" --disable-fail-fast |
1 similar comment
|
/bot run --stage-list "H100_PCIe-PyTorch-1,H100_PCIe-PyTorch-2,H100_PCIe-PyTorch-3,B200_PCIe-PyTorch-2,L40S-PyTorch-1" --disable-fail-fast |
|
PR_Github #6469 [ run ] triggered by Bot |
|
PR_Github #6469 [ run ] completed with state |
|
/bot run --stage-list "H100_PCIe-PyTorch-1,H100_PCIe-PyTorch-2,H100_PCIe-PyTorch-3,B200_PCIe-PyTorch-2,L40S-PyTorch-1" --disable-fail-fast |
|
PR_Github #6527 [ run ] triggered by Bot |
|
PR_Github #6527 [ run ] completed with state |
|
/bot run --stage-list "H100_PCIe-PyTorch-1,H100_PCIe-PyTorch-2,H100_PCIe-PyTorch-3,B200_PCIe-PyTorch-2,L40S-PyTorch-1" --disable-fail-fast |
|
PR_Github #6562 [ run ] triggered by Bot |
|
PR_Github #6562 [ run ] completed with state |
|
/bot run |
|
PR_Github #6756 [ run ] completed with state |
|
/bot run |
|
PR_Github #6784 [ run ] triggered by Bot |
|
PR_Github #6784 [ run ] completed with state |
|
/bot run |
1 similar comment
|
/bot run |
|
PR_Github #6827 [ run ] triggered by Bot |
|
PR_Github #6830 [ run ] triggered by Bot |
|
PR_Github #6830 [ run ] completed with state |
|
/bot run |
|
PR_Github #6885 [ run ] triggered by Bot |
|
PR_Github #6885 [ run ] completed with state |
zhengd-nv
force-pushed
the
fix-cache-router-test
branch
from
098bf8a to
fd77893
Compare
|
/bot run --add-multi-gpu-test |
|
PR_Github #6983 [ run ] triggered by Bot |
|
PR_Github #6983 [ run ] completed with state |
Signed-off-by: Zheng Duan <[email protected]> update test eviction Signed-off-by: Zheng Duan <[email protected]>
Signed-off-by: Zheng Duan <[email protected]>
Signed-off-by: Zheng Duan <[email protected]>
Shixiaowei02
force-pushed
the
fix-cache-router-test
branch
from
0d23c87 to
368fbe4
Compare
|
/bot reuse-pipeline |
|
PR_Github #7056 [ reuse-pipeline ] triggered by Bot |
|
PR_Github #7056 [ reuse-pipeline ] completed with state |
Signed-off-by: Zheng Duan <[email protected]> Signed-off-by: darraghdog <[email protected]>
PR title
Description
This PR tries to resolve several flaky fails of cache aware router tests, or at least root cause them with better error outputs.
Flaky fails in L0:
From QA:
Previously if the workers failed to start, we can only get a timeout error from the proxy server. After this PR, the test script will dump the worker logs into TRTLLM logger in error level to help us know what happens.
Test Coverage
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]to print this help message.See details below for each supported subcommand.
run [--disable-fail-fast --skip-test --stage-list "A10-1, xxx" --gpu-type "A30, H100_PCIe" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-[Post-Merge]-1, xxx"]Launch build/test pipelines. All previously running jobs will be killed.
--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests. Will also run L0 pre-merge pipeline.--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-[Post-Merge]-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-[Post-Merge]-1, xxx".kill
killKill all running builds associated with pull request.
skip
skip --comment COMMENTSkip testing for latest commit on pull request.
--comment "Reason for skipping build/test"is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipelineReuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.