Skip to content

Experiment with continue-on-error for flaky tests in GitHub actions #10861

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+3 −0
Open

Experiment with continue-on-error for flaky tests in GitHub actions #10861

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,13 +40,15 @@ jobs:
defaults:
run:
shell: bash -l {0}
continue-on-error: ${{ matrix.allowed-failure }}
Copy link
Contributor

@ianhi ianhi Oct 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this might have some unintended consequences.

Just for this PR I'd suggest introducing a failure in the non-flaky pytest to make sure the job still fails. My memory of how this works is that if job will not turn red.

I see now. But beware that i think this means the flaky test will always be green now

strategy:
fail-fast: false
matrix:
os: ["ubuntu-latest", "macos-latest", "windows-latest"]
# Bookend python versions
python-version: ["3.11", "3.13"]
env: [""]
allowed-failure: [false]
include:
# Minimum python version:
- env: "bare-minimum"
Expand All @@ -68,6 +70,7 @@ jobs:
- env: "flaky"
python-version: "3.13"
os: ubuntu-latest
allowed-failure: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a real command. I don't see it in here: https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh wait I was confused by this being below. this impacts the continue-on-error above?

# The mypy tests must be executed using only 1 process in order to guarantee
# predictable mypy output messages for comparison to expectations.
- env: "mypy"
Expand Down
Loading