Skip to content

src/doc/en/developer/portability_testing.rst: Update after migration #35108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
May 22, 2023
Merged

src/doc/en/developer/portability_testing.rst: Update after migration #35108

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f240c30
src/doc/en/developer/portability_testing.rst: Update after migration …
mkoeppe Feb 13, 2023
17596a7
.github/workflows/ci-linux.yml: Give access to GH packages
mkoeppe May 8, 2023
d45ec29
src/doc/en/developer/portability_testing.rst: No longer suggest to ch…
mkoeppe May 8, 2023
4c2d742
src/doc/en/developer/portability_testing.rst: Fix markup
mkoeppe May 8, 2023
c49fc13
src/doc/en/developer/portability_testing.rst: Fix markup
mkoeppe May 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci-linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@ env:
TARGETS: build doc-html
TARGETS_OPTIONAL: ptest

permissions:
packages: write

jobs:

standard-pre:
Expand Down
104 changes: 55 additions & 49 deletions src/doc/en/developer/portability_testing.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -949,49 +949,70 @@ Automatic testing on multiple platforms on GitHub Actions
=========================================================

The Sage source tree includes a default configuration for GitHub
Actions that runs our portability tests with tox on a multitude of
platforms on every pull request and on every push of a tag (but not of
a branch) to a repository for which GitHub Actions are enabled.
Actions that runs our portability tests on a multitude of platforms on
every push of a tag (but not of a branch) to a repository for which
GitHub Actions are enabled.

In particular, it automatically runs on our main repository on every
release tag.
In particular, it automatically runs on our main repository sagemath/sage
on every release tag.

This is defined in the files `$SAGE_ROOT/.github/workflows/tox*.yml
<https://github.com/sagemath/sage/tree/develop/.github/workflows/tox.yml>`_.
This is defined in the files

An additional GitHub Actions workflow for testing on Cygwin, not based
on tox, is defined in the files
`$SAGE_ROOT/.github/workflows/ci-cygwin*.yml
<https://github.com/sagemath/sage/tree/develop/.github/workflows/ci-cygwin-standard.yml>`_.
- `$SAGE_ROOT/.github/workflows/ci-linux.yml
<https://github.com/sagemath/sage/tree/develop/.github/workflows/ci-linux.yml>`_
(which calls `$SAGE_ROOT/.github/workflows/docker.yml
<https://github.com/sagemath/sage/tree/develop/.github/workflows/docker.yml>`_),

- `$SAGE_ROOT/.github/workflows/ci-macos.yml
<https://github.com/sagemath/sage/tree/develop/.github/workflows/ci-macos.yml>`_, and

- `$SAGE_ROOT/.github/workflows/ci-cygwin-standard.yml
<https://github.com/sagemath/sage/tree/develop/.github/workflows/ci-cygwin-standard.yml>`_
(which calls `$SAGE_ROOT/.github/workflows/cygwin.yml
<https://github.com/sagemath/sage/tree/develop/.github/workflows/cygwin.yml>`_).

GitHub Actions runs these build jobs on 2-core machines with 7 GB of
RAM memory and 14 GB of SSD disk space, cf.
`here <https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners#supported-runners-and-hardware-resources>`_,
and has a time limit of 6h per job. This is just barely enough for a
typical ``minimal`` build followed by ``make ptest`` to succeed; and
plenty of time for a typical ``standard`` build to succeed.
and has a time limit of 6h per job. This could be just barely enough for a
typical ``minimal`` build followed by ``make ptest`` to succeed; for
added robustness, we split it into two jobs. Our workflow stores
Docker images corresponding to various build phases within these two
jobs on `GitHub Packages <https://github.com/features/packages>`_ (ghcr.io).

Build logs can be inspected during the run and become available as
"artifacts" when all jobs of the workflow have finished. Each job
generates one tarball. "Annotations" highlight certain top-level
errors or warnings issued during the build.

In addition to these automatic runs in our main repository, all Sage
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think its a good idea to advice people to run the workflows in their own forks. This a old workaround from the time where there was no easy way to run github workflows via trac. Instead I would propose to add labels to run the workflows, similarly how the conda workflow works.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, that's exactly what we want -- so that such tests do not clog the project's Actions runners.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we have enough parallel workflows now that we have a github team account? If a person only has a free github account, then it takes quite a bit of time until the complete matrix is finished since only a couple of them run in parallel - it also blocks other workflows in other projects. (Also not everyone wants to have 3000 published "packages": https://github.com/mkoeppe?tab=packages)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have 60 runners, and the ci-linux workflow, running 24 jobs in parallel, takes a few days. I think that's too much for unleashing it on the project's runner pool.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think 24 jobs each taking a couple of hours (why would they take 'a few days'? its only compiling + running tests) should be okay with 60 runners. If not, we can remove some of the legacy systems and older python versions. I'm also a bit confused since here #35403 (comment) you denied that 'we might be hitting limits of what we have available on GitHub Actions.'

Either way, we cannot expect people to activate actions in their fork. I also don't know of any other open source project that would do this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you denied that 'we might be hitting limits of what we have available on GitHub Actions.'

Context. We are not hitting limits when only every release tag triggers this heavy-weight workflow. But we would be in trouble if 3 of those would be run in parallel; it would stall all other types of workflows.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either way, we cannot expect people to activate actions in their fork

And nobody has to do it unless they want to.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why would they take 'a few days'?

Just look at it: https://github.com/sagemath/sage/actions/workflows/ci-linux.yml

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced that its a good idea to put this in the docs. If someone wants to run it in their fork, she is welcome to do this - but I don't think we should document and support this.

In total it might take a few days, but each workflow takes only a couple of hours and thus only blocks a runner for a few hours. Moreover, due to the staged setup not all parallel runers are consumed at the same time (e.g. currently there are only 5), so that other workflows could be run in between.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's already in the docs, and I put it there for an important purpose: Namely to empower developers to learn about portability work, and that includes running Actions by themselves, not just seeing them run by the magic central repository.

developers can run the same tests on GitHub Actions in their personal
forks of the Sage repository. To prepare this, `enable GitHub Actions <https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository#managing-github-actions-permissions-for-your-repository>`_
in your fork of the Sage repository.

As usual we assume that ``origin`` is the name of the remote
corresponding to your GitHub fork of the Sage repository::

$ git remote -v | grep origin
origin https://github.com/mkoeppe/sage.git (fetch)
origin https://github.com/mkoeppe/sage.git (push)

Build logs become available as "artifacts" when all jobs of the
workflow have finished. Each job generates one tarball.
"Annotations" highlight certain top-level errors or warnings issued
during the build.
Then the following procedure triggers a run of tests with the default set
of system configurations.

The following procedure triggers a run of tests with the default set of
system configurations.
- Push your branch to ``origin`` (your fork).

- Push your changes to trac.
- Go to the `Actions page on the GitHub mirror <https://github.com/sagemath/sagetrac-mirror/actions>`_ and select the workflow you would like to run.
- Click on "Run workflow" above the list of workflow runs and select the branch where the workflow will run.
- Go to the Actions tab of your fork and select the workflow you would like to run,
for example "CI Linux".

For more information, see the `GitHub documentation <https://docs.github.com/en/free-pro-team@latest/actions/managing-workflow-runs/manually-running-a-workflow>`_.
- Click on "Run workflow" above the list of workflow runs and select
your branch as the branch on which the workflow will run.

Alternatively, you can create and push a custom tag in order to trigger a run of tests as follows.
Let's assume that ``my-github`` is the name of
the remote corresponding to your GitHub fork of the Sage repository::
For more information, see the `GitHub documentation
<https://docs.github.com/en/free-pro-team@latest/actions/managing-workflow-runs/manually-running-a-workflow>`_.

$ git remote -v | grep /my-github
my-github https://github.com/mkoeppe/sage.git (fetch)
my-github https://github.com/mkoeppe/sage.git (push)
Alternatively, you can trigger a run of tests by creating and pushing
a custom tag as follows.

- Create a ("lightweight", not "annotated") tag with an arbitrary
name, say ``ci`` (for "Continuous Integration")::
Expand All @@ -1000,36 +1021,21 @@ the remote corresponding to your GitHub fork of the Sage repository::

- Then push the tag to your GitHub repository::

git push -f my-github ci
git push -f origin ci

(In both commands, the "force" option (``-f``) allows overwriting a
previous tag of that name.)

For testing branches against a custom set of system configurations
during development, the following procedure seems to work well. It
avoids changing the CI configuration on your development branch:

- Create a branch from a recent beta release that contains the default
GitHub Actions configuration; name it ``TESTER``, say.

- Edit ``$SAGE_ROOT/.github/workflows/tox.yml`` to include the system
config you wish to test.

- Commit and push the branch to your GitHub fork of sage.

- Push your development branch to your GitHub repository and create a
pull request against the ``TESTER`` branch. This will trigger the
GitHub Actions workflow.

You will find a workflow status page in the "Actions" tab of your
Either way, when the workflow has been triggered, you can inspect it
by using the workflow status page in the "Actions" tab of your
repository.

Here is how to read it. Each of the items in the left pane represents
a full build of Sage on a particular system configuration. A test
item in the left pane is marked with a green checkmark in the left
pane if ``make build doc-html`` finished without error. (It also runs
package testsuites and the Sage doctests but failures in these are not
in reflected in the left pane; see below.)
reflected in the left pane; see below.)

The right pane ("Artifacts") offers archives of the logs for download.

Expand Down