Skip to content

Conversation

Lorak-mmk
Copy link

@Lorak-mmk Lorak-mmk commented Jul 3, 2025

The goal of this is to utilize pyproject.toml and uv tool for management of Python versions and dependencies, instead of older methods like manually creating venvs and installing stuff from requirements.txt.
This will ultimately result in tests / tools being easier to run, and less error prone (I can't even count how many hours I wasted because of some misconfigurations related to that....), and faster.

First commits change the docs to not call setup.py anymore. Without this, many changes would cause docs to break, even something simple like using license field in pyproject.toml.

After this first necessary fix, few commits do some minor improvements in pyproject.toml, like formatting, specifying Python version, or moving pytest config there.
After that, I fix the dependencies list in pyproject.toml, becuse it contained some errors making it impossible to run tests using it.

Then, the integration test pipeline is moved to uv. To make it easier, I remove the experiments section first in a separate commit - tablets are definitely not an experiment anymore.
After that, I move the other part: lib-build-and-push.yml. This allowed me to remove legacy requirements files in next commit.

Last 2 commits adjust documentation, and make one final improvement: building extensions in parallel by default.

To reviewers: Please verify that you are able to build the project locally, run unit and integration tests.
I recommend having ccache (or sccache) set up. With it, the overhead of having to compile extensions basically disappears.

TODO:

  • Verify that all the pipelines still work. I don't fully understand all the yaml stuff, especially in lib-build-and-push.yml, so I may have broken something. @dkropachev you wrote them, so please review my changes.
  • Verify our dependencies.
    • Should we have optional dependency groups per event loop? I think so, but it can be done in the future.
  • Moving docs to uv as well. This can also be done in the future.

Pre-review checklist

  • I have split my patch into logically separate commits.
  • All commit messages clearly explain what they change and why.
  • I added relevant tests for new features and bug fixes. - no new features or bug fixes
  • All commits compile, pass static checks and pass test.
  • PR description sums up the changes and reasons why they should be introduced.
  • I have provided docstrings for the public items that I want to introduce. - no new items
  • I have adjusted the documentation in ./docs/source/.
  • I added appropriate Fixes: annotations to PR description.

@Lorak-mmk Lorak-mmk force-pushed the uv branch 9 times, most recently from 2d22f14 to 040d142 Compare July 4, 2025 10:36
@Lorak-mmk Lorak-mmk self-assigned this Jul 4, 2025
Lorak-mmk added 11 commits July 4, 2025 14:26
Calling setup.py from command line is deprecated by Python for a very
long time now.
This is a new standardized way to have dev dependencies.
We use this in CI, and it is generally useful, so lets not require
manually adding it.
This reduces amount of files in root and move towards centralizing
configuration.
Those were removed some time ago in test-requirements.txt, but not here.
They make it impossible to resolve dependencies on Python 3.13 because
gevent 23.9.0 doesn't have wheels for it.
This special case was, as far as I can tell, somehow ignored in our CI.
By this I mean that even on windows build it used newer gevent. Because
of that, removing those cases shouldn't break anything.
Tablets are now available on Scylla version that we run CI with, so I
see no reason to keep it separate.
@Lorak-mmk Lorak-mmk marked this pull request as ready for review July 4, 2025 14:11
@Lorak-mmk Lorak-mmk requested review from dkropachev, Copilot and fruch July 4, 2025 14:11
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR transitions the project to use uv and pyproject.toml for Python version and dependency management, removing legacy requirements.txt, setup scripts, and manual venv setups.

  • Replace legacy test requirements files and scripts with a single source of truth in pyproject.toml and uv tooling
  • Consolidate pytest, cibuildwheel, and build settings into pyproject.toml and update CI workflows to use uv
  • Update documentation and CI pipelines to invoke uv instead of direct setup.py, pip, or shell scripts

Reviewed Changes

Copilot reviewed 12 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
test-requirements.txt Removed in favor of pyproject.toml dependency-groups.
setup.py Defaulted build_concurrency to None and adjusted parallel build condition.
scripts/run_integration_test.sh Removed legacy integration test script, replaced by CI uv steps.
pytest.ini Deleted; pytest options moved into pyproject.toml.
pyproject.toml Centralized project config, dependencies, uv cache-keys, pytest and cibuildwheel settings.
docs/pyproject.toml Updated docs build dependencies and linked local driver.
docs/installation.rst Switched docs instructions to uv and added default concurrency note.
README-dev.rst Added uv tool recommendation and updated test commands.
.github/workflows/lib-build-and-push.yml Swapped actions/setup-python for astral-sh/setup-uv, updated build and test commands.
.github/workflows/integration-tests.yml Removed shell script, added uv install, sync, and test steps.
.github/workflows/docs-pr.yaml Dropped setup.py develop step before docs build.
.github/workflows/docs-pages.yaml Dropped setup.py develop step before docs multiversion build.
Comments suppressed due to low confidence (2)

.github/workflows/docs-pr.yaml:29

  • Docs build step now omits installing the driver, which may cause Sphinx to fail importing modules. Consider adding a 'uv sync' or an equivalent install step before building docs.
      - name: Build docs

.github/workflows/docs-pages.yaml:29

  • Docs pages pipeline no longer installs the driver before building docs, which could break the documentation build. Add a step such as 'uv sync' or 'pip install -e .' prior to 'make multiversion'.
        run: make -C docs setupenv

Lorak-mmk added 8 commits July 6, 2025 16:00
No longer necessary for anything.
No longer needed for anything, and we should not have 2 places to keep
dependencies in. It will, and already has, resulted in those places
being out of sync.
Because why not? It will be faster - on my machine ~4x faster.
When a file or variable used by setup.py changes, we should rebuild the
driver. If we don't we risk very unpleasant errors caused by source
files and binary files differing.
@dkropachev dkropachev merged commit d540e14 into scylladb:master Jul 6, 2025
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants