Using `--enable-system-site-packages` is broken

[tornaria]

Using --enable-system-site-packages is broken:

• first, the version of sage_conf inside the venv is installed in such a crooked way that it will not override the one installed in the system site-packages, in spite the sage venv site-packages being listed before the system site-packages in sys.path.
• the version of sage_conf.py I have in my system doesn't define GAP_ROOT_PATHS (not a thing in 10.2). This results in an incredibly scary-looking error (to simulate try GAP_ROOT_PATHS=/ sage -c 'gap(1)' or worse GAP_ROOT_PATHS=/ sage -c 'libgap(1)').
• I don't know how to fix the search order for sage_conf so I just added GAP_ROOT_PATHS to the system sage_conf (I can't remove it since I don't want to break sage for other users)
• With that fixed, almost all tests pass except some in src/sage_docbuild and the top comment in src/sage/env.py

Originally posted by @tornaria in #36181 (comment)

EDIT: some relevant notes that are mentioned in the original post:

• I'm testing with PR Python 3.12.x #36181 at 29130ea, since my system python is 3.12
• I used ./configure --enable-system-site-packages --disable-doc

[tornaria]

More details:

Failures in src/sage_docbuild/*

These are all due to missing sphinx. Indeed, it was not installed in my system (nor was it installed in the venv, maybe because of the --disable-doc). I guess this can be fixed by adding # sage.doctest: needs sphinx at the top of the files as follows:

diff --git a/src/sage_docbuild/__main__.py b/src/sage_docbuild/__main__.py
index 0d15808a69c..acfcb8392a0 100644
--- a/src/sage_docbuild/__main__.py
+++ b/src/sage_docbuild/__main__.py
@@ -1,3 +1,4 @@
+# sage.doctest: needs sphinx
 r"""
 Sage docbuild main
 
diff --git a/src/sage_docbuild/builders.py b/src/sage_docbuild/builders.py
index a43fdda7350..0be43be307f 100644
--- a/src/sage_docbuild/builders.py
+++ b/src/sage_docbuild/builders.py
@@ -1,3 +1,4 @@
+# sage.doctest: needs sphinx
 """
 Documentation builders
 
diff --git a/src/sage_docbuild/sphinxbuild.py b/src/sage_docbuild/sphinxbuild.py
index 5ae1d2e6b10..6d5823934b8 100644
--- a/src/sage_docbuild/sphinxbuild.py
+++ b/src/sage_docbuild/sphinxbuild.py
@@ -1,4 +1,4 @@
-# -*- coding: utf-8 -*-
+# sage.doctest: needs sphinx
 r"""
 Sphinx build script
 

Failure in src/sage/env.py

This is because of a weird mismatch between ./venv/bin/python and ./sage -python:

$ ./sage -python -c 'from sage.env import SAGE_ROOT, SAGE_LOCAL;print(SAGE_ROOT);print(SAGE_LOCAL)'
/home/tornaria/src/sage/sage-git/develop
/home/tornaria/src/sage/sage-git/develop/local

$ ./venv/bin/python  -c 'from sage.env import SAGE_ROOT, SAGE_LOCAL;print(SAGE_ROOT);print(SAGE_LOCAL)'
None
/home/tornaria/src/sage/sage-git/develop/venv

[tornaria]

FWIW, #36997 (comment) goes to confirm that everything is ok with --disable-editable.

Even the sphinx failures are gone, because for non-editable builds the files in src/sage_docbuild are not tested, it seems.

[mkoeppe]

Using --enable-system-site-packages is broken

Yes, but we generally use the euphemism "experimental" for that.

It's broken on almost every tested platform -- see section "standard-sitepackages" in https://github.com/sagemath/sage/actions/workflows/ci-linux.yml
and there appears to be no effort to work on this. @orlitzky @dimpase

IMNSHO, it should not be advertised to end users at all. We already know that it does not work because we are testing it! Sending users into the frustration of trying it out and failing (obviously!) is a terrible practice, and in fact a severe regression back to the pre-2020 practice of trying to do platform support without using platform testing infrastructure.

[dimpase]

this option works for me, and for @orlitzky - I presume.

we advertise more dangerous things to users, e.g. packages which produce wrong maths results

[dimpase]

this option works on a 'generic' platform, with most packages coming straight from PyPI.

Of course it's hopeless - with our ultra-fine degree of package granularity - to accomplish this for a typical Ubuntu or Fedora. The version discrepancy is too much.

For this to work reliably one should get rid of most of this "inner" Jupyter and Sphinx dependencies.

If you must vendor jupyterlab and sphinx, vendor them as whole things, not as gazillion of pieces.

[tornaria]

Using --enable-system-site-packages is broken

Yes, but we generally use the euphemism "experimental" for that.

Which is very different from "not planned". I'm not asking for a refund, I report in good faith and then spent some time and effort in fixing issues that show up.

Is --enable-editable similarly experimental? (I gather not, since it's the default).

The worst part of this breakage is because of a bug in the editable installation of sage-conf which messes up the package search order.

This bug is really with --enable-editable, but it only manifests in a venv with --system-site-packages. Seems worth fixing not only for sage-the-distro-with-system-site-packages sake.

I insist on the following: even if each one needs or wants a different workflow, the underlying infrastructure has similar requirements (modularity, robustness, sane defaults, working out of the box with diverse systems, etc). The issue you have today may be my issue tomorrow, and vice versa.

[mkoeppe]

this option works on a 'generic' platform,

There's no such thing, as you know.

Of course it's hopeless

@dimpase Well OK, then let's please stop advertising it to users as if we'd expect it to work!

[mkoeppe]

Is --enable-editable similarly experimental?

No.

The worst part of this breakage is because of a bug in the editable installation of sage-conf which messes up the package search order.

OK, then let's focus on this one. Separate ticket and let's create a reproducer so we can report this issue to the editables project -- #36181 (comment)

[dimpase]

this option works on a 'generic' platform,

There's no such thing, as you know.

Of course it's hopeless

@dimpase Well OK, then let's please stop advertising it to users as if we'd expect it to work!

well, adding a suitable warning - fine, but hiding it - no.

It works out of the box on Gentoo linux. Why do you want to hide this?

[dimpase]

this option works on a 'generic' platform,

There's no such thing, as you know.

Why? The 'generic' platform is a venv with all needed up to date PyPI packages. No magic, it often just works.

[mkoeppe]

@dimpase Well OK, then let's please stop advertising it to users as if we'd expect it to work!

well, adding a suitable warning - fine, but hiding it - no.

It works out of the box on Gentoo linux. Why do you want to hide this?

In the documentation, it's already marked as "experimental". No change needed there.

I'm referring to you advertising it to users in the mailing lists, such as in:

• https://groups.google.com/g/sage-devel/c/9ETciH8efTE/m/Ov7lIDU_AQAJ