homebrew: Unused packages (singular, pari, ...) in /usr/local leak into sagelib/cysignals builds via distutils.cfg

[mkoeppe]

Follow-up from #31132:

It has been reported that it's still leaking with the upgraded python3 from homebrew that includes Homebrew/homebrew-core@784d292

The sagelib build collects many -I, -L directives through pkgconfig.
Such directives on the command line take precedence over the search paths set by sage-env in $CPATH, $LIBRARY_PATH.
Now if -I/usr/local/ or -L/usr/local (instead of more specific paths to /usr/local/Cellar/.../...) appear, then unwanted packages that happen to be installed can leak in.

However, here the problem is caused by homebrew installing a distutils.cfg that injects /usr/local/include into our include search path.
We fix the leak by switching to a modern configuration of setuptools.

This also affects cysignals, as reported in https://groups.google.com/g/sage-release/c/KdSKg6RdZok/m/91Ibwn8RAgAJ

Relevant tickets: #13348, #14709, #29562 (+), #29607 (+), #29697 (?), #31041 (), #30818 (), #30013 (~); possible follow-up: #31338

CC: @jhpalmieri @zlscherr @kiwifb @kliem @slel

Component: build

Author: Matthias Koeppe

Branch: e03c5ac

Reviewer: John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/31335

[mkoeppe]

Dependencies: #30589, #31132, #31227

[mkoeppe]

Description changed:

--- 
+++ 
@@ -2,4 +2,8 @@
 
 It's still leaking.
 
+The `sagelib` build collects many `-I`, `-L` directives through pkgconfig. 
+Such directives on the command line take precedence over the search paths set by `sage-env` in `$CPATH`, `$LIBRARY_PATH`.
 
+
+

[mkoeppe]

Description changed:

--- 
+++ 
@@ -5,5 +5,5 @@
 The `sagelib` build collects many `-I`, `-L` directives through pkgconfig. 
 Such directives on the command line take precedence over the search paths set by `sage-env` in `$CPATH`, `$LIBRARY_PATH`.
 
+Relevant tickets: #29562 (+), #29697 (?), #31041 (~)
 
-

[mkoeppe]

Description changed:

--- 
+++ 
@@ -5,5 +5,5 @@
 The `sagelib` build collects many `-I`, `-L` directives through pkgconfig. 
 Such directives on the command line take precedence over the search paths set by `sage-env` in `$CPATH`, `$LIBRARY_PATH`.
 
-Relevant tickets: #29562 (+), #29697 (?), #31041 (~)
+Relevant tickets: #29562 (+), #29697 (?), #31041 (~), #30818 (~), #30013 (~)
 

[mkoeppe]

Description changed:

--- 
+++ 
@@ -5,5 +5,8 @@
 The `sagelib` build collects many `-I`, `-L` directives through pkgconfig. 
 Such directives on the command line take precedence over the search paths set by `sage-env` in `$CPATH`, `$LIBRARY_PATH`.
 
-Relevant tickets: #29562 (+), #29697 (?), #31041 (~), #30818 (~), #30013 (~)
+To make sure that 
 
+
+Relevant tickets: #13348, #29562 (+), #29607 (+), #29697 (?), #31041 (~), #30818 (~), #30013 (~)
+

[mkoeppe]

Description changed:

--- 
+++ 
@@ -8,5 +8,5 @@
 To make sure that 
 
 
-Relevant tickets: #13348, #29562 (+), #29607 (+), #29697 (?), #31041 (~), #30818 (~), #30013 (~)
+Relevant tickets: #13348, #14709, #29562 (+), #29607 (+), #29697 (?), #31041 (~), #30818 (~), #30013 (~)
 

[mkoeppe]

Description changed:

--- 
+++ 
@@ -5,7 +5,15 @@
 The `sagelib` build collects many `-I`, `-L` directives through pkgconfig. 
 Such directives on the command line take precedence over the search paths set by `sage-env` in `$CPATH`, `$LIBRARY_PATH`.
 
-To make sure that 
+Now if `-I/usr/local/` or `-L/usr/local` (instead of more specific paths to `/usr/local/Cellar/.../...`) appear, then unwanted packages that happen to be installed can leak in.
+
+To check whether this comes in through `pkg-config`:
+
+```
+for a in $(pkg-config --list-all | sed 's/ .*//;'); do printf "$a: "; pkg-config --cflags $a; printf "$a: "; pkg-config --libs $a; done
+```
+
+To make sure that libraries installed in $SAGE_LOCAL (if set) take precedence over such leaks, we should essentially revert #29697.
 
 
 Relevant tickets: #13348, #14709, #29562 (+), #29607 (+), #29697 (?), #31041 (~), #30818 (~), #30013 (~)

[mkoeppe]

Description changed:

--- 
+++ 
@@ -13,8 +13,9 @@
 for a in $(pkg-config --list-all | sed 's/ .*//;'); do printf "$a: "; pkg-config --cflags $a; printf "$a: "; pkg-config --libs $a; done
 ```
 
-To make sure that libraries installed in $SAGE_LOCAL (if set) take precedence over such leaks, we should essentially revert #29697.
-
+To make sure that libraries installed in $SAGE_LOCAL (if set) take precedence over such leaks, we should essentially revert #29697, adding some refinements:
+- `SAGE_LOCAL` vs `SAGE_VENV`
+- make sure that if `$SAGE_LOCAL` is unset, nothing is added
 
 Relevant tickets: #13348, #14709, #29562 (+), #29607 (+), #29697 (?), #31041 (~), #30818 (~), #30013 (~)
 

[mkoeppe]

comment:9

To check whether this comes in through pkg-config, I used:

for a in $(pkg-config --list-all | sed 's/ .*//;'); do printf "$a: "; pkg-config --cflags $a; printf "$a: "; pkg-config --libs $a; done

However, on my system, the command line in the ticket description does not find any relevant packages that directly use /usr/local/{include,lib} (only lua and fuse do)

[mkoeppe]

Description changed:

--- 
+++ 
@@ -13,9 +13,9 @@
 for a in $(pkg-config --list-all | sed 's/ .*//;'); do printf "$a: "; pkg-config --cflags $a; printf "$a: "; pkg-config --libs $a; done
 ```
 
-To make sure that libraries installed in $SAGE_LOCAL (if set) take precedence over such leaks, we should essentially revert #29697, adding some refinements:
+To make sure that libraries installed in $SAGE_LOCAL (if set) take precedence over such leaks, we could essentially revert #29697, adding some refinements:
 - `SAGE_LOCAL` vs `SAGE_VENV`
-- make sure that if `$SAGE_LOCAL` is unset, nothing is added
+- make sure that if `$SAGE_LOCAL` is unset, nothing is added.
 
 Relevant tickets: #13348, #14709, #29562 (+), #29607 (+), #29697 (?), #31041 (~), #30818 (~), #30013 (~)
 

[mkoeppe]

Changed dependencies from #30589, #31132, #31227 to #30589, #31132, #31227, #30770

[mkoeppe]

Branch: u/mkoeppe/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build

[mkoeppe]

comment:13

With the tickets listed in Dependencies merged (that's the branch on this ticket) and after brew install singular pari ecl, I get leakage from singular:

[sagelib-9.3.beta6] [ 1/13] gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -O2 -g -march=native -I./sage/cpython -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src -I/usr/local/Cellar/python@3.9/3.9.1_7/Frameworks/Python.framework/Versions/3.9/include/python3.9 -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.9/site-packages/numpy/core/include -Ibuild/cythonized -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/usr/local/opt/tcl-tk/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include -I/usr/local/Cellar/python@3.9/3.9.1_7/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c build/cythonized/sage/algebras/letterplace/free_algebra_letterplace.cpp -o build/temp.macosx-10.15-x86_64-3.9/build/cythonized/sage/algebras/letterplace/free_algebra_letterplace.o -fno-strict-aliasing -DCYTHON_CLINE_IN_TRACEBACK=1 -DSING_NDEBUG -DOM_NDEBUG -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular -std=c++11
[sagelib-9.3.beta6] In file included from build/cythonized/sage/algebras/letterplace/free_algebra_letterplace.cpp:672:
[sagelib-9.3.beta6] In file included from /usr/local/include/singular/Singular/libsingular.h:7:
[sagelib-9.3.beta6] In file included from /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular/kernel/structs.h:23:
[sagelib-9.3.beta6] In file included from /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular/kernel/polys.h:15:
[sagelib-9.3.beta6] In file included from /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular/polys/monomials/ring.h:13:
[sagelib-9.3.beta6] In file included from /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/include/singular/coeffs/coeffs.h:19:
[sagelib-9.3.beta6] In file included from /usr/local/include/factory/factory.h:28:
[sagelib-9.3.beta6] /usr/local/include/factory/si_log2.h:6:19: error: redefinition of 'SI_LOG2'
[sagelib-9.3.beta6] static inline int SI_LOG2(int v)

[mkoeppe]

Changed branch from u/mkoeppe/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_sagelib_build to none

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,6 +1,6 @@
 Follow-up from #31132:
 
-It's still leaking.
+It has been reported that it's still leaking.
 
 The `sagelib` build collects many `-I`, `-L` directives through pkgconfig. 
 Such directives on the command line take precedence over the search paths set by `sage-env` in `$CPATH`, `$LIBRARY_PATH`.