pyzmq: Fix build errors; zeromq, pyzmq: Update; setuptools: upgrade to 59.2.0

[mkoeppe]

pyzmq fails to build on some platforms. We fix it by

• upgrading setuptools to a version that includes CCompiler.has_function: Do not fail if self.outputdir is set pypa/distutils#65
• correcting the pyzmq build configuration

We also upgrade pyzmq and zeromq to the latest versions.

For example on opensuse-tumbleweed-minimal (https://github.com/mkoeppe/sage/runs/4085305194?check_suite_focus=true)

Setting up build directory for pyzmq-22.2.1
Finished extraction
No patch files found in ../patches
****************************************************
Host system:
Linux dd2167ea4425 5.11.0-1020-azure #21~20.04.1-Ubuntu SMP Mon Oct 11 18:54:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
****************************************************
C compiler: gcc
C compiler version:
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib64/gcc/x86_64-suse-linux/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-suse-linux
Configured with: ../configure --prefix=/usr --infodir=/usr/share/info --mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64 --enable-languages=c,c++,objc,fortran,obj-c++,ada,go,d,jit --enable-offload-targets=nvptx-none,amdgcn-amdhsa, --without-cuda-driver --enable-host-shared --enable-checking=release --disable-werror --with-gxx-include-dir=/usr/include/c++/11 --enable-ssp --disable-libssp --disable-libvtv --enable-cet=auto --disable-libcc1 --enable-plugin --with-bugurl=https://bugs.opensuse.org/ --with-pkgversion='SUSE Linux' --with-slibdir=/lib64 --with-system-zlib --enable-libstdcxx-allocator=new --disable-libstdcxx-pch --enable-libphobos --enable-version-specific-runtime-libs --with-gcc-major-version-only --enable-linker-build-id --enable-linux-futex --enable-gnu-indirect-function --program-suffix=-11 --without-system-libunwind --enable-multilib --with-arch-32=x86-64 --with-tune=generic --with-build-config=bootstrap-lto-lean --enable-link-mutex --build=x86_64-suse-linux --host=x86_64-suse-linux
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.2.1 20210816 [revision 056e324ce46a7924b5cf10f61010cf9dd2ca10e9] (SUSE Linux) 
****************************************************
Package 'pyzmq' is currently not installed
No legacy uninstaller found for 'pyzmq'; nothing to do
Installing pyzmq-22.2.1
Processing /sage/local/var/lib/sage/venv-python3.9.7/var/tmp/sage/build/pyzmq-22.2.1/src
    Preparing wheel metadata: started
    Running command /sage/local/var/lib/sage/venv-python3.9.7/bin/python3 /sage/local/var/lib/sage/venv-python3.9.7/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpn08jd7qu
    INFO: Disabling color, you really want to install colorlog.
    Disabling color, you really want to install colorlog.
    running dist_info
    creating /tmp/pip-modern-metadata-oxt25qia/pyzmq.egg-info
    writing /tmp/pip-modern-metadata-oxt25qia/pyzmq.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-modern-metadata-oxt25qia/pyzmq.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-modern-metadata-oxt25qia/pyzmq.egg-info/requires.txt
    writing top-level names to /tmp/pip-modern-metadata-oxt25qia/pyzmq.egg-info/top_level.txt
    writing manifest file '/tmp/pip-modern-metadata-oxt25qia/pyzmq.egg-info/SOURCES.txt'
    running configure
    Settings obtained from pkg-config: {'library_dirs': [], 'include_dirs': [], 'libraries': ['zmq']}
    {'libraries': ['zmq'], 'include_dirs': [], 'library_dirs': [], 'runtime_library_dirs': [], 'extra_link_args': []}
    C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -g -O2 -g -O2 -fPIC

    compile options: '-c'
    gcc: build/temp.linux-x86_64-3.9/scratch/check_sys_un.c
    gcc build/temp.linux-x86_64-3.9/scratch/check_sys_un.o -o build/temp.linux-x86_64-3.9/scratch/check_sys_un
    Configure: Autodetecting ZMQ settings...
        Custom ZMQ dir:
    Checking for timer_create
    ** Errors about missing timer_create are a normal part of this process **
    ************************************************
    C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -g -O2 -g -O2 -fPIC

    creating build/temp.linux-x86_64-3.9/scratch/tmp
    compile options: '-c'
    gcc: /tmp/timer_create7kgvq7_0.c
    /tmp/timer_create7kgvq7_0.c: In function 'main':
    /tmp/timer_create7kgvq7_0.c:2:5: warning: implicit declaration of function 'timer_create' [-Wimplicit-function-declaration]
        2 |     timer_create();
          |     ^~~~~~~~~~~~
    gcc build/temp.linux-x86_64-3.9/scratch/tmp/timer_create7kgvq7_0.o -o build/temp.linux-x86_64-3.9/scratch/a.out

    error: [Errno 2] No such file or directory: 'a.out'
    Failed with default libzmq, trying again with /usr/local
    {'libraries': ['zmq'], 'include_dirs': ['/usr/local/include'], 'library_dirs': ['/usr/local/lib'], 'runtime_library_dirs': ['/usr/local/lib'], 'extra_link_args': []}
    Configure: Autodetecting ZMQ settings...
        Custom ZMQ dir:       /usr/local
    Checking for timer_create
    ** Errors about missing timer_create are a normal part of this process **
    ************************************************
    C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -g -O2 -g -O2 -fPIC

    compile options: '-c'
    gcc: /tmp/timer_createl5aoyxgr.c
    /tmp/timer_createl5aoyxgr.c: In function 'main':
    /tmp/timer_createl5aoyxgr.c:2:5: warning: implicit declaration of function 'timer_create' [-Wimplicit-function-declaration]
        2 |     timer_create();
          |     ^~~~~~~~~~~~
    Assembler messages:
    Fatal error: can't create build/temp.linux-x86_64-3.9/scratch/tmp/timer_createl5aoyxgr.o: No such file or directory
    ** The above error about timer_create is normal and not a problem! **
    no timer_create, linking librt
    C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -g -O2 -g -O2 -fPIC

    compile options: '-I/usr/local/include -Izmq/utils -c'
    gcc: build/temp.linux-x86_64-3.9/scratch/vers.c
    gcc build/temp.linux-x86_64-3.9/scratch/vers.o -L/usr/local/lib -Wl,--enable-new-dtags,-R/usr/local/lib -lzmq -lrt -o build/temp.linux-x86_64-3.9/scratch/vers
    Error running version detection script:

    build/temp.linux-x86_64-3.9/scratch/vers: error while loading shared libraries: libzmq.so.5: cannot open shared object file: No such file or directory


    error: Error running version detection script:

    build/temp.linux-x86_64-3.9/scratch/vers: error while loading shared libraries: libzmq.so.5: cannot open shared object file: No such file or directory


    Warning: Couldn't find an acceptable libzmq on the system.

    If you expected pyzmq to link against an installed libzmq, please check to make sure:

        * You have a C compiler installed
        * A development version of Python is installed (including headers)
        * A development version of ZMQ >= 3.2 is installed (including headers)
        * If ZMQ is not in a default location, supply the argument --zmq=<path>
        * If you did recently install ZMQ to a default location,
          try rebuilding the ld cache with `sudo ldconfig`
          or specify zmq's location with `--zmq=/usr/local`

    You can skip all this detection/waiting nonsense if you know
    you want pyzmq to bundle libzmq as an extension by passing:

        `--zmq=bundled`

    I will now try to build libzmq as a Python extension
    unless you interrupt me (^C) in the next 10 seconds...

    ************************************************

    10...
     9...
     8...
     7...
     6...
     5...
     4...
     3...
     2...
     1...
    Using bundled libzmq
    already have bundled/zeromq
    attempting ./configure to generate platform.hpp
    Warning: failed to configure libzmq:
    [Errno 2] No such file or directory: './configure'
    staging platform.hpp from: /sage/local/var/lib/sage/venv-python3.9.7/var/tmp/sage/build/pyzmq-22.2.1/src/buildutils/include_linux
    checking for timer_create

Also reported in #31855 comment:22

Upstream: Fixed upstream, in a later stable release.

CC: @enriqueartal @jhpalmieri @dimpase

Component: packages: standard

Author: Matthias Koeppe

Branch: b430bcf

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/32828

[dimpase]

comment:3

    Configure: Autodetecting ZMQ settings...
        Custom ZMQ dir:       /usr/local

weird.

[dimpase]

comment:4

I'd have tried updating zeromq to the latest version, 4.3.4.

[enriqueartal]

comment:5

Replying to @dimpase:

I'd have tried updating zeromq to the latest version, 4.3.4.

It didn't work for me. I tried also to use pyzmq-22.3.0; it fails to download https://github.com/zeromq/libzmq/releases/download/v4.3.4/zeromq-4.3.4.zip and stops (downloading directly works)

Compiling pyzmq outside Sage gives no error; maybe some local variable? Is is possible to use system package?

[mkoeppe]

comment:6

I think the attempted configuration with setup.cfg does not work at all. This was masked so far by SAGE_VENV and SAGE_LOCAL being the same, which has changed.

This is another case where the locations discovered by the configure tests in spkg-configure.m4 are not passed on

[dimpase]

comment:7

Sorry, but the error happens in the minimal build, spkg-configure.m4 is not involved, right?

[mkoeppe]

comment:8

You are right, it isn't; I should have written "this is also another case..."

[enriqueartal]

comment:9

Is there a temptative solution to test?

Replying to @mkoeppe:

I think the attempted configuration with setup.cfg does not work at all. This was masked so far by SAGE_VENV and SAGE_LOCAL being the same, which has changed.

This is another case where the locations discovered by the configure tests in spkg-configure.m4 are not passed on

[enriqueartal]

comment:10

Dirty trick to test the other changes in Fedora 35:

sage -pip install pyzmq==22.0.3

and comment out the install line in spkg-install.in. In other installation I did the same thing with the last version of pyzmq.

I am running tests now.

[enriqueartal]

comment:11

I finished two compilations in Fedora 35, in both cases using a package of cysignals from https://github.com/enriqueartal/cysignals/commit/b404fbef69ddd5cfd0085ab15885e9f0315f9471 which is actually https://github.com/kliem/cysignals/tree/fix_non_constant_MINSIGSTKSZ up to a right parenthesis and installing pymzq as in comment:10.

One compilation is for 9.4. The test gave the following failures:

sage -t --long --warn-long 212.7 --random-seed=0 src/sage/schemes/elliptic_curves/ell_rational_field.py  # 9 doctests failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/doctest/test.py  # 7 doctests failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/symbolic/integration/integral.py  # 1 doctest failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/rings/polynomial/polynomial_zmod_flint.pyx  # Timed out
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/tests/cmdline.py  # 2 doctests failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/modular/abvar/abvar.py  # 1 doctest failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/rings/polynomial/multi_polynomial_ideal.py  # 2 doctests failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/modular/hecke/submodule.py  # 1 doctest failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/lfunctions/sympow.py  # 13 doctests failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/schemes/elliptic_curves/lseries_ell.py  # 3 doctests failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/lfunctions/pari.py  # 1 doctest failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/libs/giac/__init__.py  # 13 doctests failed
sage -t --long --warn-long 212.7 --random-seed=0 src/sage/lfunctions/lcalc.py  # 2 doctests failed

Some of them seem to be harmless (right permissions), other ones may reflect some problems between giac and cysignals.

The second one is for the branch #30766 merged with develop, with the above cysignals and pyzmq and also gmpy2-2.1.0rc1. Besides the errors below, I did not find jupyter in $SAGE_ROOT/local/share but in $SAGE_ROOT/local/var/lib/sage/venv-python3.10/share:

sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/schemes/elliptic_curves/ell_rational_field.py  # 9 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/rings/polynomial/polynomial_zmod_flint.pyx  # Timed out
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/schemes/elliptic_curves/isogeny_small_degree.py  # Timed out
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/rings/function_field/function_field.py  # Timed out
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/doctest/test.py  # 25 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/tests/cmdline.py  # 4 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/misc/sageinspect.py  # 1 doctest failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/dynamics/arithmetic_dynamics/projective_ds.py  # 1 doctest failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/schemes/riemann_surfaces/riemann_surface.py  # 1 doctest failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/modular/abvar/abvar.py  # 1 doctest failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/rings/polynomial/multi_polynomial_ideal.py  # 2 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/modular/hecke/submodule.py  # 1 doctest failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/lfunctions/sympow.py  # 13 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/schemes/elliptic_curves/lseries_ell.py  # 3 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/plot/colors.py  # 5 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage_docbuild/utils.py  # 2 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/lfunctions/pari.py  # 1 doctest failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/libs/giac/__init__.py  # 13 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/structure/unique_representation.py  # 1 doctest failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/lfunctions/lcalc.py  # 2 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/modular/modform/numerical.py  # 3 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/symbolic/callable.py  # 5 doctests failed
sage -t --long --random-seed=212072384500939741616547276375662149298 src/sage/cpython/atexit.pyx  # 6 doctests failed

[mkoeppe]

comment:12

I think the above would be better to discuss in #30767.
Let's please keep this ticket focused on the pyzmq build failures.

[mkoeppe]

comment:13

It looks like pyzmq is incompatible with our use of SETUPTOOLS_USE_DISTUTILS=local.

[enriqueartal]

comment:14

Replying to @mkoeppe:

I think the above would be better to discuss in #30767.
Let's please keep this ticket focused on the pyzmq build failures.

OK, sorry for the noise (#32576 is also involved)

[mkoeppe]

comment:15

Replying to @mkoeppe:

It looks like pyzmq is incompatible with our use of SETUPTOOLS_USE_DISTUTILS=local.

The recent setuptools versions vendor a version of distutils that has a bug described in pypa/distutils#64

[mkoeppe]

Branch: u/mkoeppe/pyzmq__build_errors

[sagetrac-git]

Commit: 89e1dea