Response files needed sometimes even on non-windows systems

[dankegel]

A user porting www.flintlib.org to meson ran into a commandline length limit
when it tried to link > 3300 files into a single library, with error

ninja: fatal: posix_spawn: Argument list too long

https://www.in-ulm.de/~mascheck/various/argmax/ 
says cygwin has the tightest limit on commandline length, but FreeBSD also has a smallish limit.
I was able to reproduce the error on FreeBSD with the following script:

#!/bin/sh
set -ex
n=1
echo "project('toolong', 'c')" > meson.build
echo "srcs = [" >> meson.build
while test $n -lt 12000
do
    echo "int f$n(int x) { return x + $n; }" > foo$n.c
    echo " 'foo$n.c'," >> meson.build
    n=$(expr $n + 1)
done
echo "]" >> meson.build
echo "library('toolong', srcs)" >> meson.build
meson . btmp
cd btmp
ninja -v

CMake successfully avoids the problem by using a response file, and meson should, too.
(Though ideally it should avoid them when not needed, as they do make debugging a little harder.)

[dankegel]

That script also reproduces the problem with meson from git on ubuntu 20.04, and probably many other flavors of linux.
Evidently 12000 is enough files to make trouble more places.

[dankegel]

This hack fixes it for me for dynamic libraries:

--- a/mesonbuild/linkers.py
+++ b/mesonbuild/linkers.py
@@ -331,7 +331,7 @@ class DynamicLinker(LinkerEnvVarsMixin,
metaclass=abc.ABCMeta):
     def get_accepts_rsp(self) -> bool:
         # rsp files are only used when building on Windows because we want to
         # avoid issues with quoting and max argument length
-        return mesonlib.is_windows()
+        return True;  # mesonlib.is_windows()

That's probably not the right place, GnuDynamicLinker is probably where to overload it.

For static libraries, meson would have to check whether the system's ar
supported @ (ubuntu 20.04's does, but most others don't yet), and override ArLinker's get_accepts_rsp().

[maiphi]

I can confirm that the hack works in the case of FLINT. Note that

• the rsp file is deleted after linking, and
• the fix does not work if the long list of object files is generated manually, e.g. by concatenating extract_all_objects() of several static libs and linking all objects into a single library or executable.

[dankegel]

You can run ninja with -d keeprsp to inhibit deletion of rsp files.

Is the extract_all_objects case common enough to worry about for starters?

[maiphi]

You can run ninja with -d keeprsp to inhibit deletion of rsp files.

Ok.

Is the extract_all_objects case common enough to worry about for starters?

I don't think so. I just stumbled over it, because this is how I switch between my workaround (creating static libs and link them with link_whole into a shared lib) and reproducing the issue in my FLINT build system. The only use case I know is to combine several static libs into one.

[dankegel]

#7227 is ready for first review. Does it solve all the use cases you mentioned above?

[dankegel]

#7245 is merged, this should be fixed now.