Remove the necessity to set SAGE_PARALLEL_SPKG_BUILD

[jhpalmieri]

The patches on this ticket remove necessity to set the environment variable SAGE_PARALLEL_SPKG_BUILD to turn parallel spkg building on. Thus it will be on by default, but note that "parallel spkg building" only has an effect if MAKE is set to, for example, "make -j16".

In the current implementation, parallel building can be disabled by either setting the variable to "no" or by running "make build-serial".

This change was discussed in sage-devel.

Apply:

• attachment: trac_11959-sage.v2-rebased.patch to the Sage library
• attachment: trac_11959-root.v2-rebased.patch to the root repository

Depends on #11926

CC: @nexttime @kcrisman

Component: build

Keywords: SAGE_PARALLEL_SPKG_BUILD MAKE -j --jobs

Author: John Palmieri

Reviewer: Leif Leonhardy

Merged: sage-4.8.alpha0

Issue created by migration from https://trac.sagemath.org/ticket/11959

[jhpalmieri]

Description changed:

--- 
+++ 
@@ -2,3 +2,8 @@
 
 This change was discussed in [sage-devel](https://groups.google.com/d/topic/sage-devel/FW5sP5NPeLs/discussion).
 
+Apply:
+
+- [attachment: trac_11959-sage.patch](https://github.com/sagemath/sage-prod/files/10653929/trac_11959-sage.patch.gz) to the Sage library
+- [attachment: trac_11959-root.patch](https://github.com/sagemath/sage-prod/files/10653928/trac_11959-root.patch.gz) to the root repository
+

[nexttime]

Changed keywords from none to SAGE_PARALLEL_SPKG_BUILD MAKE -j --jobs

[nexttime]

comment:2

Objections to renaming the ticket to "Remove the necessity to set SAGE_PARALLEL_SPKG_BUILD"? (It's yours, so I didn't change its title. ;-) )

[nexttime]

comment:3

John, you confused the names of the patches. :)

[nexttime]

comment:4

My alternate patch to $SAGE_ROOT/spkg/install has become a bit longer:

diff --git a/spkg/install b/spkg/install
--- a/spkg/install
+++ b/spkg/install
@@ -430,12 +430,10 @@
 
 
 # Skip the rest if nothing to do (i.e., to [re]build):
-# Note that we should use $MAKE here, but we currently don't use
-# it further below either unless SAGE_PARALLEL_SPKG_BUILD=yes, which
-# I consider a "bug". -- Leif Leonhardy
-# If "make" doesn't understand the -q option, it should exit with a
-# non-zero status which is not a problem.
-if make -q -f standard/deps $1; then
+# If "make" doesn't understand the -q option (although we require
+# GNU make, which supports it), it should exit with a non-zero status
+# which is not a problem.
+if ${MAKE:-make} -q -f standard/deps $1; then
     echo "Nothing to (re)build / all up-to-date."
     exit 0
 fi
@@ -443,48 +441,57 @@
 
 # Dump environment for debugging purposes:
 echo "*** ALL ENVIRONMENT VARIABLES BEFORE BUILD: ***"
-env
+env | sort
 echo "***********************************************"
 
 
 ###############################################################################
 # NOW do the actual build:
 ###############################################################################
-if [ "$SAGE_PARALLEL_SPKG_BUILD" = "yes" ] && [ -n "$MAKE" ]; then
-    time $MAKE -f standard/deps $1
-else
-    time make -f standard/deps $1
-fi
+time ${MAKE:-make} -f standard/deps $1
 
-# added by Burcin Erocal, see trac #6295.
+# Added by Burcin Erocal, see trac #6295:
 if [ $? -ne 0 ]; then
-    echo "Error building Sage."
+    echo >&2 "Error building Sage."
     exit 1
 fi
 
+# Rename makefile to Makefile (see #10156):
 cd "$SAGE_ROOT"
-
-# Rename makefile to Makefile (see #10156)
 if [ -f makefile ]; then
-    mv makefile Makefile
-    if [ $? -ne 0 ]; then
-        echo "Error renaming 'makefile' to 'Makefile'."
-        exit 1
+    if [ ! -f Makefile ]; then
+        if ! mv makefile Makefile; then
+            echo >&2 "Error renaming 'makefile' to 'Makefile'."
+            exit 1
+        fi
+    else
+        # Both 'makefile' and 'Makefile' exist.
+        echo >&2 "Warning: Deleting old 'makefile'; 'Makefile' remains."
+        if ! rm -f makefile; then
+            echo >&2 "Error deleting old 'makefile'."
+            exit 1
+        fi
     fi
 fi
 
-# build succeeded
-if [ "$1" = "all" ]; then
-    echo "To install gap, gp, singular, etc., scripts"
-    echo "in a standard bin directory, start sage and"
-    echo "type something like"
-    echo "   sage: install_scripts('/usr/local/bin')"
-    echo "at the Sage command prompt."
-    echo ""
-    echo "To build the documentation, run"
-    echo "   make doc"
-    echo ""
-fi
-
+# Build succeeded.
 echo "Sage build/upgrade complete!"
 
+if [ "$1" = "all" ]; then
+    echo
+    echo "To install small scripts to directly run Sage's versions of GAP,"
+    echo "the PARI/GP interpreter, Maxima, or Singular etc. (by typing e.g."
+    echo "just 'gap' or 'gp') into a standard 'bin' directory, start Sage"
+    echo "by typing 'sage' (or './sage') and enter something like"
+    echo
+    echo "    install_scripts('/usr/local/bin')"
+    echo
+    echo "at the Sage command prompt ('sage:')."
+    echo
+    echo "To (re)build the HTML documentation, run 'make doc' (if you haven't"
+    echo "issued 'make' but just 'make build' or e.g. 'sage --upgrade'; other-"
+    echo "wise it will be built automatically right now)."
+    echo
+    echo "To build the PDF version of the documentation, run 'make doc-pdf'."
+    echo
+fi

[nexttime]

comment:5

W.r.t. README.txt:

I'd quote make ('make'), and directly refer to the GNU make (online) manual, i.e., its Options summary section (which IMHO explains -j and -l etc. quite good):

http://www.gnu.org/software/make/manual/make.html#Options-Summary

s/so for example if you set/so for example if you do/ (or ... type)

Perhaps also add that it's usually ok (or advantageous) to use more jobs than the machine has CPU cores (minus already running processes, or the current system load), especially if disk access isn't very fast [provided the machine isn't low of available memory or actually RAM].

[nexttime]

comment:6

W.r.t. the Sage Installation Guide:

export MAKE='make -jNUM' is not "a common setting of MAKE", but a shell command to set MAKE; 'make -jNUM' is the setting. (This has been in before.)

s/won't need to use/won't need to set/ (dito)

s/can speed up the process/can significantly speed up the process/

I'd omit "(Not all packages support this, but the ones for which this could be problematic should automatically disable it.)" since this is pretty irrelevant (or confusing); or rephrase it to say that a few packages do not support building them with multiple jobs and hence are built sequentially, although in parallel with other packages.

In the warning, substitute one-processor by single-core CPU or single-core machine. I'd also mention the platform (or operating system) to which this apparently applies; in my experience it is a non-issue on Linux. Attempting to build Sage with probably too many jobs should in general not hurt; one can simply reduce the number of jobs upon build errors and restart, or more precisely continue, the build afterwards by again typing make.

Similar what I've said w.r.t. README.txt applies to the Installation Guide.

Also we should somehow mention potential problems with ATLAS there, and perhaps what I said about self-tuning packages in general (on sage-devel). (Again, if ATLAS failed to build -- often just due to too high system load -- one can simply type make again, perhaps multiple times, since there are usually less packages left to build when make finally exits. If we haven't already, we should also mention that CPU throttling aka power-saving mode should be turned off; "on-demand" in contrast should be ok, i.e., provided it works properly. [I don't know whether that's in the README.txt; I think it is important enough that it should be there, too.])

[nexttime]

comment:7

P.S.:

I think (e.g.) MAKE="make -j3 -l10.0" or MAKE="make -j 3 -l 10.0" (or MAKE="make --jobs=3 --load-average=10.0") is more natural.

(Although AFAIK older versions of GNU make interpreted make -j 3 as "build target(!) '3' with infinitely many(!) jobs", so the first and last version may be safer.)

[williamstein]

comment:8

I'm against this as is, but I think something very close is OK.

I think SAGE_PARALLEL_SPKG_BUILD="no" should be supported, but otherwise everything should be like in this proposal. The reason is that it can be confusing trying to deal with build issues when the build log mixes up output from multiple packages being built at once. Also there can be weird build issues resulting from building multiple packages at once. However, using "make -j16" (say) is very useful in all cases on a machine with lots of processors so you don't have to wait so long for individual packages to build.