SuiteSparse segfaults on PPC64le

[staticfloat]

We haven't been running the testsuite on PPC64le so far, so when I started doing so on the next iteration of the buildbots, I noticed that we don't actually make it through the sparse/sparse tests:

julia> Base.runtests("sparse/sparse")
Test (Worker)     | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)

signal (11): Segmentation fault
while loading /home/juliabuild/buildbot/slave/package_tarballppc64le/build/usr/share/julia/test/sparse/sparse.jl, in expression starting on line 1466
umfzl_create_element at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/usr/bin/../lib/libumfpack.so (unknown line)
umfzl_kernel at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/usr/bin/../lib/libumfpack.so (unknown line)
umfpack_zl_numeric at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/usr/bin/../lib/libumfpack.so (unknown line)
umfpack_numeric! at ./sparse/umfpack.jl:240
lufact at ./sparse/umfpack.jl:148
factorize at ./sparse/linalg.jl:897
normestinv at ./sparse/linalg.jl:552
cond at ./sparse/linalg.jl:534
macro expansion; at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/usr/share/julia/test/sparse/sparse.jl:1474 [inlined]
macro expansion; at ./test.jl:842 [inlined]
anonymous at ./<missing> (unknown line)
jl_call_method_internal at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/julia_internal.h:248 [inlined]
jl_toplevel_eval_flex at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/toplevel.c:652
jl_parse_eval_all at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/ast.c:756
jl_load at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/toplevel.c:679
jl_load_ at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/toplevel.c:686
include_from_node1 at ./loading.jl:532
unknown function (ip: 0x3fff71f325f3)
jl_call_method_internal at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/julia_internal.h:248 [inlined]
jl_apply_generic at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/gf.c:2214
include at ./sysimg.jl:14
macro expansion; at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/usr/share/julia/test/testdefs.jl:13 [inlined]
macro expansion; at ./test.jl:842 [inlined]
macro expansion; at ./util.jl:288 [inlined]
macro expansion; at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/usr/share/julia/test/testdefs.jl:0 [inlined]
anonymous at ./<missing> (unknown line)
jl_call_method_internal at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/julia_internal.h:248 [inlined]
jl_toplevel_eval_flex at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/toplevel.c:652
jl_toplevel_eval_in at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/builtins.c:614
eval at ./boot.jl:238
unknown function (ip: 0x3fff71f0df97)
jl_call_method_internal at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/julia_internal.h:248 [inlined]
jl_apply_generic at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/gf.c:2214
runtests at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/usr/share/julia/test/testdefs.jl:16
#483 at ./multi.jl:1053
run_work_thunk at ./multi.jl:1024
unknown function (ip: 0x3ffd48d146d3)
jl_call_method_internal at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/julia_internal.h:248 [inlined]
jl_apply_generic at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/gf.c:2214
#remotecall_fetch#488 at ./multi.jl:1078
jl_call_method_internal at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/julia_internal.h:248 [inlined]
jl_apply_generic at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/gf.c:2214
jl_apply at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/julia.h:1413 [inlined]
jl_f__apply at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/builtins.c:556
remotecall_fetch at ./multi.jl:1078
jl_call_method_internal at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/julia_internal.h:248 [inlined]
jl_apply_generic at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/gf.c:2214
jl_apply at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/julia.h:1413 [inlined]
jl_f__apply at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/builtins.c:556
#remotecall_fetch#492 at ./multi.jl:1106
jl_call_method_internal at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/julia_internal.h:248 [inlined]
jl_apply_generic at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/gf.c:2214
jl_apply at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/julia.h:1413 [inlined]
jl_f__apply at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/builtins.c:556
remotecall_fetch at ./multi.jl:1106
jl_call_method_internal at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/julia_internal.h:248 [inlined]
jl_apply_generic at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/gf.c:2214
macro expansion at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/usr/share/julia/test/runtests.jl:65 [inlined]
#38 at ./task.jl:332
unknown function (ip: 0x3ffd48d10e17)
jl_call_method_internal at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/julia_internal.h:248 [inlined]
jl_apply_generic at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/gf.c:2214
jl_apply at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/julia.h:1413 [inlined]
start_task at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/task.c:261
jl_set_base_ctx at /home/juliabuild/buildbot/slave/package_tarballppc64le/build/src/task.c:282
Allocations: 48258430 (Pool: 48252559; Big: 5871); GC: 260
ERROR: A test has failed. Please submit a bug report (https://github.com/JuliaLang/julia/issues)
including error messages above and the output of versioninfo():
Julia Version 0.6.0-dev.2143
Commit f4cbc40 (2017-01-19 01:42 UTC)
Platform Info:
  OS: Linux (powerpc64le-unknown-linux-gnu)
  CPU: unknown
  WORD_SIZE: 64
  BLAS: libopenblas (NO_AFFINITY POWER8)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, pwr8)

Looks like it's pointing to this block of code as the culprit, but it doesn't trigger when I just run that piece of code manually. Reporting here in case anyone else wants to take a shot at this.

[tkelman]

We could attempt upgrading, which I haven't touched because I'm not sure how the better-late-than-never official upstream support for building shared libraries will interact with our home grown way of doing it.

[Keno]

The test suite definitely passed on power at some point.

[vchuravy]

I definitely remember it passing in late September.

[staticfloat]

Interesting. It's the particular matrix that we're feeding in on ppc64le. My guess is that our initialization of Ac via sprand() had a different RNG state back then, and so we didn't stumble onto whatever corner case we're experiencing here.

julia> Ac = deserialize(open("Ac.jls"))
20×20 sparse matrix with 306 Complex{Float64} nonzero entries:
	[1 ,  1]  =  0.0+0.40613im
	[2 ,  1]  =  0.337806+0.0im
	[4 ,  1]  =  0.0-0.0846339im
	[5 ,  1]  =  -1.63287-0.13798im
	[6 ,  1]  =  0.0-1.02557im
	[7 ,  1]  =  -0.44613+1.23125im
	[8 ,  1]  =  0.19799+0.027402im
	[9 ,  1]  =  1.14926-2.03573im
	[11,  1]  =  0.298043+0.0im
	[12,  1]  =  1.02148+0.575466im
	[13,  1]  =  0.0+1.31344im
	[15,  1]  =  0.131325+0.0im
	[16,  1]  =  0.0-0.107559im
	[17,  1]  =  0.206393+0.0im
	⋮
	[2 , 20]  =  -1.25471+0.0im
	[3 , 20]  =  1.81056-0.4281im
	[4 , 20]  =  -0.697832+1.29197im
	[5 , 20]  =  0.0-0.610726im
	[7 , 20]  =  0.0-0.969805im
	[9 , 20]  =  2.0124+0.0862868im
	[11, 20]  =  0.992998+0.772603im
	[13, 20]  =  0.260153+0.0im
	[14, 20]  =  -1.39448-1.20073im
	[15, 20]  =  0.0+0.832971im
	[16, 20]  =  -0.723773+0.0im
	[17, 20]  =  -0.145429+1.13026im
	[18, 20]  =  -0.93486+0.0im
	[19, 20]  =  0.194438-2.05935im
	[20, 20]  =  0.0-1.3191im

julia> cond(Ac, 1)

signal (11): Segmentation fault

The serialized Ac is available for download here, I'm going to build a debug SuiteSparse and see if that gives us any more info.

[ViralBShah]

I definitely remember the tests passing too. Perhaps this is due to change in compilers or something?

[staticfloat]

Yes, I think this is indeed a compiler change. I've managed to narrow down that compiling SuiteSparse with -O2 works, but -O2 -ftree-slp-vectorize doesn't. The default is -O3, which includes -ftree-slp-vectorize.

I'm compiling with GCC 6.3.0, built from-source.

[ViralBShah]

In general, I am more comfortable using -O2 as the default level of optimization and using higher levels only selectively. Should we just go to -O2 on only PPC or across the board?

[nalimilan]

In general I don't think it's recommended to build software with -O3. Some Linux distributions even require package maintainers to prove it's really faster than -O2, since in many cases it's actually not.

[tkelman]

This sounds similar to a gcc bug that we found when using musl libc on alpine linux a while back https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71505. That was causing an internal compiler error which I used creduce to shrink down from all of cholmod to

int a, b;
double c, d, e;
double *f;
void fn1() {
  for (;;) {
    e = f[a];
    d = f[a] = f[b];
    c = f[b + 1] = c;
    f[a] = f[a + 1] = d;
    f[b] = f[b + 1] = c;
  }
}

and it wound up due to different qsort behavior giving some out of bounds accesses in the gcc tree vectorizer. Does it work if you compile with clang at -O3 instead? If so I'd translate the test case to C and see how small creduce can shrink code that gives the right answer at -O2 but segfaults at -O3, then report as a gcc bug. If you're as lucky as I was the people who work on gcc's tree vectorizer might fix it fairly quickly.

[staticfloat]

I haven't tried clang yet, but GCC 5.4.0 works just fine, so I'm definitely chalking this up to a compiler regression. I'm not sure I have time to do the creduce stuff right now, if someone else wants to take a stab at it, I'm happy to help.

I think using -O2 by default is reasonable. I'll open a PR adding a patch for suitesparse that does this on master and release-0.5.

[tkelman]

and release-0.5.

don't waste the CI time, it's going to fail until I backport a set of fixes to get travis and appveyor working again on that branch