Segmentation fault with threads

[qiaojunfeng]

For a simple multithreading code, it seems with version 1.9.0-alpha1, when running with more than 1 threads, julia crash randomly (more than 50%, but there are few chances the code finished fine). However, it is working fine with 1.8.3.

1. The output of versioninfo()

   julia> versioninfo()
   Julia Version 1.9.0-alpha1
   Commit 0540f9d7394 (2022-11-15 14:37 UTC)
   Platform Info:
     OS: Linux (x86_64-linux-gnu)
     CPU: 20 × Intel(R) Core(TM) i9-9900X CPU @ 3.50GHz
     WORD_SIZE: 64
     LIBM: libopenlibm
     LLVM: libLLVM-14.0.6 (ORCJIT, skylake-avx512)
     Threads: 1 on 20 virtual cores

2. How you installed Julia

   juliaup add 1.9  # 1.9.0-alpha1+0.x64.linux.gnu
   juliaup add 1.8  # 1.8.3+0.x64.linux.gnu

3. A minimal working example (MWE), also known as a minimum reproducible example

   nw = 8
   nk = 1000
   A = zeros(ComplexF64, nw, nw, nk)
   B = rand(ComplexF64, nw, nw, nk)
   
   @time Threads.@threads for ik in 1:nk
       for j in 1:nk
           A[:, :, ik] += B[:, :, j]
       end
   end

The crash output is somewhat not unique, following are the output I collected:

[3544842] signal (11.1): Segmentation fault
in expression starting at /tmp/test_perf.jl:11
to_indices at ./indices.jl:354 [inlined]
to_indices at ./indices.jl:344 [inlined]
getindex at ./abstractarray.jl:1294
unknown function (ip: 0x7f97b610ef65)
_jl_invoke at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2450 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2632
macro expansion at /home/jqiao/git/wannier90_dev/examples/example16-noqe/test_perf.jl:14 [inlined]
#9#threadsfor_fun#3 at ./threadingconstructs.jl:139
#9#threadsfor_fun at ./threadingconstructs.jl:106 [inlined]
#1 at ./threadingconstructs.jl:84
unknown function (ip: 0x7f97b610db6f)
_jl_invoke at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2450 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2632
jl_apply at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/julia.h:1868 [inlined]
start_task at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/task.c:1089
Allocations: 11282400 (Pool: 11281162; Big: 1238); GC: 36

[3545031] signal (11.1): Segmentation fault
in expression starting at /tmp/test_perf.jl:11
ijl_get_nth_field at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/datatype.c:1500
ijl_get_nth_field_checked at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/datatype.c:1546
getindex at ./tuple.jl:29 [inlined]
to_indices at ./indices.jl:354 [inlined]
to_indices at ./indices.jl:344 [inlined]
getindex at ./abstractarray.jl:1294
unknown function (ip: 0x7f4e8d2bef65)
_jl_invoke at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2450 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2632
macro expansion at /tmp/test_perf.jl:14 [inlined]
#9#threadsfor_fun#3 at ./threadingconstructs.jl:139
#9#threadsfor_fun at ./threadingconstructs.jl:106 [inlined]
#1 at ./threadingconstructs.jl:84
unknown function (ip: 0x7f4e8d2bdb6f)
_jl_invoke at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2450 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2632
jl_apply at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/julia.h:1868 [inlined]
start_task at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/task.c:1089
Allocations: 5218944 (Pool: 5217706; Big: 1238); GC: 16

[3547161] signal (11.1): Segmentation fault
in expression starting at /tmp/test_perf.jl:11
jl_field_isatomic at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/julia.h:1183 [inlined]
ijl_get_nth_field at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/datatype.c:1507
ijl_get_nth_field_checked at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/datatype.c:1546
getindex at ./tuple.jl:29 [inlined]
to_indices at ./indices.jl:354 [inlined]
to_indices at ./indices.jl:344 [inlined]
setindex! at ./abstractarray.jl:1397
unknown function (ip: 0x7f4994111259)
_jl_invoke at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2450 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2632
macro expansion at /tmp/test_perf.jl:14 [inlined]
#9#threadsfor_fun#3 at ./threadingconstructs.jl:139
#9#threadsfor_fun at ./threadingconstructs.jl:106 [inlined]
#1 at ./threadingconstructs.jl:84
unknown function (ip: 0x7f499410db6f)
_jl_invoke at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2450 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/gf.c:2632
jl_apply at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/julia.h:1868 [inlined]
start_task at /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/src/task.c:1089
Allocations: 6392485 (Pool: 6391247; Big: 1238); GC: 20

[gbaraldi]

Bisected to 1a7a131 @vchuravy @vtjnash

[gbaraldi]

I get an assertion failure with a debug build

julia> @time Threads.@threads for ik in 1:nk
           for j in 1:nk
               A[:, :, ik] += B[:, :, j]
           end
       end
Assertion failed: (ly->fielddesc_type == 2), function jl_field_offset, file julia.h, line 1163.

[vtjnash]

Poking at memory in rr, I saw this was allocated in thread 4 by this thread:

julia> descend(getindex, (Array{Base.Complex{Float64}, 3}, Function, Function, Int64), debuginfo=:none)                  

Then immediately freed (set to NULL) by thread 1 before we get a chance to use it:

Emission is pretty straightforward:

define nonnull {}* @julia_getindex_1682({}* noundef nonnull align 16 dereferenceable(40) %0, {}* noundef nonnull readonly %1, {}* noundef nonnull readonly %2, i64 signext %3) #0 {                                                                
top:                                                                                                                                                                                                                                               
  %I = alloca {}*, align 8                                                                                                                                                                                                                         
...
  %20 = call nonnull {}* ({}* ({}*, {}**, i32)*, {}*, ...) @julia.call({}* ({}*, {}**, i32)* @jl_f_tuple, {}* null, {}* %1, {}* %2, {}* %19)                                                                                                       
...
  fence syncscope("singlethread") seq_cst                                                                                
  %22 = bitcast {}** %ptls to i64**                                                                                                                                                                                                                
  %23 = getelementptr inbounds i64*, i64** %22, i64 2                                                                                                                                                                                              
  %safepoint = load i64*, i64** %23, align 8                                                                             
  %24 = load volatile i64, i64* %safepoint, align 8                                                                      
  fence syncscope("singlethread") seq_cst                                                                                
...
  %70 = load {}*, {}** %I, align 8                                                                                       
  %71 = call {}* @ijl_get_nth_field_checked({}* %70, i64 0)                                                              
...

But we don't end up putting the GC store until the first real call in this function.

[vtjnash]

Seems like @vchuravy possibly already meant to fix this #47393

[gbaraldi]

Why would that fix this? I though that PR just changed the safepoint block to an intrinsic that we lowered later, or would it also impede the safepoint from moving?