Segfault in multithreaded FFTW

[s-broda]

Repro:

C:\Users\broda\AppData\Local\Julia-0.5.0\bin>julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.5.0 (2016-09-19 18:14 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-w64-mingw32

julia> versioninfo()
Julia Version 0.5.0
Commit 3c9d753 (2016-09-19 18:14 UTC)
Platform Info:
  System: NT (x86_64-w64-mingw32)
  CPU: Intel(R) Processor 5Y10 CPU @ 0.80GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, broadwell)

julia> Threads.nthreads()
1

julia> a=randn(10^7,1);

julia> rfft(a);

julia> FFTW.set_num_threads(2)

julia> rfft(a);

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x706d82b9 -- fftw_printer_destroy at C:\Users\broda\AppData\Local\Julia-0.5.0\bin\libfftw3.DLL (unknown line)
while loading no file, in expression starting on line 0
fftw_printer_destroy at C:\Users\broda\AppData\Local\Julia-0.5.0\bin\libfftw3.DLL (unknown line)
fftw_plan_awake at C:\Users\broda\AppData\Local\Julia-0.5.0\bin\libfftw3.DLL (unknown line)
fftw_plan_awake at C:\Users\broda\AppData\Local\Julia-0.5.0\bin\libfftw3.DLL (unknown line)
fftw_destroy_plan at C:\Users\broda\AppData\Local\Julia-0.5.0\bin\libfftw3.DLL (unknown line)
destroy_plan at .\fft\FFTW.jl:229
unknown function (ip: 0000000015E10C56)
jl_call_method_internal at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\julia_internal.h:189 [inlined]
jl_apply_generic at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\gf.c:1942
jl_apply at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\julia.h:1392 [inlined]
run_finalizer at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\gc.c:108
jl_gc_run_finalizers_in_list at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\gc.c:190 [inlined]
run_finalizers at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\gc.c:211
jl_gc_collect at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\gc.c:1872
jl_gc_pool_alloc at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\gc.c:828 [inlined]
jl_gc_alloc_ at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\julia_internal.h:148 [inlined]
jl_gc_alloc at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\gc.c:1881
_new_array_ at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\array.c:108
_new_array at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\array.c:149 [inlined]
jl_new_array at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\array.c:341
* at .\fft\FFTW.jl:694 [inlined]
rfft at .\dft.jl:56
unknown function (ip: 0000000015E090D6)
jl_call_method_internal at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\julia_internal.h:189 [inlined]
jl_apply_generic at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\gf.c:1942
do_call at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\interpreter.c:66
eval at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\interpreter.c:190
jl_toplevel_eval_flex at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\toplevel.c:558 [inlined]
jl_toplevel_eval at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\toplevel.c:580
jl_toplevel_eval_in_warn at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\builtins.c:590
eval at .\boot.jl:234
unknown function (ip: 000000000583FB05)
jl_call_method_internal at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\julia_internal.h:189 [inlined]
jl_apply_generic at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\gf.c:1942
eval_user_input at .\REPL.jl:64
unknown function (ip: 0000000015DFA51A)
jl_call_method_internal at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\julia_internal.h:189 [inlined]
jl_apply_generic at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\gf.c:1942
macro expansion at .\REPL.jl:95 [inlined]
#3 at .\event.jl:68
unknown function (ip: 0000000015DF1003)
jl_call_method_internal at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\julia_internal.h:189 [inlined]
jl_apply_generic at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\gf.c:1942
jl_apply at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\julia.h:1392 [inlined]
start_task at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\task.c:253
Allocations: 3546638 (Pool: 3545593; Big: 1045); GC: 5

The segfault is only triggered by this specific sequence of commands; in particular, it is not triggered if I don't call rfft(a) once before FFTW.set_num_threads(2). Also, with a=rand(10^3,1), the segfault doesn't occur right away, but when I call quit().

I'm seeing the same in Ubuntu on Windows, on a slightly outdated Master:

broda@Zenbook:~$ julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.0-dev.1534 (2016-12-10 09:07 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit aea6e89 (27 days old master)
|__/                   |  x86_64-pc-linux-gnu

julia> versioninfo()
Julia Version 0.6.0-dev.1534
Commit aea6e89 (2016-12-10 09:07 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Processor 5Y10 CPU @ 0.80GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, broadwell)

julia> Threads.nthreads()
1

julia> a=rand(10^7,1);

julia> rfft(a);

julia> FFTW.set_num_threads(2);

julia> rfft(a);

signal (11): Segmentation fault
while loading no file, in expression starting on line 0
unknown function (ip: 0x7f552d933303)
fftw_plan_awake at /home/broda/julia-aea6e89873/bin/../lib/julia/libfftw3.so.3 (unknown line)
fftw_plan_awake at /home/broda/julia-aea6e89873/bin/../lib/julia/libfftw3.so.3 (unknown line)
fftw_destroy_plan at /home/broda/julia-aea6e89873/bin/../lib/julia/libfftw3.so.3 (unknown line)
destroy_plan at ./fft/FFTW.jl:229
unknown function (ip: 0x7f5532e50492)
jl_call_method_internal at /home/centos/buildbot/slave/package_tarball64/build/src/julia_internal.h:240 [inlined]
jl_apply_generic at /home/centos/buildbot/slave/package_tarball64/build/src/gf.c:1861
jl_apply at /home/centos/buildbot/slave/package_tarball64/build/src/julia.h:1378 [inlined]
run_finalizer at /home/centos/buildbot/slave/package_tarball64/build/src/gc.c:108
jl_gc_run_finalizers_in_list at /home/centos/buildbot/slave/package_tarball64/build/src/gc.c:190 [inlined]
run_finalizers at /home/centos/buildbot/slave/package_tarball64/build/src/gc.c:218
run_finalizers at /home/centos/buildbot/slave/package_tarball64/build/src/gc.c:204 [inlined]
jl_gc_collect at /home/centos/buildbot/slave/package_tarball64/build/src/gc.c:1913
jl_gc_pool_alloc at /home/centos/buildbot/slave/package_tarball64/build/src/gc.c:837
jl_gc_alloc_ at /home/centos/buildbot/slave/package_tarball64/build/src/julia_internal.h:166 [inlined]
_new_array_ at /home/centos/buildbot/slave/package_tarball64/build/src/array.c:108 [inlined]
_new_array at /home/centos/buildbot/slave/package_tarball64/build/src/array.c:149 [inlined]
jl_alloc_array_2d at /home/centos/buildbot/slave/package_tarball64/build/src/array.c:353
* at ./fft/FFTW.jl:694 [inlined]
rfft at ./dft.jl:56
unknown function (ip: 0x7f5532e49fd2)
jl_call_method_internal at /home/centos/buildbot/slave/package_tarball64/build/src/julia_internal.h:240 [inlined]
jl_apply_generic at /home/centos/buildbot/slave/package_tarball64/build/src/gf.c:1861
do_call at /home/centos/buildbot/slave/package_tarball64/build/src/interpreter.c:71
eval at /home/centos/buildbot/slave/package_tarball64/build/src/interpreter.c:210
jl_toplevel_eval_flex at /home/centos/buildbot/slave/package_tarball64/build/src/toplevel.c:628
jl_toplevel_eval_in at /home/centos/buildbot/slave/package_tarball64/build/src/builtins.c:577
eval at ./boot.jl:236
unknown function (ip: 0x7f57444b08ff)
jl_call_method_internal at /home/centos/buildbot/slave/package_tarball64/build/src/julia_internal.h:240 [inlined]
jl_apply_generic at /home/centos/buildbot/slave/package_tarball64/build/src/gf.c:1861
eval_user_input at ./REPL.jl:66
unknown function (ip: 0x7f5532e398d6)
jl_call_method_internal at /home/centos/buildbot/slave/package_tarball64/build/src/julia_internal.h:240 [inlined]
jl_apply_generic at /home/centos/buildbot/slave/package_tarball64/build/src/gf.c:1861
macro expansion at ./REPL.jl:97 [inlined]
#3 at ./event.jl:66
unknown function (ip: 0x7f5532e30fbf)
jl_call_method_internal at /home/centos/buildbot/slave/package_tarball64/build/src/julia_internal.h:240 [inlined]
jl_apply_generic at /home/centos/buildbot/slave/package_tarball64/build/src/gf.c:1861
jl_apply at /home/centos/buildbot/slave/package_tarball64/build/src/julia.h:1378 [inlined]
start_task at /home/centos/buildbot/slave/package_tarball64/build/src/task.c:260
unknown function (ip: 0xffffffffffffffff)
Allocations: 3076701 (Pool: 3075674; Big: 1027); GC: 6
Segmentation fault (core dumped)

[giordano]

I think I experience a similar problem in Julia master. Even without performing a rfft again after changing the number of FFTW threads, julia crashes upon exit. Recipe:

a=rand(10^7,1);
rfft(a);
FFTW.set_num_threads(2)
# do anything...
exit() # or CTRL + D in the REPL

[stevengj]

My fault, I think. fftw_init_threads is called the first time you call FFTW.set_num_threads, but I forgot that this invalidates any existing plans (because plans share trigonometric tables in a global cache).

When FFTW is moved into its own module, it should just call fftw_init_threads from the module __init__. This is not an expensive function: it just creates a couple of mutexes and inserts some hooks into the planner; for now we could just call it whenever a plan is created. (It is initialized to use only 1 thread.)

It is also safe to call fftw_init_threads more than once, because it internally checks whether threads have already been initialized and does nothing if they have.

[giordano]

Thanks for looking into this, however even running

ccall((:fftw_init_threads,Base.FFTW.libfftw), Int32, ())

before the code above, Julia sometimes segfaults (but not systematically).

[stevengj]

@giordano, that's because fftw_num_threads doesn't know you've called fftw_init_threads; it calls fftw_cleanup and initializes it again.

Try doing FFTW.set_num_threads(1) before doing anything.

[stevengj]

Hmm, it might be sufficient for us to change FFTW.set_num_threads to call fftw_forget_wisdom rather than fftw_cleanup. This way, existing plans wouldn't be invalidated.

[giordano]

@giordano, that's because fftw_num_threads doesn't know you've called fftw_init_threads; it calls fftw_cleanup and initializes it again.

Oh, I see, from your previous message I thought that the single call to fftw_init_threads was sufficient.

Try doing FFTW.set_num_threads(1) before doing anything.

Tried 30 times, no crash. Thanks!