Darwin/ARM64: Julia freezes on @threads loops

[ssagl]

I'm encountering a previously documented issue when running @threads loops on the arm64 native 1.8.0-rc1 version of julia. Apparently, this has been a known issue before but marked as closed (link to issue). The bug still seems to exist and is causing troubles for me. I'll quote a minimal example from the thread that initially documented this issue (Originally posted by @gbaraldi in #41820 (comment)):

julia> function foo()
          Threads.@threads for i in 1:10
               rand()
           end
       end
foo (generic function with 1 method)

julia> for i in 1:1000
           println(i)
           for j in 1:10000
               foo()
           end
       end

This freezes for me somewhere in i = 11 on my Mac with the current 1.8.0-rc1 version of julia.

Below I attach the exact output of versioninfo():

julia> versiononfo()
Julia Version 1.8.0-rc1
Commit 6368fdc6565 (2022-05-27 18:33 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.3.0)
  CPU: 10 × Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 8 on 8 virtual cores

[gbaraldi]

I can't reproduce it all 🤔

[ssagl]

thank you @gbaraldi. I ran into this issue multiple times over the last 48 hours and was able to consistently reproduce it in the REPL in julia 1.7.2 as well as in julia 1.8.0-beta3 and julia 1.8.0-rc1 and thought this was happening all the time. After reading your post, I tried to rerun what I did earlier and it seemingly disappeared which confused me a lot.

However, there's something new and even more interesting I found out now thanks to your post: this is not happening every time I run this script. I have to run the script over and over again and it only sometimes freezes (requires quite a lot of tries). Right now I have two different terminals opened on my Mac in which I ran the above posted codes and they are frozen at i = 8 and i = 11 respectively.

[ssagl]

something more I found out by now: this is even more likely to happen when using 2 terminal sessions executing the above codes at the same time, but running two sessions is not necessary to produce the freeze on my system. Also, I'm using 8 threads but this should be apparent from my versioninfo() I posted above.

[fxcoudert]

I've been trying to reproduce this with:

Julia Version 1.9.0-DEV.612
Commit 2159bfba2f (2022-05-18 18:02 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.5.0)
  CPU: 8 × Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 6 on 6 virtual cores

but I can't. I can even run up to 8 terminals running this same code, all on 6 threads, without issue. I repeated multiple times, but cannot get it to freeze. So either it's specific to 1.8, or there is some weird trigger hidden…

[ssagl]

thanks for your effort @fxcoudert. I recorded my screen when doing this with two open terminal sessions because I'm so puzzled as to why this would only happen on my machine. I really don't know what the trigger is, but if I do something wrong in the terminal session please tell me. The function foo.jl that I run in these terminal sessions is exactly what I posted above...

threads_freeze.mov

[giordano]

I was able to reproduce #41820 very easily back when it was a problem, but #43418 fixed it for me. I tried to run the code above multiple times, it never froze:

julia> versioninfo()
Julia Version 1.9.0-DEV.732
Commit ada860fe7d* (2022-06-10 15:07 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.5.0)
  CPU: 8 × Apple M1
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 4 on 4 virtual cores

@ssagl can you check CPU usage in Activity Monitor? When #41820 was an issue it used to drop to 0 for me, indicating a deadlock situation. I agree with @fxcoudert that there may be something else going on causing the issue for you.

[ssagl]

thanks @giordano. I checked and when it freezes it seems like CPU usage is not going to zero but goes down from initially around 400 % to 100%. Screenshot from the Activity Monitor when it freezes below. Doesn't seem like a deadlock like you experienced previously.

[vchuravy]

Is there a way on Mac OS to see which cores it got scheduled on? Maybe it's something with the efficiency cores?

[ssagl]

@vchuravy yes there is, thanks! I attach a snapshot of my CPU usage history when running the codes. Right when it starts to freeze, performance cores 6, 7, 8, 9, and 10 seem to not do anything anymore. Performance core 6 seems to come back at some time but not sure what that all means.

[ngam]

Can reproduce with two concurrent sessions (i.e. two terminals with the julia-1.8 -t 8 foo.jl like above) but interestingly cannot reproduce with -t 3 or lower and it tends to get stuck at a higher value (in the counter) for lower -t x, e.g. stopping around 30 for -t 8, 70 for -t 7, 120 for -t 5, 700 for -t 5 (but sometimes, around 50 for -t 4). Only one of the two processes got stuck for me.

I will let @vchuravy take it from here... but do note the behavior of the -t x vis-a-vis "virtual cores" in the collapsed section below.

~$ julia-1.8            
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.0-rc1 (2022-05-27)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> versioninfo
versioninfo (generic function with 2 methods)

julia> versioninfo()
Julia Version 1.8.0-rc1
Commit 6368fdc6565 (2022-05-27 18:33 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.3.0)
  CPU: 10 × Apple M1 Max
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 1 on 8 virtual cores

julia> versioninfo^C

julia> 
~$ julia-1.8 -t 8
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.0-rc1 (2022-05-27)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> versioninfo()
Julia Version 1.8.0-rc1
Commit 6368fdc6565 (2022-05-27 18:33 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.3.0)
  CPU: 10 × Apple M1 Max
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 8 on 8 virtual cores

julia> 
~$ julia-1.8 -t 10
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.0-rc1 (2022-05-27)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> versioninfo()
Julia Version 1.8.0-rc1
Commit 6368fdc6565 (2022-05-27 18:33 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.3.0)
  CPU: 10 × Apple M1 Max
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 10 on 8 virtual cores

julia> 
~$ julia-1.8 -t 3 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.0-rc1 (2022-05-27)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> versioninfo(^C

julia> 
~$ julia-1.8     
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.0-rc1 (2022-05-27)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> 

[ngam]

Okay, I can now also reproduce with one process only, with julia-1.8 -t 8 foo.jl.

One is not guaranteed to get the performance cores by simply staying below the total number of performance cores fwiw. One is likely going to get assigned efficiency cores anyway if they are asking for all or most of them, even on a 8-to-2 split like here.

Stuff printed after ^C below.

^C
signal (2): Interrupt: 2
in expression starting at /Users/ngam/foo.jl:7
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
kevent at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
JL_UV_LOCK at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
ijl_exit_threaded_region at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
threading_run at ./threadingconstructs.jl:41
macro expansion at ./threadingconstructs.jl:89 [inlined]
foo at /Users/ngam/foo.jl:2 [inlined]
top-level scope at /Users/ngam/foo.jl:10
jl_toplevel_eval_flex at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_toplevel_eval_flex at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
ijl_toplevel_eval_in at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
eval at ./boot.jl:368 [inlined]
include_string at ./loading.jl:1281
ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
_include at ./loading.jl:1341
include at ./Base.jl:422
jfptr_include_28021 at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
exec_options at ./client.jl:303
_start at ./client.jl:522
jfptr__start_29833 at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
true_main at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_repl_entrypoint at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
unknown function (ip: 0x0)
Allocations: 113115497 (Pool: 113115426; Big: 71); GC: 197

[ssagl]

@ngam thanks a ton, finally somebody who can reproduce this issue! I already thought this problem is just specific to my machine. Concerning that you can only reproduce with two concurrent sessions: I think it happens extremely infrequently with just one session (but that's still how I even noted this problem). I can also confirm that with -t 3 or lower it doesn't happen on my machine as well.

[ngam]

Concerning that you can only reproduce with two concurrent sessions

I can now...

But I suspect the line of reasoning of vchuravy above is key to figuring this out.