New precompilation crashes on Julia 1.11-rc1

[MilesCranmer]

I'm seeing some precompilation crashes on Julia 1.11-rc1 when precompiling DynamicExpressions.jl with DispatchDoctor in-use on the package. (DispatchDoctor.jl is basically a package that calls promote_op on each function and uses that to flag type instabilities.)

Here is the traceback:

ERROR: The following 1 direct dependency failed to precompile:

DynamicExpressions 

Failed to precompile DynamicExpressions [a40a106e-89c9-4ca8-8020-a735e8728b6b] to "/Users/mcranmer/.julia/compiled/v1.11/DynamicExpressions/jl_cQE0v5".
[39250] signal 4: Illegal instruction: 4
in expression starting at /Users/mcranmer/PermaDocuments/SymbolicRegressionMonorepo/DynamicExpressions.jl/src/DynamicExpressions.jl:120
_eval_tree_array at /Users/mcranmer/PermaDocuments/DispatchDoctor.jl/src/stabilization.jl:301
macro expansion at /Users/mcranmer/PermaDocuments/SymbolicRegressionMonorepo/DynamicExpressions.jl/src/Evaluate.jl:92 [inlined]
#eval_tree_array#2 at /Users/mcranmer/PermaDocuments/DispatchDoctor.jl/src/stabilization.jl:306
eval_tree_array at /Users/mcranmer/PermaDocuments/DispatchDoctor.jl/src/stabilization.jl:301
#test_all_combinations#1 at /Users/mcranmer/PermaDocuments/SymbolicRegressionMonorepo/DynamicExpressions.jl/src/precompile.jl:7
test_all_combinations at /Users/mcranmer/PermaDocuments/SymbolicRegressionMonorepo/DynamicExpressions.jl/src/precompile.jl:22 [inlined]
macro expansion at /Users/mcranmer/PermaDocuments/SymbolicRegressionMonorepo/DynamicExpressions.jl/src/precompile.jl:168 [inlined]
macro expansion at /Users/mcranmer/.julia/packages/PrecompileTools/L8A3n/src/workloads.jl:78 [inlined]
macro expansion at /Users/mcranmer/PermaDocuments/SymbolicRegressionMonorepo/DynamicExpressions.jl/src/precompile.jl:153 [inlined]
macro expansion at /Users/mcranmer/.julia/packages/PrecompileTools/L8A3n/src/workloads.jl:140 [inlined]
#do_precompilation#2 at /Users/mcranmer/PermaDocuments/SymbolicRegressionMonorepo/DynamicExpressions.jl/src/precompile.jl:138
do_precompilation at /Users/mcranmer/PermaDocuments/SymbolicRegressionMonorepo/DynamicExpressions.jl/src/precompile.jl:161
unknown function (ip: 0x11570c053)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2156 [inlined]
do_call at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:126
eval_stmt_value at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:174
eval_body at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:663
jl_interpret_toplevel_thunk at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:821
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:943
jl_eval_module_expr at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:215 [inlined]
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:743
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:886
ijl_toplevel_eval at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:952 [inlined]
ijl_toplevel_eval_in at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:994
eval at ./boot.jl:429 [inlined]
include_string at ./loading.jl:2543
_include at ./loading.jl:2603
include at ./Base.jl:558 [inlined]
include_package_for_output at ./loading.jl:2721
jfptr_include_package_for_output_69600.1 at /Users/mcranmer/.julia/juliaup/julia-1.11.0-rc1+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2156 [inlined]
do_call at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:126
eval_stmt_value at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:174
eval_body at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:663
jl_interpret_toplevel_thunk at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/interpreter.c:821
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:943
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:886
ijl_toplevel_eval at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:952 [inlined]
ijl_toplevel_eval_in at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/toplevel.c:994
eval at ./boot.jl:429 [inlined]
include_string at ./loading.jl:2543
include_string at ./loading.jl:2553 [inlined]
exec_options at ./client.jl:316
_start at ./client.jl:526
jfptr__start_71098.1 at /Users/mcranmer/.julia/juliaup/julia-1.11.0-rc1+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/./julia.h:2156 [inlined]
true_main at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/jlapi.c:900
jl_repl_entrypoint at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-11/src/jlapi.c:1059
Allocations: 84690299 (Pool: 84689426; Big: 873); GC: 4

versioninfo:

julia> versioninfo()
Julia Version 1.11.0-rc1
Commit 3a35aec36d1 (2024-06-25 10:23 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 8 × Apple M1 Pro
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, apple-m1)
Threads: 6 default, 0 interactive, 3 GC (on 6 virtual cores)
Environment:
  JULIA_FORMATTER_SO = /Users/mcranmer/julia_formatter.so
  JULIA_NUM_THREADS = auto
  JULIA_OPTIMIZE = 3
  JULIA_EDITOR = code

I installed Julia with juliaup. To reproduce this issue, you can run the following code:

cd $(mktemp -d)
# Install package
julia +1.11 --startup-file=no --project=. -e 'using Pkg; pkg"add Preferences DynamicExpressions DispatchDoctor@v0.4.10"'
# Enable DispatchDoctor.jl
julia +1.11 --startup-file=no --project=. -e 'using Preferences; set_preferences!("DynamicExpressions", "instability_check" => "warn")'
# Precompile:
julia +1.11 --startup-file=no --project=. -e 'using Pkg; pkg"precompile"'

I can prevent this error with the following PR on DispatchDoctor.jl: MilesCranmer/DispatchDoctor.jl@094b165~...b223a4d. The PR basically amounts to changing some functions into @generated form:

- map_specializing_typeof(args...) = map(specializing_typeof, args)
+ map_specializing_typeof(args::Tuple) = map(specializing_typeof, args)
  
- _promote_op(f, S::Type...) = Base.promote_op(f, S...)
- _promote_op(f, S::Tuple) = _promote_op(f, S...)
+ function _promote_op(f, S::Vararg{Type})
+     if @generated
+         :(Base.promote_op(f, S...))
+     else
+         Base.promote_op(f, S...)
+     end
+ end

However, it doesn't seem like DispatchDoctor.jl or DynamicExpressions.jl is doing anything wrong, so I'm not sure what's going on. Both before and after seem to be valid Julia code. Also, the downside of that PR is it introduces a type instability in Zygote autodiff, and there doesn't seem to be a way around it that both prevents the segfault while also eliminating the type instability.

I don't understand the conditions for reproducing this, so this is so far my only example. When I make various tweaks to _promote_op within DispatchDoctor.jl, I seem to end up with different segfaults – one of which is the Unreachable reached bug.

cc @avik-pal

[giordano]

I don't see any segmentation fault in the error you shared.

[MilesCranmer]

Wasn't sure what the "Illegal instruction" is. Updated description to just be "Error".

[giordano]

It means that the processor was asked to execute instructions it doesn't support ("illegal"). Think, for example, of trying to execute avx512 instructions on on a avx/avx2 processor (maybe because you compiled the program on a different machine, with a larger instructions set than the current one): it'd have no clue of what you're talking about.

[MilesCranmer]

I see. So, guess it's a bug then?

I see it on macOS M1 and then also the GitHub actions with ubuntu-latest: https://github.com/MilesCranmer/SymbolicRegression.jl/actions/runs/9947224797/job/27479456770?pr=326#step:6:640. This one gets the Unreachable reached at 0x7fd831cd2d85 issue:

ERROR: The following 2 direct dependencies failed to precompile:

DynamicExpressions --code-coverage=@/home/runner/work/SymbolicRegression.jl/SymbolicRegression.jl --color=yes --check-bounds=yes --warn-overwrite=yes --depwarn=yes --inline=yes --startup-file=no --track-allocation=none 

Failed to precompile DynamicExpressions [a40a106e-89c9-4ca8-8020-a735e8728b6b] to "/home/runner/.julia/compiled/v1.11/DynamicExpressions/jl_0E9EhT".
Unreachable reached at 0x7fd831cd2d85

[4169] signal 4 (2): Illegal instruction
in expression starting at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/DynamicExpressions.jl:111
is_bad_array at /home/runner/.julia/packages/DispatchDoctor/eWFc7/src/stabilization.jl:301
##eval_tree_array_simulator#549#1 at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/Evaluate.jl:87 [inlined]
##eval_tree_array_simulator#549 at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/Evaluate.jl:66 [inlined]
#eval_tree_array#2 at /home/runner/.julia/packages/DispatchDoctor/eWFc7/src/stabilization.jl:306 [inlined]
eval_tree_array at /home/runner/.julia/packages/DispatchDoctor/eWFc7/src/stabilization.jl:301
unknown function (ip: 0x7fd831d0322d)
#test_all_combinations#1 at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/precompile.jl:7
test_all_combinations at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/precompile.jl:22 [inlined]
macro expansion at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/precompile.jl:169 [inlined]
macro expansion at /home/runner/.julia/packages/PrecompileTools/L8A3n/src/workloads.jl:78 [inlined]
macro expansion at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/precompile.jl:154 [inlined]
macro expansion at /home/runner/.julia/packages/PrecompileTools/L8A3n/src/workloads.jl:140 [inlined]
#do_precompilation#2 at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/precompile.jl:139
do_precompilation at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/precompile.jl:162
unknown function (ip: 0x7fd831ccfea2)
jl_apply at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/julia.h:2156 [inlined]
do_call at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:126
eval_value at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:663
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:821
jl_toplevel_eval_flex at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/toplevel.c:943
jl_eval_module_expr at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/toplevel.c:215 [inlined]
jl_toplevel_eval_flex at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/toplevel.c:743
jl_toplevel_eval_flex at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/toplevel.c:886
ijl_toplevel_eval_in at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/toplevel.c:994
eval at ./boot.jl:429 [inlined]
include_string at ./loading.jl:2543
_include at ./loading.jl:2603
include at ./Base.jl:558 [inlined]
include_package_for_output at ./loading.jl:2721
jfptr_include_package_for_output_69232.1 at /opt/hostedtoolcache/julia/1.11.0-rc1/x64/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/julia.h:2156 [inlined]
do_call at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:126
eval_value at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:663
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:821
jl_toplevel_eval_flex at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/toplevel.c:943
jl_toplevel_eval_flex at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/toplevel.c:886
ijl_toplevel_eval_in at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/toplevel.c:994
eval at ./boot.jl:429 [inlined]
include_string at ./loading.jl:2543
include_string at ./loading.jl:2553 [inlined]
exec_options at ./client.jl:316
_start at ./client.jl:526
jfptr__start_70709.1 at /opt/hostedtoolcache/julia/1.11.0-rc1/x64/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/julia.h:2156 [inlined]
true_main at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/jlapi.c:900
jl_repl_entrypoint at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/jlapi.c:1059
main at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/cli/loader_exe.c:58
unknown function (ip: 0x7fd851629d8f)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 84382407 (Pool: 84381692; Big: 715); GC: 32
SymbolicRegression --code-coverage=@/home/runner/work/SymbolicRegression.jl/SymbolicRegression.jl --color=yes --check-bounds=yes --warn-overwrite=yes --depwarn=yes --inline=yes --startup-file=no --track-allocation=none 

Failed to precompile SymbolicRegression [8254be44-1295-4e6a-a16d-4[660](https://github.com/MilesCranmer/SymbolicRegression.jl/actions/runs/9947224797/job/27479456770?pr=326#step:6:661)3ac705cb] to "/home/runner/.julia/compiled/v1.11/SymbolicRegression/jl_G4yS0y".
Unreachable reached at 0x7f2162ad2df5

[5165] signal 4 (2): Illegal instruction
in expression starting at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/DynamicExpressions.jl:111
is_bad_array at /home/runner/.julia/packages/DispatchDoctor/eWFc7/src/stabilization.jl:301
##eval_tree_array_simulator#549#1 at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/Evaluate.jl:87 [inlined]
##eval_tree_array_simulator#549 at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/Evaluate.jl:66 [inlined]
#eval_tree_array#2 at /home/runner/.julia/packages/DispatchDoctor/eWFc7/src/stabilization.jl:306 [inlined]
eval_tree_array at /home/runner/.julia/packages/DispatchDoctor/eWFc7/src/stabilization.jl:301
unknown function (ip: 0x7f2162b0322d)
#test_all_combinations#1 at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/precompile.jl:7
test_all_combinations at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/precompile.jl:22 [inlined]
macro expansion at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/precompile.jl:169 [inlined]
macro expansion at /home/runner/.julia/packages/PrecompileTools/L8A3n/src/workloads.jl:78 [inlined]
macro expansion at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/precompile.jl:154 [inlined]
macro expansion at /home/runner/.julia/packages/PrecompileTools/L8A3n/src/workloads.jl:140 [inlined]
#do_precompilation#2 at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/precompile.jl:139
do_precompilation at /home/runner/.julia/packages/DynamicExpressions/IF10i/src/precompile.jl:162
unknown function (ip: 0x7f2162acfea2)
jl_apply at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/julia.h:2156 [inlined]
do_call at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:126
eval_value at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:[663](https://github.com/MilesCranmer/SymbolicRegression.jl/actions/runs/9947224797/job/27479456770?pr=326#step:6:664)
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:821
jl_toplevel_eval_flex at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/toplevel.c:943
jl_eval_module_expr at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/toplevel.c:215 [inlined]
jl_toplevel_eval_flex at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/toplevel.c:743
jl_toplevel_eval_flex at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/toplevel.c:886
ijl_toplevel_eval_in at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/toplevel.c:994
eval at ./boot.jl:429 [inlined]
include_string at ./loading.jl:2543
_include at ./loading.jl:2603
include at ./Base.jl:558 [inlined]
include_package_for_output at ./loading.jl:2721
jfptr_include_package_for_output_69269.1 at /opt/hostedtoolcache/julia/1.11.0-rc1/x64/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/julia.h:2156 [inlined]
do_call at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:126
eval_value at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:663
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/interpreter.c:821
jl_toplevel_eval_flex at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/toplevel.c:943
jl_toplevel_eval_flex at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/toplevel.c:886
ijl_toplevel_eval_in at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/toplevel.c:994
eval at ./boot.jl:429 [inlined]
include_string at ./loading.jl:2543
include_string at ./loading.jl:2553 [inlined]
exec_options at ./client.jl:316
_start at ./client.jl:526
jfptr__start_70709.1 at /opt/hostedtoolcache/julia/1.11.0-rc1/x64/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/julia.h:2156 [inlined]
true_main at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/jlapi.c:900
jl_repl_entrypoint at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/src/jlapi.c:1059
main at /cache/build/builder-amdci5-6/julialang/julia-release-1-dot-11/cli/loader_exe.c:58
unknown function (ip: 0x7f2182429d8f)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 82059736 (Pool: 82059067; Big: [669](https://github.com/MilesCranmer/SymbolicRegression.jl/actions/runs/9947224797/job/27479456770?pr=326#step:6:670)); GC: 33
ERROR: LoadError: Failed to precompile DynamicExpressions [a40a106e-89c9-4ca8-8020-a735e8728b6b] to "/home/runner/.julia/compiled/v1.11/DynamicExpressions/jl_YBf5sc".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool; flags::Cmd, cacheflags::Base.CacheFlags, reasons::Dict{String, Int64})
    @ Base ./loading.jl:3002
  [3] (::Base.var"#1080#1081"{Base.PkgId})()
    @ Base ./loading.jl:2388
  [4] mkpidlock(f::Base.var"#1080#1081"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile /opt/hostedtoolcache/julia/1.11.0-rc1/x64/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:95
  [5] #mkpidlock#6
    @ /opt/hostedtoolcache/julia/1.11.0-rc1/x64/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:90 [inlined]
  [6] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile /opt/hostedtoolcache/julia/1.11.0-rc1/x64/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:116
  [7] #invokelatest#2
    @ ./essentials.jl:1045 [inlined]
  [8] invokelatest
    @ ./essentials.jl:1040 [inlined]
  [9] maybe_cachefile_lock(f::Base.var"#1080#1081"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:3525
 [10] maybe_cachefile_lock
    @ ./loading.jl:3522 [inlined]
 [11] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:2384
 [12] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2216
 [13] #invoke_in_world#3
    @ ./essentials.jl:1077 [inlined]
 [14] invoke_in_world
    @ ./essentials.jl:1074 [inlined]
 [15] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2207
 [16] macro expansion
    @ ./loading.jl:2146 [inlined]
 [17] macro expansion
    @ ./lock.jl:273 [inlined]
 [18] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2103
 [19] #invoke_in_world#3
    @ ./essentials.jl:1077 [inlined]
 [20] invoke_in_world
    @ ./essentials.jl:1074 [inlined]
 [21] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2096
 [22] include
    @ ./Base.jl:558 [inlined]
 [23] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
    @ Base ./loading.jl:2721
 [24] top-level scope
    @ stdin:4
in expression starting at /home/runner/work/SymbolicRegression.jl/SymbolicRegression.jl/src/SymbolicRegression.jl:1
in expression starting at stdin:4

[giordano]

Yes, it's definitely a bug, probably the compiler is emitting wrong instructions for the current ISA. There are a bunch of similar tickets: #53847, #53843, #53848, #53761, ...

[MilesCranmer]

Here's my git bisect log:

git bisect start
# status: waiting for both good and bad commits
# bad: [3a35aec36d13c3e651c97bac664da2e778d591ad] set VERSION to 1.11.0-rc1 (#54924)
git bisect bad 3a35aec36d13c3e651c97bac664da2e778d591ad
# status: waiting for good commit(s), bad commit known
# good: [48d4fd48430af58502699fdf3504b90589df3852] set VERSION to 1.10.4 (#54625)
git bisect good 48d4fd48430af58502699fdf3504b90589df3852
# good: [0ba6ec2d2282937a084d7e5e5a0b026dc953bb31] Restore link to list of packages in Base docs (#50353)
git bisect good 0ba6ec2d2282937a084d7e5e5a0b026dc953bb31
# skip: [e754f2036cbfc37ea24a33d02e86e41a9cf56af9] Add missing type annotation reported by JET (#52207)
git bisect skip e754f2036cbfc37ea24a33d02e86e41a9cf56af9
# skip: [959b474d0516df77a268d9f23ccda5d2ad32acdf] docs: update latest stable version (#52215)
git bisect skip 959b474d0516df77a268d9f23ccda5d2ad32acdf
# bad: [c5d7b87a35b5beaef9d4d3aa53c0a2686f3445b9] Fix variable name in scaling an `AbstractTriangular` with zero alpha (#52855)
git bisect bad c5d7b87a35b5beaef9d4d3aa53c0a2686f3445b9
# bad: [4115c725d25c19a86ce8d3e3a584f02d59a9a9ce] Create rand function for Base.KeySet and Base.ValueIterator{Dict} (#51608)
git bisect bad 4115c725d25c19a86ce8d3e3a584f02d59a9a9ce
# good: [8be469e275a455ca894fdc5fad8a80aafb359544] Separate foreign threads into a :foreign threadpool (#50912)
git bisect good 8be469e275a455ca894fdc5fad8a80aafb359544
# skip: [7d51502d7845246d6a231fdc4cf19451f42427e1] More missing constants from earlier libgit2 versions
git bisect skip 7d51502d7845246d6a231fdc4cf19451f42427e1
# skip: [4c3aaa2b34996708367f9d5e4472fb5a1062bf63] reflection: define `Base.generating_output` utility function (#51216)
git bisect skip 4c3aaa2b34996708367f9d5e4472fb5a1062bf63
# bad: [ca862df7bfc534d22d4d39d265d1f74d59c1ab77] fix `_tryonce_download_from_cache` (busybox.exe download error) (#51531)
git bisect bad ca862df7bfc534d22d4d39d265d1f74d59c1ab77
# skip: [5d82d8095042935be0eb044259098e0d7c695922] add tfuncs for `[and|or]_int` intrinsics (#51266)
git bisect skip 5d82d8095042935be0eb044259098e0d7c695922
# skip: [4e1c965b512967aaa20b77f37e1fe76548b1def7] Remove size(::StructuredMatrix, d) specializations (#51083)
git bisect skip 4e1c965b512967aaa20b77f37e1fe76548b1def7
# skip: [3fc4f6bb243cb623636f276cb143cf5c476bbc59] 🤖 [master] Bump the Downloads stdlib from f97c72f to 8a614d5 (#51246)
git bisect skip 3fc4f6bb243cb623636f276cb143cf5c476bbc59
# skip: [476572f749a035047d4d8e6e76ec5b701b85904e] makefile option to generate better code (#51105)
git bisect skip 476572f749a035047d4d8e6e76ec5b701b85904e
# skip: [8b3ffd8918e53d5241ad948e8500335848d3b602] cross-reference pathof and pkgdir in docstrings (#51298)
git bisect skip 8b3ffd8918e53d5241ad948e8500335848d3b602
# bad: [15f34aa649dbbb34e53ff6d16db15cd11ae4a887] [NFC] rng_split: some elaboration and clarification (#50680)
git bisect bad 15f34aa649dbbb34e53ff6d16db15cd11ae4a887
# skip: [f3d50b7de66b351dfdaa826fa529fefb75a829e1] Fix extended help hint to give full line to enter (#51193)
git bisect skip f3d50b7de66b351dfdaa826fa529fefb75a829e1
# skip: [dcb4060b58797cf64517a694fcab3ea16278cb87] docs: manual: point to `MutableArithmetics` in the Performance tips (#50987)
git bisect skip dcb4060b58797cf64517a694fcab3ea16278cb87
# skip: [70000ac7c3d5d5f21e42555cdf99e699a246f8ec] sysimg: Allow loading a system image that is already present in memory (#51121)
git bisect skip 70000ac7c3d5d5f21e42555cdf99e699a246f8ec
# skip: [7cadc6d70c0a3d2b2c20e50d4b3555475756f785] 🤖 [master] Bump the SHA stdlib from 2d1f84e to aaf2df6 (#51049)
git bisect skip 7cadc6d70c0a3d2b2c20e50d4b3555475756f785
# skip: [39a53168a824a9a223adc6642da31e7a26b6890a] optimize: fix `effect_free` refinement in post-opt dataflow analysis (#51185)
git bisect skip 39a53168a824a9a223adc6642da31e7a26b6890a
# skip: [dd0ce50f389981839d96969b279c3a11e0b4088e] Fix typo in command-line-interface.md (#51055)
git bisect skip dd0ce50f389981839d96969b279c3a11e0b4088e
# skip: [fbf73f44c000ee79d12b7bf1645f076b640fd10c] 🤖 [master] Bump the Pkg stdlib from 047734e4c to f570abd39 (#51186)
git bisect skip fbf73f44c000ee79d12b7bf1645f076b640fd10c
# skip: [b2dfa1db9e4d7b1cd499ba58943df17bc77fe1d8] Deprecate `permute!!` and `invpermute!!` (#51337)
git bisect skip b2dfa1db9e4d7b1cd499ba58943df17bc77fe1d8
# skip: [eab8d6b96b05f7e84103f66a902e4ee7ad395b48] Fix getfield codegen for tuple inputs and unknown symbol fields. (#51234)
git bisect skip eab8d6b96b05f7e84103f66a902e4ee7ad395b48
# skip: [a355403080167056d2af4ccee8eadfffd8fce97f] Annotate fieldnames for default cgparams [NFC]
git bisect skip a355403080167056d2af4ccee8eadfffd8fce97f
# skip: [354c36742eb1c2c4c5bfe454d6d4fe975565de96] Allow SparseArrays to catch `lu(::WrappedSparseMatrix)` (#51161)
git bisect skip 354c36742eb1c2c4c5bfe454d6d4fe975565de96
# bad: [e85f0a5a718f68e581b07eb60fd0d8203b0cd0da] complete false & true more generally as vals (#51326)
git bisect bad e85f0a5a718f68e581b07eb60fd0d8203b0cd0da
# skip: [91b8c9b99f05b99db8b259257adeb1997f8c4415] Add `JL_DLLIMPORT` to `small_typeof` declaration (#50892)
git bisect skip 91b8c9b99f05b99db8b259257adeb1997f8c4415
# skip: [27fa5de3f0e245cfff8c5cd1c850353742362cbf] Introduce cholesky and qr hooks for wrapped sparse matrices (#51220)
git bisect skip 27fa5de3f0e245cfff8c5cd1c850353742362cbf
# good: [74ce6cf070a2a04e836c3e5a2211228a3ac978ef] minor NFC in GC codebase (#50991)
git bisect good 74ce6cf070a2a04e836c3e5a2211228a3ac978ef
# skip: [3527213ccb1bfe0c48feab5da64d30cadbd4c526] simplify call to promote_eltype with repeated elements (#51135)
git bisect skip 3527213ccb1bfe0c48feab5da64d30cadbd4c526
# bad: [d51ad06f664b3439b4aee51b5cd5edd6b9d53c69] Avoid infinite loop when doing SIGTRAP in arm64-apple (#51284)
git bisect bad d51ad06f664b3439b4aee51b5cd5edd6b9d53c69

(Most of the skips are due to hanging precompilation.)

[MilesCranmer]

Only 100 revisions left in the git bisect log but I unfortunately need to run. The bisect log is above if someone wants to start where I left off. (My computer seems pretty slow at compilation unfortunately)

[gbaraldi]

Executing unreachable code triggers a trap which on most architectures just becomes a SIGILL so it might be this

[maleadt]

It's also a good idea to test an assertions build first.

[MilesCranmer]

Just tested assertions build; no extra info.

Is there a way I can see what Julia CI builds are successful for each commit? It seems like I have to skip a lot of commits which is making this bisecting take much longer than expected

[MilesCranmer]

I see a commit

Avoid infinite loop when doing SIGTRAP in arm64-apple

It seems like all of the commits before that I hit infinite precompilation... So not sure I will be able to bisect this further.