tests crashing with memory / stack corruption

[vtjnash]

https://build.julialang.org/#/builders/55/builds/2685/steps/5/logs/stdio

Downloads                          (6) |        started at 2021-08-25T20:56:57.493
      From worker 6:	Internal error: encountered unexpected error in runtime:
      From worker 6:	StackOverflowError()
      From worker 6:	jfptr_MethodCallResult_7509.clone_1 at /buildworker/worker/tester_linux32/build/lib/julia/sys.so (unknown line)
      From worker 6:	_jl_invoke at /buildworker/worker/package_linux32/build/src/gf.c:2226 [inlined]
      From worker 6:	jl_apply_generic at /buildworker/worker/package_linux32/build/src/gf.c:2427
      From worker 6:	abstract_call_method at ./compiler/abstractinterpretation.jl:511
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:105
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	typeinf_edge at ./compiler/typeinfer.jl:823 [inlined]
      From worker 6:	abstract_call_method at ./compiler/abstractinterpretation.jl:507
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:105
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	typeinf_edge at ./compiler/typeinfer.jl:823 [inlined]
      From worker 6:	abstract_call_method at ./compiler/abstractinterpretation.jl:507
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:105
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	typeinf_edge at ./compiler/typeinfer.jl:823 [inlined]
      From worker 6:	abstract_call_method at ./compiler/abstractinterpretation.jl:507
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:105
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	typeinf_edge at ./compiler/typeinfer.jl:823 [inlined]
      From worker 6:	abstract_call_method at ./compiler/abstractinterpretation.jl:507
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:105
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	typeinf_ext at ./compiler/typeinfer.jl:909
      From worker 6:	typeinf_ext_toplevel at ./compiler/typeinfer.jl:942
      From worker 6:	typeinf_ext_toplevel at ./compiler/typeinfer.jl:938
      From worker 6:	jfptr_typeinf_ext_toplevel_12789.clone_1 at /buildworker/worker/tester_linux32/build/lib/julia/sys.so (unknown line)
      From worker 6:	_jl_invoke at /buildworker/worker/package_linux32/build/src/gf.c:2226 [inlined]
      From worker 6:	jl_apply_generic at /buildworker/worker/package_linux32/build/src/gf.c:2427
      From worker 6:	jl_apply at /buildworker/worker/package_linux32/build/src/julia.h:1771 [inlined]
      From worker 6:	jl_type_infer at /buildworker/worker/package_linux32/build/src/gf.c:295
      From worker 6:	jl_generate_fptr at /buildworker/worker/package_linux32/build/src/jitlayers.cpp:338
      From worker 6:	jl_compile_method_internal at /buildworker/worker/package_linux32/build/src/gf.c:1978
      From worker 6:	jl_compile_method_internal at /buildworker/worker/package_linux32/build/src/gf.c:2428 [inlined]
      From worker 6:	_jl_invoke at /buildworker/worker/package_linux32/build/src/gf.c:2237 [inlined]
      From worker 6:	jl_apply_generic at /buildworker/worker/package_linux32/build/src/gf.c:2427
      From worker 6:	#1 at ./experimental.jl:77
      From worker 6:	jfptr_YY.1_44291.clone_1 at /buildworker/worker/tester_linux32/build/lib/julia/sys.so (unknown line)
      From worker 6:	_jl_invoke at /buildworker/worker/package_linux32/build/src/gf.c:2226 [inlined]
      From worker 6:	jl_apply_generic at /buildworker/worker/package_linux32/build/src/gf.c:2427
      From worker 6:	jl_apply at /buildworker/worker/package_linux32/build/src/julia.h:1771 [inlined]
      From worker 6:	start_task at /buildworker/worker/package_linux32/build/src/task.c:880
      From worker 6:	Internal error: encountered unexpected error in runtime:
      From worker 6:	StackOverflowError()
      From worker 6:	jfptr_MethodCallResult_7509.clone_1 at /buildworker/worker/tester_linux32/build/lib/julia/sys.so (unknown line)
      From worker 6:	_jl_invoke at /buildworker/worker/package_linux32/build/src/gf.c:2226 [inlined]
      From worker 6:	jl_apply_generic at /buildworker/worker/package_linux32/build/src/gf.c:2427
      From worker 6:	abstract_call_method at ./compiler/abstractinterpretation.jl:511
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:105
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:556
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:113
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:556
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:113
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:556
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:113
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:556
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:113
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:556
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:113
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1877
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	typeinf_ext at ./compiler/typeinfer.jl:909
      From worker 6:	typeinf_ext_toplevel at ./compiler/typeinfer.jl:942
      From worker 6:	typeinf_ext_toplevel at ./compiler/typeinfer.jl:938
      From worker 6:	jfptr_typeinf_ext_toplevel_12789.clone_1 at /buildworker/worker/tester_linux32/build/lib/julia/sys.so (unknown line)
      From worker 6:	_jl_invoke at /buildworker/worker/package_linux32/build/src/gf.c:2226 [inlined]
      From worker 6:	jl_apply_generic at /buildworker/worker/package_linux32/build/src/gf.c:2427
      From worker 6:	jl_apply at /buildworker/worker/package_linux32/build/src/julia.h:1771 [inlined]
      From worker 6:	jl_type_infer at /buildworker/worker/package_linux32/build/src/gf.c:295
      From worker 6:	jl_generate_fptr at /buildworker/worker/package_linux32/build/src/jitlayers.cpp:338
      From worker 6:	jl_compile_method_internal at /buildworker/worker/package_linux32/build/src/gf.c:1978
      From worker 6:	jl_compile_method_internal at /buildworker/worker/package_linux32/build/src/gf.c:2428 [inlined]
      From worker 6:	_jl_invoke at /buildworker/worker/package_linux32/build/src/gf.c:2237 [inlined]
      From worker 6:	jl_apply_generic at /buildworker/worker/package_linux32/build/src/gf.c:2427
      From worker 6:	jl_apply at /buildworker/worker/package_linux32/build/src/julia.h:1771 [inlined]
      From worker 6:	start_task at /buildworker/worker/package_linux32/build/src/task.c:880
      From worker 6:	��� Error: curl_multi_socket_action: 8
      From worker 6:	��� @ Downloads.Curl /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:36
      From worker 6:	��� Error: curl_multi_socket_action: 8
      From worker 6:	��� @ Downloads.Curl /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:36
      From worker 6:	��� Error: curl_multi_socket_action: 8
      From worker 6:	��� @ Downloads.Curl /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:36
      From worker 6:	��� Error: curl_multi_socket_action: 8
      From worker 6:	��� @ Downloads.Curl /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:36
      From worker 6:	��� Error: curl_multi_socket_action: 8
      From worker 6:	��� @ Downloads.Curl 
...
      From worker 6:	
      From worker 6:	signal (11): Segmentation fault
      From worker 6:	in expression starting at /buildworker/worker/tester_linux32/build/share/julia/stdlib/v1.8/Downloads/test/runtests.jl:3
      From worker 6:	terminate called after throwing an instance of 'std::bad_alloc'
      From worker 6:	  what():  std::bad_alloc
      From worker 6:	
      From worker 6:	signal (6): Aborted
      From worker 6:	in expression starting at /buildworker/worker/tester_linux32/build/share/julia/stdlib/v1.8/Downloads/test/runtests.jl:3
Downloads                          (6) |         failed at 2021-08-25T20:57:49.740

[StefanKarpinski]

A possibility here is that the task is getting migrated to a different thread and that's causing issues, related: #110. This would not have happened previously, of course, since we couldn't migrate tasks.

[vtjnash]

I don't think we run the tests with threads

[DilumAluthge]

On Buildbot, we don't run the tests with threads.

On Buildkite, there is a single job that runs the tests with threads - this job has _mt in the name to indicate that it uses multithreading. All other Buildkite jobs do not run the tests with threads.

[StefanKarpinski]

Now that the implementation has been significantly refactored, I'm going to close this and we can open new issues for the problems that crop up with the new, threadsafe version.