mono_os_mutex_destroy: pthread_mutex_destroy failed with "Device or resource busy"

[jkotas]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=599402
Build error leg or test failing: System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest
Pull request: #99595

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "mono_os_mutex_destroy: pthread_mutex_destroy failed",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=599402
Error message validated: [mono_os_mutex_destroy: pthread_mutex_destroy failed]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 3/12/2024 3:23:03 PM UTC

Report

Build	Definition	Test	Pull Request
641780	dotnet/runtime	System.Threading.Tests.EtwTests.WaitHandleWaitEventTest	#100386
641646	dotnet/runtime	System.Threading.Tests.EtwTests.WaitHandleWaitEventTest	#100993
640958	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100765
640844	dotnet/runtime	System.Threading.Tests.EtwTests.WaitHandleWaitEventTest	#100951
640688	dotnet/runtime	System.Threading.Tests.EtwTests.WaitHandleWaitEventTest	#93014
640353	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100743
639824	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd	#100767
639593	dotnet/runtime	System.Threading.Tests.EtwTests.WaitHandleWaitEventTest	#100669
639437	dotnet/runtime	System.Threading.Tests.EtwTests.WaitHandleWaitEventTest	#100520
638524	dotnet/runtime	System.Threading.Tests.EtwTests.WaitHandleWaitEventTest	#100767
637807	dotnet/runtime	System.Threading.Tests.EtwTests.WaitHandleWaitEventTest	#100489
637015	dotnet/runtime	System.Threading.Tests.EtwTests.WaitHandleWaitEventTest	#100828
636603	dotnet/runtime	System.Threading.Tests.EtwTests.WaitHandleWaitEventTest	#100802
636443	dotnet/runtime	System.Threading.Tests.EtwTests.WaitHandleWaitEventTest	#100821
635603	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd	#100743
635054	dotnet/runtime	System.Threading.Tests.EtwTests.WaitHandleWaitEventTest	#100778
633229	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd	#100731
631865	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100676
631246	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#99659
631044	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100497
630711	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100650
630272	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd	#100280
629948	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100525
629121	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100489
629043	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100589
628975	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100141
628043	dotnet/runtime	System.Transactions.Tests.AsyncTransactionScopeTests.AsyncTSTest
627578	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100550
626716	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest
626182	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#92057
625897	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100379
625012	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#99908
624382	dotnet/runtime	System.Diagnostics.Tests.ProcessWaitingTests.WaitAsyncForPeerProcess
623236	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100176
623242	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100386
623223	dotnet/runtime	System.Threading.Tests.EtwTests.WaitHandleWaitEventTest	#100283
622469	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd	#100141
622371	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100362
621284	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100352
621302	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100266
620319	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#99433
619690	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#99596
619142	dotnet/runtime	System.Threading.Tests.EtwTests.WaitHandleWaitEventTest	#99840
618646	dotnet/runtime	System.Threading.Tests.EtwTests.WaitHandleWaitEventTest	#100227
618638	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100272
618562	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100004
618336	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100004
617941	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100265
617442	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100252
616081	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#99860
614850	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#97589
614368	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd	#100152
614460	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100156
613929	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd	#100134
614036	dotnet/runtime	System.Threading.Tests.EtwTests.WaitHandleWaitEventTest	#100146
614003	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#99760
613807	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100038
611976	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#100087
611769	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd	#100056
611393	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd	#99956
611360	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest
610653	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#99973
610384	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#96332
609332	dotnet/runtime	System.Threading.Tests.EtwTests.WaitHandleWaitEventTest	#99970
608721	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd	#99956
607317	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#99924
607174	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#99920
606180	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest
605861	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd	#97759
603435	dotnet/runtime	System.Threading.ThreadPools.Tests.ThreadPoolTests.ThreadPoolMinMaxThreadsEventTest	#99766
603223	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd	#99761

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
5	17	71

[jkotas]

This is same as #97565 . The fix does not seem to work.

[jkotas]

The eventpipe mutex is being destroyed here:

#17 0x0000ffff93701468 in mono_os_mutex_destroy (mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:97
#18 mono_coop_mutex_destroy (mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:45
#19 ep_rt_spin_lock_free (spin_lock=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/eventpipe/ep-rt-mono.h:1213
#20 ep_rt_mono_fini () at /__w/1/s/src/mono/mono/eventpipe/ep-rt-mono.c:892
#21 0x0000ffff935b4d68 in mini_cleanup (domain=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:5259
#22 0x0000ffff937814ac in ves_icall_System_Environment_Exit (result=42) at /__w/1/s/src/mono/mono/metadata/icall.c:6151

While eventpipe is still running on a different thread here:

#4  0x0000ffff93643a38 in mono_handle_native_crash (signal=0xffff934e04f3 "SIGSEGV", mctx=0xffff90255670, info=0xffff902559d0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2967
#5  0x0000ffff935b0774 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xffff902559d0, context=0xffff90255a50, debug_fault_addr=0x78) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3888
#6  <signal handler called>
#7  mono_atomic_cas_ptr (dest=0x78, exch=0x0, comp=0x0) at /__w/1/s/src/mono/mono/metadata/../../mono/utils/atomic.h:585
#8  mono_atomic_load_ptr (src=0x78) at /__w/1/s/src/mono/mono/metadata/../../mono/utils/atomic.h:783
#9  update_callback (location=0x78, new_=<optimized out>, counter=<optimized out>) at /__w/1/s/src/mono/mono/metadata/profiler.c:805
#10 mono_profiler_set_jit_begin_callback (handle=0x0, cb=0x0, cb@entry=0xffff90256c30) at /__w/1/s/src/native/public/./mono/metadata/profiler-events.h:47
#11 0x0000ffff93702300 in EventPipeEtwCallbackDotNETRuntime (source_id=<optimized out>, is_enabled=is_enabled@entry=0, level=level@entry=4 '\004', match_any_keywords=match_any_keywords@entry=65536, match_all_keywords=<optimized out>, filter_data=<optimized out>, callback_data=callback_data@entry=0x0) at /__w/1/s/src/mono/mono/eventpipe/ep-rt-mono-runtime-provider.c:4396
#12 0x0000ffff936f3524 in ep_rt_provider_invoke_callback (callback_func=0xffff9370226c <EventPipeEtwCallbackDotNETRuntime>, source_id=0x0, is_enabled=0, level=4 '\004', match_any_keywords=65536, match_all_keywords=0, filter_data=0x0, callback_data=0x0) at /__w/1/s/src/mono/mono/mini/../../mono/eventpipe/ep-rt-mono.h:520
#13 provider_invoke_callback (provider_callback_data=<optimized out>, provider_callback_data@entry=0xffff90256cf0) at /__w/1/s/src/native/eventpipe/ep-provider.c:419
#14 0x0000ffff936f3cdc in disable_helper (id=<optimized out>) at /__w/1/s/src/native/eventpipe/ep.c:665

[tommcdon]

This is same as #97565 . The fix does not seem to work.

@davmason please take a look