Hang during AddRef/Release calls from GC callbacks for IReferenceTracker support

[Sergio0694]

Description

Note

Somewhat related to #109538. I got this hang from a .NET 10 build with the fix for that issue already there.

Description

I hit another hang on Native AOT when testing the Microsoft Store:

Stack trace (click to expand):

ntdll.dll!ZwWaitForAlertByThreadId() Line 4059 Unknown
ntdll.dll!RtlAcquireSRWLockExclusive(_RTL_SRWLOCK * SRWLock) Line 1296  C
Windows.UI.Xaml.dll!DirectUI::ReferenceTrackerManager::ReferenceTrackingStarted() Line 512   C++
<MICROSOFT_STORE>.exe!S_P_CoreLib_System_Runtime_InteropServices_TrackerObjectManager__BeginReferenceTracking() Line 107   Unknown
<MICROSOFT_STORE>.exe!S_P_CoreLib_System_Runtime_InteropServices_TrackerObjectManager__GCStartCollection() Line 165   Unknown
<MICROSOFT_STORE>.exe!RestrictedCallouts::InvokeGcCallouts(GcRestrictedCalloutKind eKind, unsigned int uiCondemnedGeneration) Line 195      C++
<MICROSOFT_STORE>.exe!WKS::gc_heap::garbage_collect(int n) Line 24362   C++
> <MICROSOFT_STORE>.exe!WKS::GCHeap::GarbageCollectGeneration(unsigned int gen, gc_reason reason) Line 51305 C++
[Inline Frame] <MICROSOFT_STORE>.exe!VolatileStore(int *) Line 261      C++
[Inline Frame] <MICROSOFT_STORE>.exe!Volatile<int>::Store(const int &) Line 349   C++
[Inline Frame] <MICROSOFT_STORE>.exe!Volatile<int>::operator=(int) Line 388  C++
[Inline Frame] <MICROSOFT_STORE>.exe!WKS::leave_spin_lock(WKS::GCDebugSpinLock *) Line 1704    C++
<MICROSOFT_STORE>.exe!WKS::gc_heap::trigger_gc_for_alloc(int gen_number, gc_reason gr, WKS::GCDebugSpinLock * msl, bool loh_p, WKS::msl_take_state take_state) Line 19008 C++
[Inline Frame] <MICROSOFT_STORE>.exe!WKS::gc_heap::try_allocate_more_space(alloc_context *) Line 19133   C++
<MICROSOFT_STORE>.exe!WKS::gc_heap::allocate_more_space(alloc_context * acontext, unsigned __int64 size, unsigned int flags, int alloc_generation_number) Line 19633      C++
[Inline Frame] <MICROSOFT_STORE>.exe!WKS::gc_heap::allocate_uoh_object(unsigned __int64) Line 45648   C++
<MICROSOFT_STORE>.exe!WKS::GCHeap::Alloc(gc_alloc_context * context, unsigned __int64 size, unsigned int flags) Line 50185     C++
<MICROSOFT_STORE>.exe!GcAllocInternal(MethodTable * pEEType, unsigned int uFlags, unsigned __int64 numElements, Thread * pThread) Line 606     C++
<MICROSOFT_STORE>.exe!RhAllocateNewArray(MethodTable * pArrayEEType, unsigned int numElements, unsigned int flags, Array * * pResult) Line 694 C++
<MICROSOFT_STORE>.exe!S_P_CoreLib_System_GC___AllocateUninitializedArray_g__AllocateNewUninitializedArray_52_0<UInt8>() Line 827   Unknown
<MICROSOFT_STORE>.exe!S_P_CoreLib_System_Buffers_SharedArrayPool_1<UInt8>__Rent() Line 110   Unknown
<MICROSOFT_STORE>.exe!CommunityToolkit_HighPerformance_CommunityToolkit_HighPerformance_ArrayPoolExtensions__Resize<UInt8>() Line 47    Unknown

Dump shared internally, opening this just for tracking.

From @jkotas:

It turns out that the AddRef/Release implementations in regular CoreCLR are in C/C++ and do not have GC transition in them. ProjectN has AddRef implemented in C/C++ as well with a long comment explaining the reason [...]. I think the fix should be to move the NAOT AddRef implementations (and potentially other methods nearby) to C/C++ to match regular CoreCLR and ProjectN to avoid the GC transitions that lead to deadlock."

cc. @AaronRobinsonMSFT @jkoritzinsky @manodasanW @VSadov @tommcdon @MichalStrehovsky

[MichalPetryka]

Why would GC transitions cause deadlocking here? And wouldn't just changing the APIs here leave other APIs that do GC transitions broken?

[jkotas]

The native Xaml engine takes SRWLock around AddRef calls. It is the same lock that is involved in the deadlock above. You can model a GC as a lock for this analysis. In fact, there is ThreadStore lock that is taken for the duration of the GC.

The GC transition in AddRef call leads to a classic A-B B-A deadlock:

• The thread above: GC -> SRWLock
• Some other thread: SRWLock -> AddRef -> GC

https://github.com/dotnet/corert/blob/db8fd94184c69e9e9307641f4ed860e388e55aef/src/System.Private.Interop/src/Shared/StandardInterfaces.cs#L121-L131 comment from the old .NET Native WinRT interop implementation talks about this.