Roslyn crashes with NRE when building clr+libs locally

[janvorli]

I have started to hit a NRE in the Roslyn stuff after updating to the latest main and trying to build clr+libs. The issue reproduces for me quite reliably. I suspect is has started after the SDK that runtime repo uses was updated to preview 6. The call stack at the exception is

System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.CodeAnalysis.CSharp.Syntax.ParameterListSyntax.get_Parameters() in Microsoft.CodeAnalysis.CSharp.dll:token 0x6003cee+0x0
   at Microsoft.CodeAnalysis.CSharp.CSharpDeclarationComputer.GetParameterListInitializersAndAttributes(BaseParameterListSyntax parameterList) in Microsoft.CodeAnalysis.CSharp.dll:token 0x6002fd4+0x0
   at Microsoft.CodeAnalysis.CSharp.CSharpDeclarationComputer.ComputeDeclarations(SemanticModel model, ISymbol associatedSymbol, SyntaxNode node, Func`3 shouldSkip, Boolean getSymbol, ArrayBuilder`1 builder, Nullable`1 levelsToCompute, CancellationToken cancellationToken) in Microsoft.CodeAnalysis.CSharp.dll:token 0x6002fd2+0x0
   at Microsoft.CodeAnalysis.CSharp.CSharpDeclarationComputer.ComputeDeclarationsInNode(SemanticModel model, ISymbol associatedSymbol, SyntaxNode node, Boolean getSymbol, ArrayBuilder`1 builder, CancellationToken cancellationToken, Nullable`1 levelsToCompute) in Microsoft.CodeAnalysis.CSharp.dll:token 0x6002fcf+0x1c
   at Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver`1.ComputeDeclarationAnalysisData(ISymbol symbol, SyntaxReference declaration, SemanticModel semanticModel, AnalysisScope analysisScope, CancellationToken cancellationToken) in Microsoft.CodeAnalysis.dll:token 0x6002229+0x0
   at Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver`1.<>c__DisplayClass12_0.<TryExecuteDeclaringReferenceActions>b__0() in Microsoft.CodeAnalysis.dll:token 0x6003baa+0x0
   at Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver.CompilationData.GetOrComputeDeclarationAnalysisData(SyntaxReference declaration, Func`1 computeDeclarationAnalysisData, Boolean cacheAnalysisData) in Microsoft.CodeAnalysis.dll:token 0x6003b33+0x84
   at Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver`1.TryExecuteDeclaringReferenceActions(SyntaxReference decl, Int32 declarationIndex, SymbolDeclaredCompilationEvent symbolEvent, AnalysisScope analysisScope, AnalysisState analysisState, GroupedAnalyzerActions coreActions, GroupedAnalyzerActions additionalPerSymbolActions, Boolean shouldExecuteSyntaxNodeActions, Boolean shouldExecuteOperationActions, Boolean shouldExecuteCodeBlockActions, Boolean shouldExecuteOperationBlockActions, Boolean isInGeneratedCode, CancellationToken cancellationToken) in Microsoft.CodeAnalysis.dll:token 0x600222b+0x0
   at Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver`1.TryExecuteDeclaringReferenceActions(SymbolDeclaredCompilationEvent symbolEvent, AnalysisScope analysisScope, AnalysisState analysisState, Boolean isGeneratedCodeSymbol, IGroupedAnalyzerActions additionalPerSymbolActions, CancellationToken cancellationToken) in Microsoft.CodeAnalysis.dll:token 0x6002227+0x0
   at Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver.TryProcessSymbolDeclaredAsync(SymbolDeclaredCompilationEvent symbolEvent, AnalysisScope analysisScope, AnalysisState analysisState, CancellationToken cancellationToken) in Microsoft.CodeAnalysis.dll:token 0x60021f5+0x22d
   at Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver.TryProcessEventCoreAsync(CompilationEvent compilationEvent, AnalysisScope analysisScope, AnalysisState analysisState, CancellationToken cancellationToken) in Microsoft.CodeAnalysis.dll:token 0x60021f4+0x1bf
   at Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver.ProcessEventAsync(CompilationEvent e, AnalysisScope analysisScope, AnalysisState analysisState, CancellationToken cancellationToken) in Microsoft.CodeAnalysis.dll:token 0x60021f2+0x172
   at Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver.ProcessCompilationEventsCoreAsync(AnalysisScope analysisScope, AnalysisState analysisState, Boolean prePopulatedEventQueue, CancellationToken cancellationToken) in Microsoft.CodeAnalysis.dll:token 0x60021f1+0x19c

I was able to catch it with WinDbg and so far it seems like it may be a JIT issue. The failure location looks like this:

0000006f is a safepoint: 
0000006e +rbp(interior)
00007ffe`a674aeff 488bfd          mov     rdi,rbp
00007ffe`a674af02 488d742438      lea     rsi,[rsp+38h]
>>> 00007ffe`a674af07 48a5            movs    qword ptr [rdi],qword ptr [rsi]

The RDI is 0 here. As you can see, the RDI is loaded from RBP. The RBP is set at the beginning of the function to store RDX (the 2nd argument register) and never modified. And looking at the call site, I can see this:

00007ffe`a6e40092 33d2            xor     edx,edx
00007ffe`a6e40094 ff15f6e356f9    call    qword ptr [00007ffe`a03ae490] (Microsoft.CodeAnalysis.CSharp.Syntax.ParameterListSyntax.get_Parameters(), mdToken: 0000000006003CEE)

So the JIT explicitly passes in NULL.

Both the caller and the crashing callee are optimized tier 1 code.

[janvorli]

cc: @dotnet/jit-contrib
I am still looking into it, will try to get jitdump of the caller. I have already patched the runtime in the .dotnet folder to a checked build so that I can do interesting things.

[janvorli]

I was able to capture jitdump of the caller and created a gist with it: https://gist.github.com/janvorli/94c6fcafb18ab6ab1818654dfa13d7fd
Please note that the file contains both the tier 0 and tier 1 versions, the tier 1 at the end is the one with the issue - see xor rdx, rdx there.

[janvorli]

Update: I've just noticed that I have these set (remainder of a previous debugging) when the issue happens:

COMPlus_ReadyToRun             0
COMPlus_TC_QuickJitForLoops    1
COMPlus_TieredCompilation      1
COMPlus_TieredPGO              1

[janvorli]

I am hitting it when running this command line:

.\build.cmd clr+libs -c Release -rc Debug

[jakobbotsch]

Looks like things go wrong quite early in tier-1. The get_Parameters call needs a ret buffer, so we are actually passing null for the ret buffer in tier-1. Also, immediately after the return the ret buffer is boxed, which leads us to do an optimization where we pass the box directly as the ret buffer (from what I can tell). In tier-0 (where this optimization also happens, it seems?) that looks like:

Importing BB03 (PC=009) of 'Microsoft.CodeAnalysis.CSharp.CSharpDeclarationComputer:GetParameterListInitializersAndAttributes(Microsoft.CodeAnalysis.CSharp.Syntax.BaseParameterListSyntax):System.Collections.Generic.IEnumerable`1[[Microsoft.CodeAnalysis.SyntaxNode, Microsoft.CodeAnalysis, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]]'
    [ 0]   9 (0x009) ldarg.0
    [ 1]  10 (0x00a) callvirt 060046A0
In Compiler::impImportCall: opcode is callvirt, kind=4, callRetType is struct, structSize is 24

 ... marking [000005] in BB03 for class profile instrumentation

    [ 1]  15 (0x00f) box 1B000182
Compiler::impImportAndPushBox -- handling BOX(value class) via inline allocate/copy sequence

lvaGrabTemp returning 2 (V02 tmp1) called for Reusable Box Helper.


STMT00001 (IL 0x009...  ???)
               [000009] -A----------              *  ASG       ref   
               [000008] D------N----              +--*  LCL_VAR   ref    V02 tmp1         
               [000007] ------------              \--*  ALLOCOBJ  ref   
               [000006] H-----------                 \--*  CNS_INT(h) long   0x7ffebee4da28 class


STMT00002 (IL   ???...  ???)
               [000005] S-C-G-------              *  CALLV vt-ind void   Microsoft.CodeAnalysis.CSharp.Syntax.BaseParameterListSyntax.get_Parameters
               [000004] ------------ this in rcx  +--*  LCL_VAR   ref    V00 arg0         
               [000012] ------------ arg1         \--*  ADD       byref 
               [000010] ------------                 +--*  LCL_VAR   ref    V02 tmp1         
               [000011] ------------                 \--*  CNS_INT   long   8

but somehow, in tier-1 with PGO data, it seems like there is some interaction between guarded devirt and inlining which means we end up with things in the wrong order:

Importing BB03 (PC=009) of 'Microsoft.CodeAnalysis.CSharp.CSharpDeclarationComputer:GetParameterListInitializersAndAttributes(Microsoft.CodeAnalysis.CSharp.Syntax.BaseParameterListSyntax):System.Collections.Generic.IEnumerable`1[[Microsoft.CodeAnalysis.SyntaxNode, Microsoft.CodeAnalysis, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]]'
    [ 0]   9 (0x009) ldarg.0
    [ 1]  10 (0x00a) callvirt 060046A0
In Compiler::impImportCall: opcode is callvirt, kind=4, callRetType is struct, structSize is 24

impDevirtualizeCall: Trying to devirtualize virtual call:
    class for 'this' is Microsoft.CodeAnalysis.CSharp.Syntax.BaseParameterListSyntax (attrib 21000400)
    base method is Microsoft.CodeAnalysis.CSharp.Syntax.BaseParameterListSyntax::get_Parameters
    devirt to Microsoft.CodeAnalysis.CSharp.Syntax.BaseParameterListSyntax::get_Parameters -- inexact or not final
               [000005] --C-G-------              *  CALLV vt-ind struct Microsoft.CodeAnalysis.CSharp.Syntax.BaseParameterListSyntax.get_Parameters
               [000004] ------------ this in rcx  \--*  LCL_VAR   ref    V00 arg0         
    Class not final or exact, and method not final
Considering guarded devirtualization at IL offset 10 (0xa)
Likely class for 00007FFEBC651000 (Microsoft.CodeAnalysis.CSharp.Syntax.BaseParameterListSyntax) is 00007FFEBC651988 (Microsoft.CodeAnalysis.CSharp.Syntax.ParameterListSyntax) [likelihood:100 classes seen:1]
virtual call would invoke method get_Parameters
Marking call [000005] as guarded devirtualization candidate; will guess for class Microsoft.CodeAnalysis.CSharp.Syntax.ParameterListSyntax


STMT00001 (IL 0x009...  ???)
               [000005] &-C-G-------              *  CALLV vt-ind struct Microsoft.CodeAnalysis.CSharp.Syntax.BaseParameterListSyntax.get_Parameters (exactContextHnd=0x00007FFEBC651001)
               [000004] ------------ this in rcx  \--*  LCL_VAR   ref    V00 arg0         

    [ 1]  15 (0x00f) box 1B000182
Compiler::impImportAndPushBox -- handling BOX(value class) via inline allocate/copy sequence

lvaGrabTemp returning 2 (V02 tmp1) called for Single-def Box Helper.
Marking V02 as a single def local

lvaSetClass: setting class for V02 to (00007FFEBF707200) Microsoft.CodeAnalysis.SeparatedSyntaxList`1[[Microsoft.CodeAnalysis.CSharp.Syntax.ParameterSyntax, Microsoft.CodeAnalysis.CSharp, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]]  [exact]


STMT00002 (IL   ???...  ???)
               [000010] -A----------              *  ASG       ref   
               [000009] D------N----              +--*  LCL_VAR   ref    V02 tmp1         
               [000008] ------------              \--*  ALLOCOBJ  ref   
               [000007] H-----------                 \--*  CNS_INT(h) long   0x7ffebf707200 class


STMT00003 (IL   ???...  ???)
               [000006] --C---------              *  RET_EXPR  void  (inl return expr [000005])

I can continue looking tomorrow although @AndyAyersMS might just know what the problem is.

[janvorli]

@jakobbotsch Thank you! I have also verified that clearing the COMPlus_TieredPGO fixes the issue, so it is really PGO related.

[AndyAyersMS]

This sounds exactly like #53549, which is fixed in main via #56126.

[briansull]

Yes, I took a look at this as well.
And the initialization for
lvaGrabTemp returning 2 (V02 tmp1) called for Single-def Box Helper.
Comes after the use of V02 in the call.

[AndyAyersMS]

The box expansion has a bad interaction with inlining where the IR is arguably left out of order, normally it gets fixed up ok, whether the inline succeeds or fails.

But GDV interfereres and exposes the out of order IR.

We either need to allocate the box before the call, or expand the GDV at the point of the RET_EXPR, neither was a trivial fix, so I just disabled GDV for this case.

[janvorli]

It sounds like this issue can be closed then.

[JulieLeeMSFT]

closing.