NullReferenceException inside ConcurrentDictionary.GrowTable

[jakobbotsch]

Pipeline: https://dev.azure.com/dnceng/public/_build/results?buildId=1764865&view=results
Job: Installer Build and Test coreclr Linux_arm Release

/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error : Error: Object reference not set to an instance of an object. [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error : System.NullReferenceException: Object reference not set to an instance of an object. [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at System.Collections.Concurrent.ConcurrentDictionary`2.GrowTable(Tables tables) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at System.Collections.Concurrent.ConcurrentDictionary`2.TryAddInternal(TKey key, Nullable`1 nullableHashcode, TValue value, Boolean updateIfExists, Boolean acquireLock, TValue& resultingValue) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.DependencyAnalysis.NodeCache`2.GetOrAdd(TKey key) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.DependencyAnalysis.NodeFactory.CompiledMethodNode(MethodDesc method) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.Compilation.RootingServiceProvider.AddCompilationRoot(MethodDesc method, Boolean rootMinimalDependencies, String reason) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.ReadyToRunRootProvider.RootMethods(TypeDesc type, String reason, IRootingServiceProvider rootProvider) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.ReadyToRunRootProvider.AddCompilationRoots(IRootingServiceProvider rootProvider) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.Compilation..ctor(DependencyAnalyzerBase`1 dependencyGraph, NodeFactory nodeFactory, IEnumerable`1 compilationRoots, ILProvider ilProvider, DevirtualizationManager devirtualizationManager, IEnumerable`1 modulesBeingInstrumented, Logger logger, InstructionSetSupport instructionSetSupport) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.ReadyToRunCodegenCompilation..ctor(DependencyAnalyzerBase`1 dependencyGraph, NodeFactory nodeFactory, IEnumerable`1 roots, ILProvider ilProvider, Logger logger, DevirtualizationManager devirtualizationManager, IEnumerable`1 inputFiles, String compositeRootPath, InstructionSetSupport instructionSetSupport, Boolean resilient, Boolean generateMapFile, Boolean generateMapCsvFile, Boolean generatePdbFile, Func`2 printReproInstructions, String pdbPath, Boolean generatePerfMapFile, String perfMapPath, Int32 perfMapFormatVersion, Boolean generateProfileFile, Int32 parallelism, ProfileDataManager profileData, ReadyToRunMethodLayoutAlgorithm methodLayoutAlgorithm, ReadyToRunFileLayoutAlgorithm fileLayoutAlgorithm, Int32 customPESectionAlignment, Boolean verifyTypeAndFieldLayout) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.ReadyToRunCodegenCompilationBuilder.ToCompilation() [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.Program.RunSingleCompilation(Dictionary`2 inFilePaths, InstructionSetSupport instructionSetSupport, String compositeRootPath, Dictionary`2 unrootedInputFilePaths, HashSet`1 versionBubbleModulesHash, CompilerTypeSystemContext typeSystemContext) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.Program.Run(String[] args) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(463,5): error :    at ILCompiler.Program.Main(String[] args) [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
/root/runtime/artifacts/bin/Crossgen2Tasks/Debug/net7.0/Microsoft.NET.CrossGen.targets(351,5): error NETSDK1096: Optimizing assemblies for performance failed. You can either exclude the failing assemblies from being optimized, or set the PublishReadyToRun property to false. [/root/runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]

@jkotas do you think this could also be a symptom of #68443?

[jkotas]

It can be a symptom of #68443. It is hard to tell with any level of confidence without a crash dump.

[jakobbotsch]

Another similar-looking-but-not-quite example in #69068:

    Generating native code
    Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
       at System.Collections.Concurrent.ConcurrentDictionary`2.TryGetValueInternal(TKey key, Int32 hashcode, TValue& value)
       at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
       at ILCompiler.DependencyAnalysis.NodeFactory.NodeCache`2.GetOrAdd(TKey key) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NodeFactory.cs:line 156
       at ILCompiler.DependencyAnalysis.NodeFactory.MethodMetadata(MethodDesc method) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NodeFactory.cs:line 998
       at ILCompiler.UsageBasedMetadataManager.GeneratedTypesAndCodeMetadataPolicy.GeneratesMetadata(MethodDesc methodDef) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/UsageBasedMetadataManager.cs:line 873
       at ILCompiler.Metadata.Transform`1.HandleQualifiedMethod(MethodDesc method) in /_/src/coreclr/tools/aot/ILCompiler.MetadataTransform/ILCompiler/Metadata/Transform.Method.cs:line 36
       at ILCompiler.MetadataManager.CreateStackTraceRecord(MetadataTransform transform, MethodDesc method) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/MetadataManager.cs:line 573

Pipeline run: https://dev.azure.com/dnceng/public/_build/results?buildId=1769350&view=results
Job name: Build windows x64 debug NativeAOT

[VincentBu]

Failed again in: runtime-coreclr outerloop 20220526.2

Failed test:

R2R-CG2 windows x64 Checked no_tiered_compilation @ Windows.10.Amd64.Open

- JIT\\opt\\virtualstubdispatch\\bigvtbl\\bigvtbl_cs_do\\bigvtbl_cs_do.cmd

Error message:

Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
at Internal.NativeFormat.VertexArray.VertexTree.Save(NativeWriter writer) in //src/coreclr/tools/Common/Internal/NativeFormat/NativeFormatWriter.cs:line 1806
at Internal.NativeFormat.NativeWriter.Save(Stream stream) in //src/coreclr/tools/Common/Internal/NativeFormat/NativeFormatWriter.cs:line 334
at ILCompiler.DependencyAnalysis.ReadyToRun.DebugInfoTableNode.GetData(NodeFactory factory, Boolean relocsOnly) in //src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/DebugInfoTableNode.cs:line 123
at ILCompiler.DependencyAnalysis.ReadyToRunObjectWriter.EmitPortableExecutable() in //src/coreclr/tools/aot/ILCompiler.ReadyToRun/CodeGen/ReadyToRunObjectWriter.cs:line 256
at ILCompiler.ReadyToRunCodegenCompilation.Compile(String outputFile) in //src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilation.cs:line 347
at ILCompiler.Program.RunSingleCompilation(Dictionary2 inFilePaths, InstructionSetSupport instructionSetSupport, String compositeRootPath, Dictionary2 unrootedInputFilePaths, HashSet`1 versionBubbleModulesHash, CompilerTypeSystemContext typeSystemContext) in //src/coreclr/tools/aot/crossgen2/Program.cs:line 769
at ILCompiler.Program.Run(String[] args) in //src/coreclr/tools/aot/crossgen2/Program.cs:line 545
at ILCompiler.Program.Main(String[] args) in //src/coreclr/tools/aot/crossgen2/Program.cs:line 1046

Return code:      1
Raw output file:      C:\h\w\B9CC0A01\w\C1A90AAD\uploads\Reports\JIT.opt\virtualstubdispatch\bigvtbl\bigvtbl_cs_do\bigvtbl_cs_do.output.txt
Raw output:
BEGIN EXECUTION
bigvtbl_cs_do.dll
1 file(s) copied.
20:36:39.30
Response file: C:\h\w\B9CC0A01\w\C1A90AAD\e\JIT\opt\virtualstubdispatch\bigvtbl\bigvtbl_cs_do\bigvtbl_cs_do.dll.rsp
C:\h\w\B9CC0A01\w\C1A90AAD\e\JIT\opt\virtualstubdispatch\bigvtbl\bigvtbl_cs_do\IL-CG2\bigvtbl_cs_do.dll
-o:C:\h\w\B9CC0A01\w\C1A90AAD\e\JIT\opt\virtualstubdispatch\bigvtbl\bigvtbl_cs_do\bigvtbl_cs_do.dll
--targetarch:x64
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\B9CC0A01\p\System..dll
-r:C:\h\w\B9CC0A01\p\Microsoft..dll
-r:C:\h\w\B9CC0A01\p\mscorlib.dll
-r:C:\h\w\B9CC0A01\p\netstandard.dll
-O
" "dotnet" "C:\h\w\B9CC0A01\p\crossgen2\crossgen2.dll" @"C:\h\w\B9CC0A01\w\C1A90AAD\e\JIT\opt\virtualstubdispatch\bigvtbl\bigvtbl_cs_do\bigvtbl_cs_do.dll.rsp"   -r:C:\h\w\B9CC0A01\w\C1A90AAD\e\JIT\opt\virtualstubdispatch\bigvtbl\bigvtbl_cs_do\IL-CG2*.dll"
Emitting R2R PE file: C:\h\w\B9CC0A01\w\C1A90AAD\e\JIT\opt\virtualstubdispatch\bigvtbl\bigvtbl_cs_do\bigvtbl_cs_do.dll
20:37:01.30
Crossgen2 failed with exitcode - -1073741819
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=C:\h\w\B9CC0A01\p
C:\h\w\B9CC0A01\w\C1A90AAD\e\JIT\opt\virtualstubdispatch\bigvtbl\bigvtbl_cs_do\bigvtbl_cs_do.cmd
Expected: True
Actual:   False


Stack trace
   at JIT_opt._virtualstubdispatch_bigvtbl_bigvtbl_cs_do_bigvtbl_cs_do_._virtualstubdispatch_bigvtbl_bigvtbl_cs_do_bigvtbl_cs_do_cmd()

[AaronRobinsonMSFT]

Failed in #70251

2022-06-14T06:19:40.6750905Z     Reflection -> D:\a\_work\1\s\artifacts\tests\coreclr\windows.x64.Debug\nativeaot\SmokeTests\Reflection\Reflection\Reflection.dll
2022-06-14T06:19:48.1234901Z        Creating library D:\a\_work\1\s\artifacts\tests\coreclr\windows.x64.Debug\nativeaot\SmokeTests\Exceptions\Exceptions\native\Exceptions.lib and object D:\a\_work\1\s\artifacts\tests\coreclr\windows.x64.Debug\nativeaot\SmokeTests\Exceptions\Exceptions\native\Exceptions.exp
2022-06-14T06:19:48.2128877Z     Optimizing assemblies for size may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink
2022-06-14T06:19:48.2982092Z     Generating native code
2022-06-14T06:19:49.0561844Z     Dataflow -> D:\a\_work\1\s\artifacts\tests\coreclr\windows.x64.Debug\nativeaot\SmokeTests\Dataflow\Dataflow\Dataflow.dll
2022-06-14T06:20:44.7213835Z     Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
2022-06-14T06:20:44.7215826Z        at System.Collections.Concurrent.ConcurrentDictionary`2.TryGetValueInternal(TKey key, Int32 hashcode, TValue& value)
2022-06-14T06:20:46.3259241Z        at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
2022-06-14T06:20:46.5183248Z        at ILCompiler.DependencyAnalysis.NodeFactory.NodeCache`2.GetOrAdd(TKey key) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NodeFactory.cs:line 156
2022-06-14T06:20:46.7880895Z        at ILCompiler.DependencyAnalysis.NodeFactory.MethodMetadata(MethodDesc method) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NodeFactory.cs:line 1009
2022-06-14T06:20:48.7366778Z        at ILCompiler.UsageBasedMetadataManager.GeneratedTypesAndCodeMetadataPolicy.GeneratesMetadata(MethodDesc methodDef) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/UsageBasedMetadataManager.cs:line 884
2022-06-14T06:20:49.7672564Z        at ILCompiler.Metadata.Transform`1.HandleQualifiedMethod(MethodDesc method) in /_/src/coreclr/tools/aot/ILCompiler.MetadataTransform/ILCompiler/Metadata/Transform.Method.cs:line 36
2022-06-14T06:20:49.7965896Z        at ILCompiler.MetadataManager.CreateStackTraceRecord(MetadataTransform transform, MethodDesc method) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/MetadataManager.cs:line 594
2022-06-14T06:20:49.8022336Z        at ILCompiler.GeneratingMetadataManager.ComputeMetadata[TPolicy](TPolicy policy, NodeFactory factory, Byte[]& metadataBlob, List`1& typeMappings, List`1& methodMappings, List`1& fieldMappings, List`1& stackTraceMapping) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/GeneratingMetadataManager.cs:line 88
2022-06-14T06:20:49.8063628Z        at ILCompiler.UsageBasedMetadataManager.ComputeMetadata(NodeFactory factory, Byte[]& metadataBlob, List`1& typeMappings, List`1& methodMappings, List`1& fieldMappings, List`1& stackTraceMapping) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/UsageBasedMetadataManager.cs:line 199
2022-06-14T06:20:49.8094842Z        at ILCompiler.MetadataManager.EnsureMetadataGenerated(NodeFactory factory) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/MetadataManager.cs:line 574
2022-06-14T06:20:49.8120382Z        at ILCompiler.MetadataManager.GetMetadataBlob(NodeFactory factory) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/MetadataManager.cs:line 651
2022-06-14T06:20:49.8153898Z        at ILCompiler.DependencyAnalysis.MetadataNode.GetData(NodeFactory factory, Boolean relocsOnly) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/MetadataNode.cs:line 45
2022-06-14T06:20:49.8178075Z        at ILCompiler.DependencyAnalysis.ObjectWriter.EmitObject(String objectFilePath, IReadOnlyCollection`1 nodes, NodeFactory factory, ObjectWritingOptions options, IObjectDumper dumper, Logger logger) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ObjectWriter.cs:line 1018
2022-06-14T06:20:49.8195831Z        at ILCompiler.RyuJitCompilation.CompileInternal(String outputFile, ObjectDumper dumper) in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilation.cs:line 103
2022-06-14T06:20:49.8212995Z        at ILCompiler.Compilation.ILCompiler.ICompilation.Compile(String outputFile, ObjectDumper dumper) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Compilation.cs:line 526
2022-06-14T06:20:49.8274259Z        at ILCompiler.Program.Run(String[] args) in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 893
2022-06-14T06:20:49.8360953Z        at ILCompiler.Program.Main(String[] args) in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 1076