System.Net.Requests.Tests failing with a crash

[runfoapp]

Runfo Tracking Issue: System.Net.Requests.Tests failing with a crash

Build	Definition	Kind	Run Name	Console	Core Dump	Test Results	Run Client
1238781	runtime	Rolling	net6.0-Linux-Release-x64-mono_interpreter_release-Debian.9.Amd64.Open	console.log			runclient.py
1238781	runtime	Rolling	net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440	console.log			runclient.py
1238781	runtime	Rolling	net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e	console.log			runclient.py
1238781	runtime	Rolling	net6.0-Linux-Release-arm64-CoreCLR_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-helix-arm64v8-20210106155927-56c6673	console.log			runclient.py
1236633	runtime	PR 55613	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1236304	runtime	PR 54580	net6.0-windows-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open	console.log	core dump		runclient.py
1236304	runtime	PR 54580	net6.0-Linux-Debug-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1236278	runtime	PR 55607	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1236197	runtime	PR 55531	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1236179	runtime	Rolling	net6.0-Linux-Release-x64-Mono_release-SLES.12.Amd64.Open	console.log			runclient.py
1236179	runtime	Rolling	net6.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open	console.log			runclient.py
1236005	runtime	PR 55384	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1235600	runtime	PR 55592	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1235397	runtime	PR 55586	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1235376	runtime	PR 55564	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1235349	runtime	Rolling	net6.0-Linux-Release-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1235034	runtime	PR 55565	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1234989	runtime	PR 55430	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1234982	runtime	PR 55327	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1234728	runtime	PR 55258	net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440	console.log	core dump		runclient.py
1234594	runtime	Rolling	net6.0-Linux-Release-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1234580	runtime	PR 55398	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1234437	runtime	Rolling	net6.0-Linux-Release-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1234093	runtime	PR 55425	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1234093	runtime	PR 55425	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1234041	runtime	PR 55499	net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e	console.log	core dump		runclient.py
1234024	runtime	PR 55458	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1233610	runtime	PR 55533	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1233354	runtime	PR 55480	net6.0-Linux-Debug-x64-CoreCLR_release-SLES.15.Amd64.Open	console.log	core dump		runclient.py
1233170	runtime	PR 55369	net6.0-Linux-Debug-x64-CoreCLR_release-(Centos.8.Amd64.Open)Ubuntu.1604.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:centos-8-helix-20201229003624-c1bf759	console.log	core dump		runclient.py
1233170	runtime	PR 55369	net6.0-Linux-Debug-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1233089	runtime	PR 55395	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1233024	runtime	PR 55520	net6.0-Linux-Debug-x64-CoreCLR_release-(Debian.10.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-10-helix-amd64-bfcd90a-20200121150006	console.log	core dump		runclient.py
1232915	runtime	PR 55430	net6.0-Linux-Debug-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1232861	runtime	Rolling	net6.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1232822	runtime	PR 55338	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1232777	runtime	PR 55513	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1232777	runtime	PR 55513	net6.0-Linux-Debug-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1232718	runtime	PR 55395	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1232607	runtime	PR 55504	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1232607	runtime	PR 55504	net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440	console.log	core dump		runclient.py
1232415	runtime	PR 55500	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1232399	runtime	PR 55499	net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1232381	runtime	PR 55460	net6.0-Linux-Debug-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1232366	runtime	PR 55478	net6.0-Linux-Debug-x64-CoreCLR_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408	console.log	core dump		runclient.py
1232365	runtime	Rolling	net6.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1232171	runtime	PR 55460	net6.0-Linux-Debug-x64-CoreCLR_release-SLES.15.Amd64.Open	console.log	core dump		runclient.py
1232164	runtime	PR 51653	net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440	console.log	core dump		runclient.py
1232111	runtime	Rolling	net6.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1231886	runtime	PR 55467	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open	console.log			runclient.py
1231886	runtime	PR 55467	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1014.Amd64.Open	console.log	core dump		runclient.py
1231886	runtime	PR 55467	net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1231886	runtime	PR 55467	net6.0-windows-Debug-x64-CoreCLR_release-(Windows.Nano.1809.Amd64.Open)windows.10.amd64.serverrs5.open@mcr.microsoft.com/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504	console.log			runclient.py
1231886	runtime	PR 55467	net6.0-windows-Debug-x64-CoreCLR_release-Windows.81.Amd64.Open	console.log	core dump		runclient.py
1231886	runtime	PR 55467	net6.0-windows-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open	console.log	core dump		runclient.py
1231434	runtime	PR 55467	net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open	console.log			runclient.py
1231434	runtime	PR 55467	net6.0-windows-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open	console.log	core dump		runclient.py
1231434	runtime	PR 55467	net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e	console.log	core dump		runclient.py
1231434	runtime	PR 55467	net6.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.312.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59	console.log	core dump		runclient.py
1231434	runtime	PR 55467	net6.0-Linux-Debug-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1230534	runtime	PR 54954	net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440	console.log	core dump		runclient.py
1228008	runtime	PR 54954	net6.0-windows-Release-x86-CoreCLR_checked-Windows.10.Amd64.Open	console.log	core dump		runclient.py
1228008	runtime	PR 54954	net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e	console.log	core dump		runclient.py
1228008	runtime	PR 54954	net6.0-Linux-Release-arm-CoreCLR_checked-(Alpine.313.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.13-helix-arm32v7-20210414141857-1ea6b0a	console.log	core dump		runclient.py
1228008	runtime	PR 54954	net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440	console.log	core dump		runclient.py
1228008	runtime	PR 54954	net6.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.312.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59	console.log	core dump		runclient.py
1228008	runtime	PR 54954	net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open	console.log	core dump		runclient.py
1227837	runtime	PR 55372	net6.0-windows-Release-x86-CoreCLR_checked-Windows.10.Amd64.Open	console.log	core dump		runclient.py
1227837	runtime	PR 55372	net6.0-windows-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open	console.log	core dump		runclient.py
1227837	runtime	PR 55372	net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e	console.log	core dump		runclient.py
1227837	runtime	PR 55372	net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440	console.log	core dump		runclient.py
1227837	runtime	PR 55372	net6.0-Linux-Release-arm-CoreCLR_checked-(Alpine.313.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.13-helix-arm32v7-20210414141857-1ea6b0a	console.log	core dump		runclient.py
1227837	runtime	PR 55372	net6.0-Linux-Debug-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open	console.log	core dump		runclient.py
1227837	runtime	PR 55372	net6.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.312.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59	console.log	core dump		runclient.py
1227631	runtime	PR 55367	net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open	console.log
1227631	runtime	PR 55367	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open	console.log
1227625	runtime	PR 55346	net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open	console.log
1227625	runtime	PR 55346	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open	console.log
1227555	runtime	PR 55279	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open	console.log
1227555	runtime	PR 55279	net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open	console.log
1227539	runtime	PR 55362	net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open	console.log
1227539	runtime	PR 55362	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open	console.log
1227470	runtime	PR 55295	net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open	console.log
1227470	runtime	PR 55295	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open	console.log
1227447	runtime	PR 55150	net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open	console.log
1227447	runtime	PR 55150	net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open	console.log

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
1	51	51

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Runfo Creating Tracking Issue (data being generated)

Author:	runfoapp[bot]
Assignees:	-
Labels:	area-System.Net, untriaged
Milestone:	-

[ManickaP]

There's several problems combined here.

1. Infra issue - some of the console logs without a dump

Tests:

=== TEST EXECUTION SUMMARY ===
   System.Net.Requests.Tests  Total: 1109, Errors: 0, Failed: 0, Skipped: 6, Time: 48.794s
/root/helix/work/workitem
----- end Thu Jul 15 00:15:40 UTC 2021 ----- exit code 0 ----------------------------------------------------------
exit code 0 means Exited Successfully
+ export _commandExitCode=0

Infra after the tests passing:

2021-07-15 00:18:53,404: INFO: 548316835384: packing_test_reporter(34): report_results: Packed 317813 bytes
Traceback (most recent call last):
  File "/root/helix/work/correlation/reporter/run.py", line 73, in <module>
    main()
  File "/root/helix/work/correlation/reporter/run.py", line 69, in main
    if os.path.isfile(packed_file):
  File "/usr/lib/python3.8/genericpath.py", line 30, in isfile
    st = os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not method
+ exit 1
+ export '_commandExitCode=1'
+ chmod -R 777 /home/helixbot/dotnetbuild/dumps
+ exit 1

[END EXECUTION]
Exit Code:1

cc: @premun

2. Mono crashes - with dump and runtime is mono

Console.log

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================
...

cc: @marek-safar

3. CoreCLR crashes - #54580 draft pr

Different errors, looks like work in progress:

----- end Wed 07/14/2021  1:03:58.23 ----- exit code -1073740791 ----------------------------------------------------------

----- end Wed Jul 14 01:49:23 UTC 2021 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.

cc: @jkoritzinsky

4. Seg fault in FTP tests - majority of CoreCLR with a dump

Two errors, console.log:

./RunTests.sh: line 162:    24 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem
----- end Mon Jul 12 19:32:39 UTC 2021 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.

Assert failure(PID 59 [0x0000003b], Thread: 139 [0x008b]): !"Heap contamination detected! HeapFree was called on a heap other than the one that memory was allocated from.\n" "Possible cause: you used new (executable) to allocate the memory, but didn't use DeleteExecutable() to free it."
    File: /__w/1/s/src/coreclr/utilcode/clrhost_nodependencies.cpp Line: 296
    Image: /root/helix/work/correlation/dotnet

./RunTests.sh: line 162:    59 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE

This is the first that looks like it might be System.Net problem. Or it could have been caused by #54954, since this PR has higher occurrence of this error.
cc: @karelz @janvorli

5. Only on PR #55467

Looks like work in progress, console.log:

Unhandled exception. System.IO.FileLoadException: Could not load file or assembly 'xunit.runner.utility.netcoreapp10, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c'. Uncaught exception during type initialization. (0x80131534)
File name: 'xunit.runner.utility.netcoreapp10, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c'
 ---> System.TypeInitializationException: The type initializer for 'Xunit.DependencyContextAssemblyCache' threw an exception.
 ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Text.RegularExpressions.RegexParser.AssignNameSlots() in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexParser.cs:line 1957
   at System.Text.RegularExpressions.RegexParser.CountCaptures() in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexParser.cs:line 1907
   at System.Text.RegularExpressions.RegexParser.Parse(String pattern, RegexOptions options, CultureInfo culture) in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexParser.cs:line 88
   at System.Text.RegularExpressions.Regex.Init(String pattern, RegexOptions options, TimeSpan matchTimeout, CultureInfo culture) in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/Regex.cs:line 107
   at System.Text.RegularExpressions.Regex..ctor(String pattern, CultureInfo culture) in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/Regex.cs:line 68
   at System.Text.RegularExpressions.Regex..ctor(String pattern) in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/Regex.cs:line 46
   at Xunit.DependencyContextAssemblyCache..cctor() in /_/src/Microsoft.DotNet.XUnitConsoleRunner/src/common/AssemblyResolution/DependencyContextAssemblyCache.cs:line 20
   --- End of inner exception stack trace ---
   at Xunit.DependencyContextAssemblyCache.ResolveManagedAssembly(String assemblyName, Func`2 managedAssemblyLoader) in /_/src/Microsoft.DotNet.XUnitConsoleRunner/src/common/AssemblyResolution/DependencyContextAssemblyCache.cs:line 210
   at Xunit.DependencyContextAssemblyCache.LoadManagedDll(String assemblyName, Func`2 managedAssemblyLoader) in /_/src/Microsoft.DotNet.XUnitConsoleRunner/src/common/AssemblyResolution/DependencyContextAssemblyCache.cs:line 159
   at Xunit.AssemblyHelper.OnResolving(AssemblyLoadContext context, AssemblyName name) in /_/src/Microsoft.DotNet.XUnitConsoleRunner/src/common/AssemblyResolution/AssemblyHelper.cs:line 83
   at System.Runtime.Loader.AssemblyLoadContext.GetFirstResolvedAssemblyFromResolvingEvent(AssemblyName assemblyName) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/Loader/AssemblyLoadContext.cs:line 625
   at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(AssemblyName assemblyName) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/Loader/AssemblyLoadContext.cs:line 693
   at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr gchManagedAssemblyLoadContext, AssemblyName assemblyName) in /_/src/coreclr/System.Private.CoreLib/src/System/Runtime/Loader/AssemblyLoadContext.CoreCLR.cs:line 134
   at Xunit.ConsoleClient.ConsoleRunner..ctor(Object consoleLock)
   at Xunit.ConsoleClient.Program.Main(String[] args) in /_/src/Microsoft.DotNet.XUnitConsoleRunner/src/Program.cs:line 16

cc: @eiriktsarpalis

6. Another CoreCLR crash, only #55372 PR

Happened only on this PR, so seems like fixed before it was merged, console.log:

Assert failure(PID 23 [0x00000017], Thread: 33 [0x0021]): At least one interface has to implement the method, otherwise there's a bug in JIT/verification.
    File: /__w/1/s/src/coreclr/vm/methodtable.cpp Line: 9578
    Image: /root/helix/work/correlation/dotnet

./RunTests.sh: line 162:    23 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE

7. The last batch without a dump

CI timeout, console.log:

If you’re reading this, that means the Helix work item you’re trying to find the logs for has dead-lettered.

Looks like vast majority of problems is not networking related.

[janvorli]

I have fixed the issue 4 from this list yesterday, the fix is already in.

[janvorli]

I mean, the 2nd one in the issue 4.

[karelz]

@jaredpar this seems to be mix of many issues. We will take out the one crash that may be Networking related.
How do you want to track the rest?

Also, it seems that Runfo should not mix up the crashes as it did, but perhaps that is too much to ask from a simple tool?

[jaredpar]

Also, it seems that Runfo should not mix up the crashes as it did, but perhaps that is too much to ask from a simple tool?

The tool right now only has indexes over certain strings: test names and issues in the timeline. Can design queries for pretty much anything along those lines.

The detail here to differentiate is in the console logs. That doesn't have any indexes yet. It's possible to do just havent' had time to do so.

[karelz]

Closing this mega-issue in favor of folks tracking down sub-problems as they are tagged above ...