[QUIC] Intermittent hangs in CI #55642
Description
Sporadic hangs of System.Net.Quic.Functional.Tests.
Since yesterday we're seeing that sometime the work item fails with "timeout" meaning a test hang and subsequent kill by the helix agent.
The potential culprit that might have caused this is #55505. Firstly, because the time when it started occurring correlates with the time of its merge. Secondly, the changes come from #52929 and there's been multiple test hangs historically.
Unique tests picked from the console logs:
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:14:04
[Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:14:00
[Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.LargeDataSentAndReceived', Elapsed: 00:14:04
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_MessagesSmallerThanReadBuffer_Success', Elapsed: 00:14:05
[Long Running Test] 'System.Net.Quic.Tests.MsQuicTests.ByteMixingOrNativeAVE_MinimalFailingTest', Elapsed: 00:14:33
Kusto queries to get the data
Failing work items (9 records):
WorkItems
| where FriendlyName == "System.Net.Quic.Functional.Tests" and Status != "Pass" and ExitCode == -3
| join kind=inner Jobs on JobId
| where DockerTag == "mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408"
| order by Queued descSuccess for comparison (465 total, 135 since the first failure):
WorkItems
| where FriendlyName == "System.Net.Quic.Functional.Tests" and Status == "Pass"
| join kind=inner Jobs on JobId
| where DockerTag == "mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408"
| order by Queued desccc: @CarnaViire @karelz
| Build | Kind | Start Time |
|---|---|---|
| 1236570 | PR 55594 | 2021-14-07 |