Assert failed: System.Net.Http.Http3Connection.SendAsync in coreclr Linux_musl x64 Debug

[carlossanlop]

Found in this RC1 backport: #74145
@dotnet/ncl Please determine if this is a bug that requires a fix that meets the bar for an RC1 backport.

• Libraries Test Run release coreclr Linux_musl x64 Debug
• CI job: https://dev.azure.com/dnceng/public/_build/results?buildId=1951341&view=logs&j=73b7db51-6fbc-594d-25c9-15b2737b0bf6&t=d8035a69-bad8-5a32-b132-2fa64554b4a9&l=56
• Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-74145-merge-7f7fed172d694507b7/System.Net.Http.Functional.Tests/3/console.90b28f63.log?helixlogtype=result

Process terminated. Assertion failed.
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 192
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 183
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 324
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 795
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3410
   at System.Threading.Tasks.Task.FinishSlow(Boolean userDelegateExecute) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 1999
   at System.Threading.Tasks.Task.TrySetException(Object exceptionObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3324
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception, Task`1& taskField) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 472
   at System.Net.Quic.QuicConnection.OpenOutboundStreamAsync(QuicStreamType type, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicConnection.cs:line 376
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 183
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 324
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 795
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3410
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder.SetResult() in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncValueTaskMethodBuilder.cs:line 41
   at System.Net.Quic.QuicStream.DisposeAsync() in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 673
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 268
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 328
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 919
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 77

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Found in this RC1 backport: #74145
@dotnet/ncl Please determine if this is a bug that requires a fix that meets the bar for an RC1 backport.

• Libraries Test Run release coreclr Linux_musl x64 Debug
• CI job: https://dev.azure.com/dnceng/public/_build/results?buildId=1951341&view=logs&j=73b7db51-6fbc-594d-25c9-15b2737b0bf6&t=d8035a69-bad8-5a32-b132-2fa64554b4a9&l=56
• Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-74145-merge-7f7fed172d694507b7/System.Net.Http.Functional.Tests/3/console.90b28f63.log?helixlogtype=result

Process terminated. Assertion failed.
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 192
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 183
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 324
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 795
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3410
   at System.Threading.Tasks.Task.FinishSlow(Boolean userDelegateExecute) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 1999
   at System.Threading.Tasks.Task.TrySetException(Object exceptionObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3324
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception, Task`1& taskField) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 472
   at System.Net.Quic.QuicConnection.OpenOutboundStreamAsync(QuicStreamType type, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicConnection.cs:line 376
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 183
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 324
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 795
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3410
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder.SetResult() in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncValueTaskMethodBuilder.cs:line 41
   at System.Net.Quic.QuicStream.DisposeAsync() in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 673
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 268
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 328
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 919
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 77

Author:	carlossanlop
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[CarnaViire]

@rzikm is this the same assert you're looking at?

[rzikm]

yes

[rzikm]

Assert hit when running HttpClientHandlerTest_Http3.SendAsync_RequestRejected_ClientRetries

Investigation shows that the connection was closed locally while sending the request. However, Http3Connection._abortException is set to QuicException with OperationAborted. Since we never explicitly call Http3Connection.Abort(Exception) with OperationAborted, it is a little wonder how it got there. Will continue investigation on monday

[rzikm]

the Http3Connection.AbortException has following stack trace:

at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 157
   at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 221
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 550
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 572
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 601

Combined with the type of the exception (Operation aborted), I conclude that following happened in sequence:

• Connection was closed locally by application
• Some catch block in Http3Connection caught the QuicException
• Http3Connection.Abort was called with this exception (and it got saved to Http3Connection._abortException

The only way the QuicConnection gets closed is either via

• Http3Connection.Abort - this would set _abortException and prevent QuicException from getting there, so we can assume this wasn't the case
• Http3Connection.CheckForShutdown() - and this happens only when there is no active request going on...

[rzikm]

The Http3Connection.AbortException is populated via ProcessServerStream when server control stream throws. However, that happens after we check the AbortException in SendAsync (causing the assert).

[rzikm]

Looks like the _abortException threw me of the track into a blind alley... the _abortException may actually be null if we close the connection "gracefully" in response to a GOAWAY🤦, so the assert was invalid, it was just rare that we receive a GOAWAY when pending on OpenOutboundStream.

[carlossanlop]

Hit this in #74376
Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-74376-merge-60107d99ec574018b5/System.Net.Http.Functional.Tests/1/console.66be8dfb.log?helixlogtype=result