CreateXHarnessAppleWorkItems task failed unexpectedly when building iOS arm64 AllSubsets_Mono

[joperezr]

Rolling Build failed this morning on the Build iOS arm64 Release AllSubsets_Mono job at the Send to Helix step, with a problem when trying to use the tool microsoft.dotnet.xharness.cli to run the CreateXHarnessAppleWorkItems task. The issue seems to be just a failure in the SSL handshake communication. To view more details, you can get to the log of the failing build here.

For convenience, I'm pasting the stack trace of the failure below:

##[error].packages/microsoft.dotnet.helix.sdk/7.0.0-beta.22171.2/tools/xharness-runner/XHarnessRunner.targets(122,5): error MSB4018: The "CreateXHarnessAppleWorkItems" task failed unexpectedly.
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: RevocationStatusUnknown
   at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.DotNet.Helix.Sdk.ProvisioningProfileProvider.<>c__DisplayClass14_0.<<DownloadProvisioningProfile>b__0>d.MoveNext() in /_/src/Microsoft.DotNet.Helix/Sdk/ProvisioningProfileProvider.cs:line 190
--- End of stack trace from previous location ---
   at Microsoft.Arcade.Common.Helpers.<>c__DisplayClass7_0.<DirectoryMutexExec>b__0() in /_/src/Common/Microsoft.Arcade.Common/Helpers.cs:line 71
   at Microsoft.Arcade.Common.Helpers.MutexExec[T](Func`1 function, String mutexName) in /_/src/Common/Microsoft.Arcade.Common/Helpers.cs:line 45
   at Microsoft.Arcade.Common.Helpers.DirectoryMutexExec[T](Func`1 function, String path) in /_/src/Common/Microsoft.Arcade.Common/Helpers.cs:line 64
   at Microsoft.Arcade.Common.Helpers.DirectoryMutexExec(Func`1 function, String path) in /_/src/Common/Microsoft.Arcade.Common/Helpers.cs:line 71
   at Microsoft.DotNet.Helix.Sdk.ProvisioningProfileProvider.DownloadProvisioningProfile(ApplePlatform platform) in /_/src/Microsoft.DotNet.Helix/Sdk/ProvisioningProfileProvider.cs:line 179
   at Microsoft.DotNet.Helix.Sdk.ProvisioningProfileProvider.AddProfileToPayload(String archivePath, String testTarget) in /_/src/Microsoft.DotNet.Helix/Sdk/ProvisioningProfileProvider.cs:line 99
   at Microsoft.DotNet.Helix.Sdk.CreateXHarnessAppleWorkItems.PrepareWorkItem(IZipArchiveManager zipArchiveManager, IFileSystem fileSystem, IProvisioningProfileProvider provisioningProfileProvider, ITaskItem appBundleItem) in /_/src/Microsoft.DotNet.Helix/Sdk/CreateXHarnessAppleWorkItems.cs:line 189
   at Microsoft.DotNet.Helix.Sdk.CreateXHarnessAppleWorkItems.ExecuteTask(IProvisioningProfileProvider provisioningProfileProvider, IZipArchiveManager zipArchiveManager, IFileSystem fileSystem) in /_/src/Microsoft.DotNet.Helix/Sdk/CreateXHarnessAppleWorkItems.cs:line 81

@radical do you mind tagging the right people that should be looking into this?

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Rolling Build failed this morning on the Build iOS arm64 Release AllSubsets_Mono job at the Send to Helix step, with a problem when trying to use the tool microsoft.dotnet.xharness.cli to run the CreateXHarnessAppleWorkItems task. The issue seems to be just a failure in the SSL handshake communication. To view more details, you can get to the log of the failing build here.

For convenience, I'm pasting the stack trace of the failure below:

##[error].packages/microsoft.dotnet.helix.sdk/7.0.0-beta.22171.2/tools/xharness-runner/XHarnessRunner.targets(122,5): error MSB4018: The "CreateXHarnessAppleWorkItems" task failed unexpectedly.
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: RevocationStatusUnknown
   at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.DotNet.Helix.Sdk.ProvisioningProfileProvider.<>c__DisplayClass14_0.<<DownloadProvisioningProfile>b__0>d.MoveNext() in /_/src/Microsoft.DotNet.Helix/Sdk/ProvisioningProfileProvider.cs:line 190
--- End of stack trace from previous location ---
   at Microsoft.Arcade.Common.Helpers.<>c__DisplayClass7_0.<DirectoryMutexExec>b__0() in /_/src/Common/Microsoft.Arcade.Common/Helpers.cs:line 71
   at Microsoft.Arcade.Common.Helpers.MutexExec[T](Func`1 function, String mutexName) in /_/src/Common/Microsoft.Arcade.Common/Helpers.cs:line 45
   at Microsoft.Arcade.Common.Helpers.DirectoryMutexExec[T](Func`1 function, String path) in /_/src/Common/Microsoft.Arcade.Common/Helpers.cs:line 64
   at Microsoft.Arcade.Common.Helpers.DirectoryMutexExec(Func`1 function, String path) in /_/src/Common/Microsoft.Arcade.Common/Helpers.cs:line 71
   at Microsoft.DotNet.Helix.Sdk.ProvisioningProfileProvider.DownloadProvisioningProfile(ApplePlatform platform) in /_/src/Microsoft.DotNet.Helix/Sdk/ProvisioningProfileProvider.cs:line 179
   at Microsoft.DotNet.Helix.Sdk.ProvisioningProfileProvider.AddProfileToPayload(String archivePath, String testTarget) in /_/src/Microsoft.DotNet.Helix/Sdk/ProvisioningProfileProvider.cs:line 99
   at Microsoft.DotNet.Helix.Sdk.CreateXHarnessAppleWorkItems.PrepareWorkItem(IZipArchiveManager zipArchiveManager, IFileSystem fileSystem, IProvisioningProfileProvider provisioningProfileProvider, ITaskItem appBundleItem) in /_/src/Microsoft.DotNet.Helix/Sdk/CreateXHarnessAppleWorkItems.cs:line 189
   at Microsoft.DotNet.Helix.Sdk.CreateXHarnessAppleWorkItems.ExecuteTask(IProvisioningProfileProvider provisioningProfileProvider, IZipArchiveManager zipArchiveManager, IFileSystem fileSystem) in /_/src/Microsoft.DotNet.Helix/Sdk/CreateXHarnessAppleWorkItems.cs:line 81

@radical do you mind tagging the right people that should be looking into this?

Author:	joperezr
Assignees:	-
Labels:	arch-arm64, blocking-clean-ci, area-Infrastructure-mono, os-ios
Milestone:	-

[radical]

System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: RevocationStatusUnknown

cc @steveisok

[steveisok]

@premun looks like there might be a problem trying to grab the provisioning profile.

[premun]

I guess this is a fluke? Or possibly something to do with the TLS 1.2 changes happening all around, maybe the agent didn't have this enabled somehow?

I tried to run this again and seems to work but I will add retries: dotnet/arcade#8703

[steveisok]

Thanks, @premun.

[joperezr]

This doesn't seem fixed, it failed again in today's build: https://dev.azure.com/dnceng/public/_build/results?buildId=1694490&view=logs&j=f4520fb1-1559-5885-1d9c-3cb3f6a85e23&t=6c7a8cfe-f92e-569a-eef9-b2ad3e13056d

It does seem flaky as we have had successful builds in between the two with the failure, but it does indeed happen some times so should be investigated as it is now twice this week where this is the only error keeping the rolling build from being green.