[mono] Tests failing with `Failed to find compatible device: arm64-v8a`

[ivanpovazan]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=298055

Build error leg:

• linux_bionic-arm64 Release AllSubsets_Mono

Tests failing:

• Microsoft.Bcl.AsyncInterfaces.Tests.WorkItemExecution
• Microsoft.Extensions.DependencyInjection.ExternalContainers.Tests.WorkItemExecution
• System.Diagnostics.DiagnosticSource.Switches.Tests.WorkItemExecution
• System.IO.Pipes.Tests.WorkItemExecution
• System.Linq.Queryable.Tests.WorkItemExecution

Pull request: #86652

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "crit: Failed to find compatible device: arm64-v8a",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Test	Pull Request
476263	dotnet/runtime	Microsoft.VisualBasic.Core.Tests.WorkItemExecution
476004	dotnet/runtime	Microsoft.XmlSerializer.Generator.Tests.WorkItemExecution
472380	dotnet/runtime	System.ComponentModel.TypeConverter.Tests.WorkItemExecution	#94844
475900	dotnet/runtime	Microsoft.Extensions.FileProviders.Composite.Tests.WorkItemExecution
475292	dotnet/runtime	Microsoft.Extensions.Configuration.Binder.Tests.WorkItemExecution
475327	dotnet/runtime	IntrinsicsInSystemPrivateCoreLib.Tests.WorkItemExecution
475076	dotnet/runtime	Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.WorkItemExecution
474832	dotnet/runtime	System.Composition.AttributeModel.Tests.WorkItemExecution
471584	dotnet/runtime	System.Threading.ThreadPool.Tests.WorkItemExecution
471508	dotnet/runtime	System.Text.RegularExpressions.Unit.Tests.WorkItemExecution
471393	dotnet/runtime	System.Xml.RW.CustomReader.Tests.WorkItemExecution
471313	dotnet/runtime	System.Threading.RateLimiting.Tests.WorkItemExecution
470965	dotnet/runtime	System.Threading.RateLimiting.Tests.WorkItemExecution	#93220
470621	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
470544	dotnet/runtime	System.Security.Cryptography.Xml.Tests.WorkItemExecution
470317	dotnet/runtime	System.Security.Cryptography.Xml.Tests.WorkItemExecution
470325	dotnet/runtime	System.Xml.Linq.Axes.Tests.WorkItemExecution
469832	dotnet/runtime	System.Runtime.Serialization.Json.ReflectionOnly.Tests.WorkItemExecution
469799	dotnet/runtime	System.Threading.Tasks.Dataflow.Tests.WorkItemExecution
469455	dotnet/runtime	System.Threading.Tasks.Dataflow.Tests.WorkItemExecution
469234	dotnet/runtime	System.Threading.RateLimiting.Tests.WorkItemExecution
468967	dotnet/runtime	System.Xml.RW.SubtreeReader.Tests.WorkItemExecution
468943	dotnet/runtime	System.Security.Cryptography.ProtectedData.Tests.WorkItemExecution
468844	dotnet/runtime	System.Xml.Linq.Misc.Tests.WorkItemExecution
468831	dotnet/runtime	System.Threading.Tasks.Extensions.Tests.WorkItemExecution
468599	dotnet/runtime	System.Threading.Tasks.Tests.WorkItemExecution
468590	dotnet/runtime	System.Xml.RW.NameTable.Tests.WorkItemExecution
468420	dotnet/runtime	System.Threading.Channels.Tests.WorkItemExecution
468095	dotnet/runtime	System.Xml.RW.CustomReader.Tests.WorkItemExecution
467978	dotnet/runtime	System.ValueTuple.Tests.WorkItemExecution
467759	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#94602
467535	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution
467341	dotnet/runtime	System.Threading.Channels.Tests.WorkItemExecution
467123	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution
466947	dotnet/runtime	System.Web.HttpUtility.Tests.WorkItemExecution
466956	dotnet/runtime	System.Threading.RateLimiting.Tests.WorkItemExecution
466670	dotnet/runtime	System.Threading.Tasks.Parallel.Tests.WorkItemExecution
466386	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
466152	dotnet/runtime	System.Threading.Tasks.Parallel.Tests.WorkItemExecution
462819	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution
462611	dotnet/runtime	System.CodeDom.Tests.WorkItemExecution
462515	dotnet/runtime	System.CodeDom.Tests.WorkItemExecution
462059	dotnet/runtime	Microsoft.Bcl.AsyncInterfaces.Tests.WorkItemExecution
461730	dotnet/runtime	Microsoft.Extensions.FileProviders.Composite.Tests.WorkItemExecution
458877	dotnet/runtime	Microsoft.XmlSerializer.Generator.Tests.WorkItemExecution
458245	dotnet/runtime	System.Diagnostics.Contracts.Tests.WorkItemExecution
457943	dotnet/runtime	Microsoft.Extensions.Configuration.EnvironmentVariables.Tests.WorkItemExecution
457916	dotnet/runtime	Microsoft.Extensions.Configuration.Json.Tests.WorkItemExecution	#94274
457319	dotnet/runtime	System.ComponentModel.EventBasedAsync.Tests.WorkItemExecution
456975	dotnet/runtime	Microsoft.Extensions.FileProviders.Composite.Tests.WorkItemExecution
456789	dotnet/runtime	System.AppContext.Tests.WorkItemExecution
456721	dotnet/runtime	System.Buffers.Tests.WorkItemExecution	#94038
456534	dotnet/runtime	Microsoft.Extensions.Options.Tests.WorkItemExecution
456177	dotnet/runtime	System.Threading.Tasks.Extensions.Tests.WorkItemExecution
455954	dotnet/runtime	System.Threading.Tests.WorkItemExecution
455930	dotnet/runtime	System.Threading.Tasks.Parallel.Tests.WorkItemExecution
455771	dotnet/runtime	System.Runtime.Serialization.Schema.Tests.WorkItemExecution
455739	dotnet/runtime	System.Threading.Tasks.Parallel.Tests.WorkItemExecution
455556	dotnet/runtime	System.Threading.RateLimiting.Tests.WorkItemExecution
455375	dotnet/runtime	System.Text.Encoding.Extensions.Tests.WorkItemExecution	#94038
454951	dotnet/runtime	System.Threading.Overlapped.Tests.WorkItemExecution
454774	dotnet/runtime	System.Threading.RateLimiting.Tests.WorkItemExecution
454509	dotnet/runtime	System.Threading.Channels.Tests.WorkItemExecution
454434	dotnet/runtime	System.Threading.Tasks.Extensions.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	8	64

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323893
Error message validated: crit: Failed to find compatible device: arm64-v8a
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:22:27 PM UTC

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=298055
Build error leg or test failing:

• Microsoft.Bcl.AsyncInterfaces.Tests.WorkItemExecution
• Microsoft.Extensions.DependencyInjection.ExternalContainers.Tests.WorkItemExecution
• System.Diagnostics.DiagnosticSource.Switches.Tests.WorkItemExecution
• System.IO.Pipes.Tests.WorkItemExecution
• System.Linq.Queryable.Tests.WorkItemExecution

Pull request: #86652

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "ERROR: The process \"corerun.exe\" not found",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Author:	ivanpovazan
Assignees:	-
Labels:	area-Infrastructure-libraries, blocking-clean-ci, Known Build Error
Milestone:	-

[jakobbotsch]

@ivanpovazan I think this log statement is from

runtime/src/tests/Common/helixpublishwitharcade.proj

Line 702 in 03f2be6

<HelixPostCommand Include="taskkill.exe /f /im corerun.exe"/>

and is harmless. E.g. it triggered on my #87165 for the JIT.opt workitem but the actual failure there was #87258

[ivanpovazan]

@jakobbotsch thanks for the info!
Does it mean that the issue goes away on retries?

[jakobbotsch]

I don't think there is an issue related to that log statement and I don't think it causes the work items to fail. So you should look for other failures in the logs. E.g. in
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-86652-merge-1c9c9cfd4ebd4ec7b3/System.IO.Pipes.Tests/1/console.5b1e5bba.log?helixlogtype=result
(for the System.IO.Pipelines failure in net8.0-linux-Release-arm64-Mono_Release_LinuxBionic-Windows.10.Amd64.Android.Open), the log ends with "ERROR: The process "corerun.exe" not found.", but the actual failure is above that:

[02:00:43] info: An error occurred running System.IO.Pipes.Tests.sh
[02:00:43] dbug: Exit code: 1
                 Std out:
                   Discovering: System.IO.Pipes.Tests (method display = ClassAndMethod, method display options = None)
                   Discovered:  System.IO.Pipes.Tests (found 107 of 538 test cases)
                   Starting:    System.IO.Pipes.Tests (parallel test collections = on, max threads = 8)
                       System.UnauthorizedAccessException : Access to the path '/data/local/tmp/CoreFxPipe_g2n3xo52.px0' is denied.
                       Stack Trace:
                         /_/src/libraries/System.IO.Pipes/src/System/IO/Pipes/NamedPipeServerStream.Unix.cs(336,0): at System.IO.Pipes.NamedPipeServerStream.SharedServer..ctor(String path, Int32 maxCount, Boolean isFirstPipeInstance)
                         /_/src/libraries/System.IO.Pipes/src/System/IO/Pipes/NamedPipeServerStream.Unix.cs(279,0): at System.IO.Pipes.NamedPipeServerStream.SharedServer.Get(String path, Int32 maxCount, PipeOptions pipeOptions)
                         /_/src/libraries/System.IO.Pipes/src/System/IO/Pipes/NamedPipeServerStream.Unix.cs(43,0): at System.IO.Pipes.NamedPipeServerStream.Create(String pipeName, PipeDirection direction, Int32 maxNumberOfServerInstances, PipeTransmissionMode transmissionMode, PipeOptions options, Int32 inBufferSize, Int32 outBufferSize, HandleInheritability inheritability)
                         /_/src/libraries/System.IO.Pipes/src/System/IO/Pipes/NamedPipeServerStream.cs(82,0): at System.IO.Pipes.NamedPipeServerStream..ctor(String pipeName, PipeDirection direction, Int32 maxNumberOfServerInstances, PipeTransmissionMode transmissionMode, PipeOptions options, Int32 inBufferSize, Int32 outBufferSize, HandleInheritability inheritability)
                         /_/src/libraries/System.IO.Pipes/src/System/IO/Pipes/NamedPipeServerStream.cs(39,0): at System.IO.Pipes.NamedPipeServerStream..ctor(String pipeName, PipeDirection direction, Int32 maxNumberOfServerInstances, PipeTransmissionMode transmissionMode, PipeOptions options)
                         /_/src/libraries/System.IO.Pipes/tests/NamedPipeTests/NamedPipeTest.CreateServer.cs(263,0): at System.IO.Pipes.Tests.NamedPipeTest_CreateServer.PipeOptions_FirstPipeInstanceWithSameNameReuse_Throws_UnauthorizedAccessException()
                         /_/src/mono/System.Private.CoreLib/src/System/Reflection/MethodInvoker.Mono.cs(30,0): at System.Reflection.MethodInvoker.InterpretedInvoke(Object obj, IntPtr* args)
                         /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodInvoker.cs(59,0): at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
                   Finished:    System.IO.Pipes.Tests
                 === TEST EXECUTION SUMMARY ===
                    System.IO.Pipes.Tests  Total: 150, Errors: 0, Failed: 1, Skipped: 0, Time: 2.364s
                 
                 
                 Std err:
                     System.IO.Pipes.Tests.NamedPipeTest_CreateServer.PipeOptions_FirstPipeInstanceWithSameNameReuse_Throws_UnauthorizedAccessException [FAIL]

So for example you could use System.IO.Pipes.Tests.NamedPipeTest_CreateServer.PipeOptions_FirstPipeInstanceWithSameNameReuse_Throws_UnauthorizedAccessException [FAIL] as the error message in the known error template. Using "ERROR: The process "corerun.exe" not found",` is causing tons of unrelated PRs to be linked to this issue.

[ivanpovazan]

Thank you very much for the detailed explanation, and sorry for the confusion.

I have updated the search term to be: crit: Failed to find compatible device: arm64-v8a instead of wrongly accused corerun.exe not found

This issue seems to be related to a missing device on linux_bionic-arm64 Release AllSubsets_Mono, while on the linux_bionic-x64 Release AllSubsets_Mono (note the difference in architecture) there is the failing test @jakobbotsch noted above:

System.IO.Pipes.Tests.NamedPipeTest_CreateServer.PipeOptions_FirstPipeInstanceWithSameNameReuse_Throws_UnauthorizedAccessException [FAIL]

I will create a separate tracking issue as these don't seem to be related.

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=298055

Build error leg:

• linux_bionic-arm64 Release AllSubsets_Mono

Tests failing:

• Microsoft.Bcl.AsyncInterfaces.Tests.WorkItemExecution
• Microsoft.Extensions.DependencyInjection.ExternalContainers.Tests.WorkItemExecution
• System.Diagnostics.DiagnosticSource.Switches.Tests.WorkItemExecution
• System.IO.Pipes.Tests.WorkItemExecution
• System.Linq.Queryable.Tests.WorkItemExecution

Pull request: #86652

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "crit: Failed to find compatible device: arm64-v8a",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Test	Pull Request
299742	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
299732	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
299717	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ServerCertificates_Test.NoCallback_RevokedCertificate_NoRevocationChecking_Succeeds
299738	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
299736	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
299735	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
299727	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
299719	dotnet/runtime	System.Text.RegularExpressions.Tests.RegexMatchTests.StressTestDeepNestingOfLoops
299731	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
299527	dotnet/runtime	Workloads-NoWebcil-Wasm.Build.Tests.Blazor.MiscTests.WorkItemExecution	#86546
295712	dotnet/runtime	chrome-DebuggerTests.WorkItemExecution	#86934
299396	dotnet/runtime	JIT.opt.WorkItemExecution
299429	dotnet/runtime	JIT.opt.WorkItemExecution
299427	dotnet/runtime	profiler.rejit.WorkItemExecution
299459	dotnet/runtime	BasicEventSourceTests.TestsManifestGeneration.Test_EventSource_EtwManifestGenerationRollover
299397	dotnet/runtime	JIT.opt.WorkItemExecution
299370	dotnet/runtime	JIT.opt.WorkItemExecution
299406	dotnet/runtime	nativeaot.SmokeTests.WorkItemExecution	#87148
299242	dotnet/runtime	profiler.rejit.WorkItemExecution
299286	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#87241
299094	dotnet/runtime	JIT.opt.WorkItemExecution	#87165
298055	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#86652
298971	dotnet/runtime	chrome-DebuggerTests.WorkItemExecution	#87232
298972	dotnet/runtime	chrome-DebuggerTests.WorkItemExecution	#87233
299037	dotnet/runtime	System.Net.Http.Functional.Tests.SocksProxyTest_Http2.TestLoopbackAsync
298965	dotnet/runtime	BasicEventSourceTests.TestsManifestGeneration.Test_EventSource_EtwManifestGenerationRollover	#87232
298917	dotnet/runtime	System.Data.OleDb.Tests.WorkItemExecution	#87229
298677	dotnet/runtime	profiler.rejit.WorkItemExecution
298816	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest_Http3.ContentLength_DoesNotMatchRequestContentLength_Throws
298813	dotnet/runtime	WasmTestOnBrowser-simd-System.Runtime.Intrinsics.Tests.WorkItemExecution
298779	dotnet/runtime	ComInterfaceGenerator.Unit.Tests.ComHostingDoesNotSupportGeneratedComInterfaceTests.ComVisibleType_ComImportInterfacesOnly_DoesNotReportDiagnostic	#87223
297507	dotnet/runtime	WasmTestOnBrowser-System.Runtime.Loader.Tests.WorkItemExecution	#85796
298658	dotnet/runtime	System.Text.Json.SourceGeneration.UnitTests.JsonSourceGeneratorDiagnosticsTests.JsonStringEnumConverterWarns	#87149
298623	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ConnectionPooling_Test.Http2_SmallConnectionTimeout_SubsequentRequestUsesDifferentConnection	#86759
298540	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#86875
298472	dotnet/runtime	System.Numerics.Tests.Matrix4x4Tests.Matrix4x4CreateShadowTest02	#87126
297988	dotnet/runtime	Wasm.Build.NativeRebuild.Tests.NoopNativeRebuildTest.NoOpRebuildForNativeBuilds	#86255
298474	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#87126
298200	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
298206	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
298223	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Http3_MsQuic.Public_Interop_Upgrade_Success
298221	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
298203	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_Http3_MsQuic.Public_Interop_Upgrade_Success
298218	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
298213	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
298209	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
298210	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
298219	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
298199	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
298211	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution
298224	dotnet/runtime	PayloadGroup0.WorkItemExecution
298226	dotnet/runtime	System.Net.Http.Functional.Tests.PlatformHandler_HttpClientHandler_ServerCertificates_Test.NoCallback_RevokedCertificate_NoRevocationChecking_Succeeds
298124	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_RequestContentLengthMismatchTest_Http2.ContentLength_DoesNotMatchRequestContentLength_Throws	#87162
297984	dotnet/runtime	Workloads-Wasm.Build.Tests.Blazor.BuildPublishTests.WorkItemExecution	#87051
298068	dotnet/runtime	System.Security.Cryptography.OpenSsl.Tests.WorkItemExecution
298000	dotnet/runtime	JIT/SIMD/VectorAdd_ro/VectorAdd_ro.dll
297910	dotnet/runtime	profiler.rejit.WorkItemExecution
297843	dotnet/runtime	JIT.opt.WorkItemExecution
297912	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_MaxResponseHeadersLength_Http2.ThresholdExceeded_ThrowsException	#87203

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
40	59	59

Author:	ivanpovazan
Assignees:	-
Labels:	blocking-clean-ci, untriaged, area-Infrastructure-mono, Known Build Error
Milestone:	-