Work item tests crash on OSX x64 Debug

[eerhardt]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300158
Build error leg or test failing: System.IO.FileSystem.Watcher.Tests.WorkItemExecution
Pull request: #86974

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "END OF WORK ITEM LOG: Command exited with 127",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Test	Pull Request
475355	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
474849	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
474532	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
473727	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
473597	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
473431	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
473397	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
472898	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
472852	dotnet/runtime	System.IO.Compression.Tests.WorkItemExecution
472282	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
471895	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
471740	dotnet/runtime	System.IO.Compression.Tests.WorkItemExecution
471580	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
471389	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
470637	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
470321	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
469012	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
468964	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
468841	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
468587	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
468089	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
467619	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
467146	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
466691	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
465768	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
465695	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
465642	dotnet/runtime	XUnitLogChecker.WorkItemExecution	#93906
465587	dotnet/runtime	XUnitLogChecker.WorkItemExecution	#93906
465447	dotnet/runtime	XUnitLogChecker.WorkItemExecution	#93906
465319	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
465327	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
464592	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
463142	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
462608	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
461771	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
460596	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
460238	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
459941	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
459223	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
458289	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
457386	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
456786	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
456243	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
456245	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
455009	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
454543	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
454533	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
454134	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
452981	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
451747	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
450630	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
449186	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
447511	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
446965	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	21	54

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323604
Error message validated: Work item System.IO.FileSystem.Watcher.Tests in job
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:26:54 PM UTC

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300158
Build error leg or test failing: System.IO.FileSystem.Watcher.Tests.WorkItemExecution
Pull request: #86974

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Work item System.IO.FileSystem.Watcher.Tests in job",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Author:	eerhardt
Assignees:	-
Labels:	area-Infrastructure-libraries, blocking-clean-ci, untriaged, Known Build Error
Milestone:	-

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300158
Build error leg or test failing: System.IO.FileSystem.Watcher.Tests.WorkItemExecution
Pull request: #86974

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Work item System.IO.FileSystem.Watcher.Tests in job",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Author:	eerhardt
Assignees:	-
Labels:	area-System.IO, blocking-clean-ci, untriaged, Known Build Error
Milestone:	-

[jozkee]

The logs not only show FileSystem.Watcher.Tests. I think this is a broader issue.

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/334444/logs/123

e.g:

2023-07-11T00:12:22.4993970Z /Users/runner/work/1/s/.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : Work item Common.Tests in job 9199670f-aa4c-4e8e-8150-b650ad0bb59f has failed. [/Users/runner/work/1/s/src/libraries/sendtohelixhelp.proj]
2023-07-11T00:12:22.4995970Z /Users/runner/work/1/s/.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : Failure log: https://helix.dot.net/api/2019-06-17/jobs/9199670f-aa4c-4e8e-8150-b650ad0bb59f/workitems/Common.Tests/console [/Users/runner/work/1/s/src/libraries/sendtohelixhelp.proj]
2023-07-11T00:12:22.5108980Z ##[error].packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item Common.Tests in job 9199670f-aa4c-4e8e-8150-b650ad0bb59f has failed.
Failure log: https://helix.dot.net/api/2019-06-17/jobs/9199670f-aa4c-4e8e-8150-b650ad0bb59f/workitems/Common.Tests/console
2023-07-11T00:12:22.5122420Z /Users/runner/work/1/s/.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : Work item IcuAppLocal.Tests in job 9199670f-aa4c-4e8e-8150-b650ad0bb59f has failed. [/Users/runner/work/1/s/src/libraries/sendtohelixhelp.proj]
2023-07-11T00:12:22.5126220Z /Users/runner/work/1/s/.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : Failure log: https://helix.dot.net/api/2019-06-17/jobs/9199670f-aa4c-4e8e-8150-b650ad0bb59f/workitems/IcuAppLocal.Tests/console [/Users/runner/work/1/s/src/libraries/sendtohelixhelp.proj]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300158
Build error leg or test failing: System.IO.FileSystem.Watcher.Tests.WorkItemExecution
Pull request: #86974

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Work item System.IO.FileSystem.Watcher.Tests in job",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Step Name	Console log	Pull Request
347502	dotnet/runtime	Send to Helix	Log
347514	dotnet/runtime	Send to Helix	Log
347516	dotnet/runtime	Send to Helix	Log
347506	dotnet/runtime	Send to Helix	Log
347504	dotnet/runtime	Send to Helix	Log
347527	dotnet/runtime	Send to Helix	Log
346179	dotnet/runtime	Send to Helix	Log
346184	dotnet/runtime	Send to Helix	Log
346183	dotnet/runtime	Send to Helix	Log
346189	dotnet/runtime	Send to Helix	Log
346180	dotnet/runtime	Send to Helix	Log
346188	dotnet/runtime	Send to Helix	Log
345385	dotnet/runtime	Send to Helix	Log	#89145
344799	dotnet/runtime	Send to Helix	Log
344802	dotnet/runtime	Send to Helix	Log
344773	dotnet/runtime	Send to Helix	Log
344775	dotnet/runtime	Send to Helix	Log
344801	dotnet/runtime	Send to Helix	Log
344791	dotnet/runtime	Send to Helix	Log
344313	dotnet/runtime	Send to Helix	Log	#89145
344014	dotnet/runtime	Send to Helix	Log	#89061
343534	dotnet/runtime	Send to Helix	Log
343456	dotnet/runtime	Send to Helix	Log	#89074
343307	dotnet/runtime	Send to Helix	Log
343300	dotnet/runtime	Send to Helix	Log
343299	dotnet/runtime	Send to Helix	Log
343301	dotnet/runtime	Send to Helix	Log
343298	dotnet/runtime	Send to Helix	Log
343296	dotnet/runtime	Send to Helix	Log
343319	dotnet/runtime	Send to Helix	Log	#89074
342971	dotnet/runtime	Send to Helix	Log
342902	dotnet/runtime	Send to Helix	Log	#87847
342704	dotnet/runtime	Send to Helix	Log	#88974
342426	dotnet/runtime	Send to Helix	Log
342101	dotnet/runtime	Send to Helix	Log
341764	dotnet/runtime	Send to Helix	Log
341747	dotnet/runtime	Send to Helix	Log
341763	dotnet/runtime	Send to Helix	Log
341766	dotnet/runtime	Send to Helix	Log
341752	dotnet/runtime	Send to Helix	Log
341745	dotnet/runtime	Send to Helix	Log
341422	dotnet/runtime	Send to Helix	Log	#88502
341117	dotnet/runtime	Send to Helix	Log
341130	dotnet/runtime	Send to Helix	Log
341115	dotnet/runtime	Send to Helix	Log
341116	dotnet/runtime	Send to Helix	Log
341137	dotnet/runtime	Send to Helix	Log
341114	dotnet/runtime	Send to Helix	Log
340730	dotnet/runtime	Send to Helix	Log
340706	dotnet/runtime	Send to Helix	Log
340711	dotnet/runtime	Send to Helix	Log
340713	dotnet/runtime	Send to Helix	Log
340722	dotnet/runtime	Send to Helix	Log
340731	dotnet/runtime	Send to Helix	Log
340606	dotnet/runtime	Send to Helix	Log	#74623
339887	dotnet/runtime	Send to Helix	Log
339546	dotnet/runtime	Send to Helix	Log
339567	dotnet/runtime	Send to Helix	Log
339550	dotnet/runtime	Send to Helix	Log
339557	dotnet/runtime	Send to Helix	Log
339565	dotnet/runtime	Send to Helix	Log
339554	dotnet/runtime	Send to Helix	Log
339453	dotnet/runtime	Send to Helix	Log	#88892
339194	dotnet/runtime	Send to Helix	Log	#74623
339190	dotnet/runtime	Send to Helix	Log
338975	dotnet/runtime	Send to Helix	Log
338958	dotnet/runtime	Send to Helix	Log
338949	dotnet/runtime	Send to Helix	Log
338278	dotnet/runtime	Send to Helix	Log
338279	dotnet/runtime	Send to Helix	Log
338284	dotnet/runtime	Send to Helix	Log
338258	dotnet/runtime	Send to Helix	Log
338277	dotnet/runtime	Send to Helix	Log
338291	dotnet/runtime	Send to Helix	Log
338281	dotnet/runtime	Send to Helix	Log
337928	dotnet/runtime	Send to Helix	Log
337971	dotnet/runtime	Send to Helix	Log
336905	dotnet/runtime	Send to Helix	Log
336900	dotnet/runtime	Send to Helix	Log
336626	dotnet/runtime	Send to Helix	Log
336635	dotnet/runtime	Send to Helix	Log
336627	dotnet/runtime	Send to Helix	Log
336647	dotnet/runtime	Send to Helix	Log
336638	dotnet/runtime	Send to Helix	Log
336646	dotnet/runtime	Send to Helix	Log
336353	dotnet/runtime	Send to Helix	Log
336347	dotnet/runtime	Send to Helix	Log
335895	dotnet/runtime	Send to Helix	Log	#88626
335021	dotnet/runtime	Send to Helix	Log
335024	dotnet/runtime	Send to Helix	Log
335003	dotnet/runtime	Send to Helix	Log
335004	dotnet/runtime	Send to Helix	Log
334953	dotnet/runtime	Send to Helix	Log
335012	dotnet/runtime	Send to Helix	Log
335002	dotnet/runtime	Send to Helix	Log
334959	dotnet/runtime	Send to Helix	Log
334772	dotnet/runtime	Send to Helix	Log
334717	dotnet/runtime	Send to Helix	Log
334699	dotnet/runtime	Send to Helix	Log
334444	dotnet/runtime	Send to Helix	Log
Displaying 100 of 262 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
6	55	262

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323604
Error message validated: Work item System.IO.FileSystem.Watcher.Tests in job
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:26:54 PM UTC

Author:	eerhardt
Assignees:	-
Labels:	blocking-clean-ci, area-Infrastructure, untriaged, Known Build Error
Milestone:	-

[carlossanlop]

Yea, doesn't seem to be a FileSystemWatcher-only issue. The jobs are getting unexpectedly killed, according to the console log outputs. Example:

• Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-release-70-staging-9199670faa4c4e8e81/Microsoft.Bcl.AsyncInterfaces.Tests/1/console.7114ae01.log?helixlogtype=result
  Output:

2023-07-11T00:05:26.562Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2023-07-11T00:05:26.563Z	INFO   	run.py	__init__(42)	read_results	Searching '/private/tmp/helix/working/A8D4097B/w/9393087D/e' for test results files
2023-07-11T00:05:26.563Z	INFO   	run.py	__init__(42)	read_results	Searching '/tmp/helix/working/A8D4097B/w/9393087D/uploads' for test results files
2023-07-11T00:05:26.563Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2023-07-11T00:05:26.563Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/tmp/helix/working/A8D4097B/w/9393087D/e/__test_report.json'
2023-07-11T00:05:26.564Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1523 bytes
+ exit 127
['Microsoft.Bcl.AsyncInterfaces.Tests' END OF WORK ITEM LOG: Command exited with 127]