[macos][arm64][CoreCLR] DllImportGenerator.Unit.Tests failed: Segmentation fault

[MaximLipnin]

DllImportGenerator.Unit.Tests have been failing for a while with

exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.

Caught by the rolling build:

https://dev.azure.com/dnceng/public/_build/results?buildId=1538735&view=ms.vss-test-web.build-test-results-tab&runId=43545378&resultId=192461&paneView=dotnet-dnceng.dnceng-anon-build-release-tasks.helix-anon-test-information-tab

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-b058599c5a6c422d80/DllImportGenerator.Unit.Tests/1/console.de9a3c30.log?sv=2019-07-07&se=2022-01-26T08%3A11%3A56Z&sr=c&sp=rl&sig=UZe9m6%2Bc9K7V6taIX%2B1f91gSYbe7Y4JyL1p8KgcL83g%3D

---

Tests were disabled with #63887

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

---

DllImportGenerator.Unit.Tests have been failing for a while with

exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.

Caught by the rolling build:

https://dev.azure.com/dnceng/public/_build/results?buildId=1538735&view=ms.vss-test-web.build-test-results-tab&runId=43545378&resultId=192461&paneView=dotnet-dnceng.dnceng-anon-build-release-tasks.helix-anon-test-information-tab

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-b058599c5a6c422d80/DllImportGenerator.Unit.Tests/1/console.de9a3c30.log?sv=2019-07-07&se=2022-01-26T08%3A11%3A56Z&sr=c&sp=rl&sig=UZe9m6%2Bc9K7V6taIX%2B1f91gSYbe7Y4JyL1p8KgcL83g%3D

Author:	MaximLipnin
Assignees:	-
Labels:	area-CodeGen-coreclr, untriaged
Milestone:	-

[akoeplinger]

/cc @jkoritzinsky

[danmoseley]

FWIW this started in the ~12 hours ending in 2021-12-19 07:59:19.6390000. It's not in release/6.0. And it's not specific to OS version, nor architecture.

If I do git log --since="Dec 17 2021" --until="Dec 19 2021" ** I don't see anything obviously relevant but someone else can take a better look (or just start with a dump)

Execute: Web | Desktop | Web (Lens) | Desktop (SAW)

https://engsrvprod.kusto.windows.net/engineeringdata

WorkItems 
| where Started > now(-100d)  
| where FriendlyName == "DllImportGenerator.Unit.Tests"
| where ExitCode == 139
//| where Status == "Timeout"
| join kind= inner (
   Jobs  | where Started > now(-30d) | project  QueueName , JobId, Build, Type, Source 
) on JobId
| summarize count() by ExitCode, QueueName, Started, Source// format_datetime(bin(Started, 1d), 'MM-dd-yyyy')
| order by Started asc

ExitCode	QueueName	Started	Source	count_
139	osx.1100.arm64.open	2021-12-19 07:59:19.6390000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-19 19:59:32.7310000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-20 08:03:59.9440000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-20 20:16:34.9020000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-21 08:10:23.9820000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-21 20:15:18.1970000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-22 08:05:47.0280000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-22 20:13:47.6630000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-23 08:05:14.0520000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-23 20:11:06.8350000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-24 08:01:36.1130000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1015.amd64.open	2021-12-24 18:37:33.2990000	pr/public/dotnet/runtime/refs/pull/63126/merge	1
139	osx.1100.arm64.open	2021-12-24 20:00:40.3050000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-25 08:05:38.3650000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-25 20:01:28.2980000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-26 08:07:27.0760000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-26 20:40:01.5180000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-27 08:08:59.5390000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-27 20:08:47.7360000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-28 08:02:27.3820000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-28 21:27:55.1970000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-29 08:06:17.9550000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-29 20:01:34.8200000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-30 08:07:14.2340000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-30 20:09:09.7350000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-31 08:03:18.5490000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2021-12-31 20:01:28.0370000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2022-01-01 08:06:36.6720000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2022-01-01 19:59:42.7930000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2022-01-02 08:03:02.7800000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2022-01-02 20:00:56.5200000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2022-01-03 08:01:22.9460000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2022-01-03 20:21:03.3490000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2022-01-04 08:02:36.3290000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2022-01-04 21:53:27.9320000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2022-01-05 08:04:51.5520000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2022-01-05 20:12:49.5880000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2022-01-06 08:12:02.3830000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2022-01-06 20:21:58.3540000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2022-01-07 08:02:15.8460000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2022-01-07 20:16:55.6270000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2022-01-10 20:16:00.1620000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1100.arm64.open	2022-01-11 08:18:47.8870000	ci/public/dotnet/runtime/refs/heads/main	1
139	ubuntu.1804.amd64.open.rt	2022-01-11 19:08:51.2300000	pr/public/dotnet/runtime/refs/pull/63636/merge	1
139	osx.1015.amd64.open	2022-01-12 16:53:17.8630000	pr/public/dotnet/runtime/refs/pull/63633/merge	1
139	osx.1014.amd64.open	2022-01-12 17:34:15.3130000	pr/public/dotnet/runtime/refs/pull/63674/merge	1
139	osx.1014.amd64.open	2022-01-12 18:41:32.2890000	pr/public/dotnet/runtime/refs/pull/63026/merge	1
139	osx.1014.amd64.open	2022-01-13 01:15:32.7380000	pr/public/dotnet/runtime/refs/pull/63150/merge	1
139	osx.1014.amd64.open	2022-01-14 10:23:32.1020000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1015.amd64.open	2022-01-14 15:05:04.8080000	pr/public/dotnet/runtime/refs/pull/63734/merge	1
139	osx.1015.amd64.open	2022-01-14 17:19:04.4680000	pr/public/dotnet/runtime/refs/pull/63775/merge	1
139	osx.1015.amd64.open	2022-01-14 20:55:27.7610000	pr/public/dotnet/runtime/refs/pull/63722/merge	1
139	osx.1014.amd64.open	2022-01-14 21:21:55.6750000	pr/public/dotnet/runtime/refs/pull/63720/merge	1
139	osx.1014.amd64.open	2022-01-15 03:14:16.2200000	pr/public/dotnet/runtime/refs/pull/63320/merge	1
139	osx.1014.amd64.open	2022-01-15 06:33:08.8340000	pr/public/dotnet/runtime/refs/pull/63787/merge	1
139	osx.1015.amd64.open	2022-01-15 16:44:21.6450000	pr/public/dotnet/runtime/refs/pull/63285/merge	1
139	osx.1015.amd64.open	2022-01-15 17:13:06.7110000	pr/public/dotnet/runtime/refs/pull/63011/merge	1
139	osx.1100.arm64.open	2022-01-15 20:53:02.0430000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1014.amd64.open	2022-01-16 06:00:33.5560000	pr/public/dotnet/runtime/refs/pull/63556/merge	1
139	osx.1100.arm64.open	2022-01-16 08:57:51.3810000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1015.amd64.open	2022-01-16 19:24:58.2810000	pr/public/dotnet/runtime/refs/pull/63594/merge	1
139	osx.1100.arm64.open	2022-01-16 20:02:09.6470000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1014.amd64.open	2022-01-16 20:48:34.0200000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1014.amd64.open	2022-01-16 20:52:54.3580000	pr/public/dotnet/runtime/refs/pull/63690/merge	1
139	osx.1100.arm64.open	2022-01-17 08:19:12.5670000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1015.amd64.open	2022-01-17 08:43:19.2230000	pr/public/dotnet/runtime/refs/pull/63733/merge	1
139	osx.1014.amd64.open	2022-01-17 08:56:16.8920000	ci/public/dotnet/runtime/refs/heads/main	1
139	osx.1015.amd64.open	2022-01-17 10:13:41.9730000	pr/public/dotnet/runtime/refs/pull/63441/merge	1
139	osx.1015.amd64.open	2022-01-17 16:00:26.6500000	pr/public/dotnet/runtime/refs/pull/63594/merge	1
139	osx.1015.amd64.open	2022-01-17 19:09:14.2420000	pr/public/dotnet/runtime/refs/pull/63655/merge	1
139	osx.1100.arm64.open	2022-01-17 20:05:34.0750000	ci/public/dotnet/runtime/refs/heads/main	1

[danmoseley]

Doh - I missed the 30d in my query. Fixing that, the oldest hit I see is 2021-10-20 19:59:52.3070000, which corresponds to

commit b201a16
Merge: 1c949ac 5a79d96
Author: Santiago Fernandez Madero safern@microsoft.com
Date: Mon Oct 18 16:12:39 2021 -0700

Merge feature/use-dllimport-generator to main #59579

[danmoseley]

I see there are core dumps available.

[jkoritzinsky]

cc: @jkotas, @dotnet/interop-contrib I think this might be a GC hole, either the one we've seen already or another one. Can someone who has a Mac take a look?

[jkotas]

Yes, we have been seeing intermittent GC hole type of crashes recently in the CI.

It affects more than OSX. For example, #63636 (comment) have hit the crash on Linux. I have seen it the crash on Windows too.

It affects more than just DllImportGenerator tests. For example, #63835 failed with:

  Starting:    Microsoft.Extensions.DependencyInjection.ExternalContainers.Tests (parallel test collections = on, max threads = 4)

Assert failure(PID 32527 [0x00007f0f], Thread: 135455 [0x2111f]): m_alignpad == 0
    File: /Users/runner/work/1/s/src/coreclr/vm/syncblk.cpp Line: 2952
    Image: /private/tmp/helix/working/A94209AB/p/dotnet

I think that the crash is most common for DllImportGenerator tests on OSX because of DllImportGenerator tests are doing a ton more work (running whole Roslyn) compared to other tests, and OSX machines are among the fastest machines in the CI (intermittent crashes tend to repo on faster machines more frequently).

cc @agocke