System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout hangs in CI

[ViktorHofer]

https://dnceng.visualstudio.com/public/_build/results?buildId=332322&view=ms.vss-test-web.build-test-results-tab&runId=9745396&resultId=175895&paneView=attachments

Configuration: netcoreapp-Linux-Release-x64-(Alpine.38.Amd64.Open)ubuntu.1604.amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.8-helix-09ca40b-20190620184125

  Discovering: System.Net.Mail.Functional.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Mail.Functional.Tests (found 137 test cases)
  Starting:    System.Net.Mail.Functional.Tests (parallel test collections = on, max threads = 2)
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout', Elapsed: 00:02:10
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout', Elapsed: 00:04:10
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout', Elapsed: 00:06:10
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout', Elapsed: 00:08:10
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout', Elapsed: 00:10:10
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout', Elapsed: 00:12:10
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestZeroTimeout', Elapsed: 00:14:10

@davidsh, @wfurt, @scalablecory, @eiriktsarpalis, @karelz

[stephentoub]

@tmds just added this in https://github.com/dotnet/corefx/pull/37462/files#diff-a1e1c77e4753d51b3aa361a5130872f1R319.

We should disable it.

[wfurt]

did we collect core file @ViktorHofer?
And yes, disabling this for now seems like best step.
However it may be hard to improver the test if it does not fail locally.

[ViktorHofer]

did we collect core file

you mean a crash dump? I don't know if dumps are collected for work items that are hanging by terminating them. Also I don't know if crash dumps are already working for this configuration (alpine on docker).

cc @MattGal who might be able to answer some of these.

[tmds]

It's not a test bug. The test is is working, it's showing an issue that was there, and unfortunately not fixed.
A coredump would be nice indeed.
I'll run the test in a loop and see if I can reproduce the hang.

[tmds]

I'll run the test in a loop and see if I can reproduce the hang.

I ran the test project 1000 times completely, and 1000 times filtered to TestZeroTimeout. I couldn't reproduce the hang.

If we can get a coredump from CI it would help figure out the issue.

[tmds]

can we re-enable the test to get a coredump?

[ViktorHofer]

As said, I don't think we collect a dump when tests are hanging.

[tmds]

If I change the test so it calls Environment.FailFast for cases it is now hanging. Will that produce a coredump?

[ViktorHofer]

Yes

[safern]

It failed again and it produced a coredump:

https://helix.dot.net/api/2019-06-17/jobs/5335d74d-89b3-4543-9c31-2236a792653d/workitems/System.Net.Mail.Functional.Tests/files/core.1000.2870

https://dev.azure.com/dnceng/public/_build/results?buildId=408989&view=ms.vss-test-web.build-test-results-tab&runId=12790968&resultId=176542&paneView=attachments

[tmds]

https://helix.dot.net/api/2019-06-17/jobs/5335d74d-89b3-4543-9c31-2236a792653d/workitems/System.Net.Mail.Functional.Tests/files/core.1000.2870

I tried to use dotnet dump analyze but the clrstack and clrthread commands are hanging. Not sure if it is supposed to work.

I'll have a look at debugging this using libsosplugin.so next week.

[ViktorHofer]

@jkotas @danmosemsft can you help Tom? I tried it myself and didn't get very far with WinDbg.