VSTest with coverlet collector fails

[arabelaa]

I have a .NETCore 3.1 test project with "Microsoft.NET.Test.Sdk" 16.7.1 of and "coverlet.collector" 1.3.0.

I am getting an exception when running tests using VSTest (I can't switch to dotnet test approach, unfortunatly - there is a strong requirement to stick to vstest)

vstest.console.exe --collect:"XPlat Code Coverage" C:\Projects\test\Web.Tests\bin\Debug\Web.Tests.dll /TestAdapterPath:C:\.nuget\packages\coverlet.collector\1.3.0\build --Diag:"C:\Projects\diag.txt"

I get the following warning in the console:

Data collection : Unable to find a datacollector with friendly name 'XPlat code coverage'.
Data collection : Could not find data collector 'XPlat code coverage'

And in diag.txt file I see:

datacollector.exe, TestPluginDiscoverer: Failed to get types from assembly 'coverlet.collector, Version=1.3.0.0, Culture=neutral, PublicKeyToken=4beb1ad7f05a1c71'.  Skipping test extension scan for this assembly.  Error: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.Assembly.GetTypes()
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginDiscoverer.GetTestExtensionsFromAssembly[TPluginInfo,TExtension](Assembly assembly, Dictionary`2 pluginInfos)
TpTrace Warning: 0 : 60664, 1, 2020/11/14, 22:42:00.029, 8944115731653, datacollector.exe, LoaderExceptions: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
File name: 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
File name: 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

I followed the discussion mentioned on microsoft/vstest#2278 but it seems like a separate issue.
It feels I am missing some type of configuration.

I can't figure out the source of the problem. Any help is highly appreciated. Thanks in advance.

[MarcoRossignoli]

Do you have net4x target framework on your projects?

Can you try with the temporary version on my "debug" feed? https://f.feedz.io/marcorossignoli/coverletunofficial/nuget/index.json 3.0.0-preview.7 version

[arabelaa]

I added the Framework parameter --Framework:".NETCoreApp,Version=v3.1"

and now I am getting

TpTrace Error: 0 : 63980, 4, 2020/11/16, 12:28:41.887, 10304255399683, testhost.dll, InProcDataCollectionExtensionManager: Error occurred while loading the InProcDataCollector : coverlet.collector.dll , Exception Details : System.ArgumentException: Absolute path information is required. (Parameter 'assemblyPath')
   at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath)
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformAssemblyLoadContext.LoadAssemblyFromPath(String assemblyPath)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.InProcDataCollector.LoadInProcDataCollectorExtension(String codeBase)
TpTrace Information: 0 : 63980, 4, 2020/11/16, 12:28:41.887, 10304255403083, testhost.dll, AssemblyResolver.AddSearchDirectories: Adding more searchDirectories 
TpTrace Error: 0 : 63980, 4, 2020/11/16, 12:28:41.888, 10304255409679, testhost.dll, InProcDataCollectionExtensionManager: Error occurred while Initializing the datacollectors : System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.InProcDataCollector.CreateObjectFromType(Type type)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.InProcDataCollector.LoadDataCollector(IDataCollectionSink inProcDataCollectionSink)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.InProcDataCollectionExtensionManager.CreateDataCollector(String assemblyQualifiedName, String codebase, XmlElement configuration, TypeInfo interfaceTypeInfo)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.InProcDataCollectionExtensionManager.InitializeInProcDataCollectors(String runSettings)

[MarcoRossignoli]

The error above seem related to .NET Framework test process that tries to load .NET Core coverlet collectors. This is not supported in coverlet until version 3.0.0(not released yet).
Since next version collectors will be a netstandard one so should be usable by a .NET Framework runtime(.NET core assemblies are different from .NET Framework one, so the resolution errors).

[arabelaa]

I used the nuget from the suggested feed https://f.feedz.io/marcorossignoli/coverletunofficial/nuget/index.json and I no longer see the exception.

But now I have other errors with datacollector.exe (our tests create a kestrel process and we assert the response we get from an API endpoint)

TpTrace Warning: 0 : 61228, 1, 2020/11/16, 19:08:19.453, 10544031072556, datacollector.exe, [coverlet]Unable to resolve method reference "System.Threading.Tasks.Task`1<!0> System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.Collections.Generic.IList`1<AssetDto>>::get_Task()", assuming calls to will return
TpTrace Verbose: 0 : 61228, 1, 2020/11/16, 19:08:19.453, 10544031075300, datacollector.exe, [coverlet]TryWithCustomResolverOnDotNetCore for System.Threading.Tasks, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
TpTrace Verbose: 0 : 61228, 1, 2020/11/16, 19:08:19.453, 10544031075781, datacollector.exe, [coverlet]Not a dotnet core app

------------------------------------------------------------------------------------------------------------------------------------
TpTrace Warning: 0 : 61228, 1, 2020/11/16, 19:08:24.188, 10544078419027, datacollector.exe, [coverlet]Unable to resolve method reference "!0 System.Nullable`1<System.DateTime>::GetValueOrDefault()", assuming calls to will return

[arabelaa]

Looking over #991 I realized that you don't support API tests (integration tests) coverage. And this will be released in 3.0.0 release. But it still feel strange to see this errors.

[MarcoRossignoli]

Seems that there is an issue resolving one assembly System.Threading.Tasks, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
You app is not a .NET Core one so coverlet cannot try to resolve browsing SDK installed on machine, can you try to copy that assembly on publish folder?

Last error on GetValueOrDefault() is a warning no prob for it.

[arabelaa]

Our app is a .NET Core application. The problem is that the system under test is an API which is hosted in kestrel. And this is not yet supported by coverlet. Do you have a release date for 3.0.0 version?

[MarcoRossignoli]

The problem is that the system under test is an API which is hosted in kestrel. And this is not yet supported by coverlet.

It depends, if the test process opens a port and there some call on it, should work. I mean a scenario like dotnet test where the test open port and some client inside test call on that should work with no problem. Can you describe your scenario?

[arabelaa]

Our test process launches a dotnet process using ProcessStartInfo.Start with some launchSettings arguments

 "profiles": {
    "Kestrel": {
      "commandName": "Project",
      "launchBrowser": true,
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      },
      "use64Bit": true,
      "applicationUrl": "https://localhost:6000"
    } 

[MarcoRossignoli]

Ok if the process hosts instrumented assemblies should work..I mean if test keeps the process alive and shutdown at the end and the ProcessExit will fire hits should be collected. Btw it's hard to understand without a complete picture, if you can create a "repro" I can take a look.

[arabelaa]

The test project has integration tests (that call a web application - which is hosted in IIS or Kestrel depending on a configuration).

First question: Is IIS scenario supported? There is a configuration in .runsettings that allows to collect from childProcesses.

<CollectFromChildProcesses>True</CollectFromChildProcesses>
 <!-- When set to True, restarts the IIS process and collects coverage information from it. -->
<CollectAspDotNet>True</CollectAspDotNet>

For Kestrel we start the process when the test starts. Something along this lines

var url = "http://localhost:2345";
string dotnetDir = Environment.GetEnvironmentVariable("DOTNET_INSTALL_DIR") ?? Path.Combine(Environment.GetEnvironmentVariable("ProgramW6432"), "dotnet");
 var dotnet = Path.Combine(dotnetDir, "dotnet.exe");
 var project = Path.Combine("src", "Web", "Web.Host", "bin", "Web.dll");
 var processToStart = new Process
 {
   StartInfo = new ProcessStartInfo()
    {
       FileName = dotnet,
       Arguments = $@"{project} --urls ""{url}""",
       UseShellExecute = false,
       CreateNoWindow = false
    }
 };
 processToStart.Start();

When running
.\vstest.console.exe /settings:C:\CodeCoverage.runsettings C:\bin\Debug\Web.Tests.dll /diag:C:\Projects\Diag.txt /TestAdapterPath:C:\.nuget\packages\coverlet.collector\3.0.0-preview.9\build

we see no code coverage for the API's.
Is my understanding correct that with 991 the behavior I mentioned will be supported? Or this should already work and I am missing some configuration?

[MarcoRossignoli]

Can you share /diag files(be sure to not share secrets)
Can you also enable tracker logger to understand how hits are collected https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Troubleshooting.md#enable-injected-tracker-log

Is my understanding correct that with 991 the behavior I mentioned will be supported? Or this should already work and I am missing some configuration?

Will be supported but not through dotnet test command you should publish file somewhere and after use .NET tool to spin up process that host asm to cover.
Take a look here https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/GlobalTool.md#code-coverage-for-integration-tests-and-end-to-end-tests

It's not related to host process it's related to instrumentation phase, I try to explain better how coverlet workflow works:

1. in case of collectors/msbuild integration msbuild or out of process collectors(orchestrated by vstest plat) will instrument all dll(applying passed parameters).
2. host process load dll and run tests
3. when ProcessExit will fire for test host process hits will be persisted to disk.

So it's not a problem the "host process" if ProcessExit event will fire on it...if for some reason a "kill" is used no hits will be collected.

Your use case is particular, I mean you do integration using a test framework(usually I mock server for instance using https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-5.0#aspnet-core-integration-tests). Anyway coverlet can collect hits from concurrent process and also with old app domain, so the only problem I see is a premature kill of process(if you kill after that processToStart.Start(); at the end of test and not shutdown it gracefully, kill process prevent process exit event to fire).

[arabelaa]

Let me see if I got this right.

If I have a .NET Core app hosted in IIS(w3wp.exe)/kestrel and I have a suite of integration tests I should be able to collect code coverage using coverlet (the preview version which support .netstandard targetframework) - because the IIS process has the dll&pdb and they are available for instrumentation - even if the IIS process was no started by the test process.

I used the tracker logger but the file was not generated for the module I was interested in instrumenting (it's related to the amount of errors in diagnostics file).
I've attached the logs from host and datacollector after running the vstest command (with IIS process - that was started upfront and not during the test execution).
Note: the same behavior I have with kestrel.

diag.host.20-11-27_11-09-50_76174_4.txt
diag.datacollector.20-11-27_11-09-24_20487_4.txt

[MarcoRossignoli]

Ok thanks for help on investigation...I see only 2 asm instrumented

TpTrace Verbose: 0 : 11588, 1, 2020/11/27, 11:09:28.725, 3383297749099, datacollector.exe, [coverlet]Instrumented module: 'C:\Projects\Testing\test\Web.Tests\bin\Debug\Resources.dll'
TpTrace Verbose: 0 : 11588, 1, 2020/11/27, 11:09:25.403, 3383264537040, datacollector.exe, [coverlet]Instrumented module: 'C:\Projects\Testing\test\Web.Tests\bin\Debug\System.Reactive.Linq.dll'

And a lot of resolution errors

TpTrace Verbose: 0 : 11588, 1, 2020/11/27, 11:09:28.746, 3383297961850, datacollector.exe, [coverlet]TryWithCustomResolverOnDotNetCore for System.Threading.Tasks, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a

Also I don't see in-process collector loaded(host log file).

Are you able to provide a similar repo?I think it's a corner case and need to investigate further.
Or you can try with beta package and coverlet tool https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-5.0#aspnet-core-integration-tests but it won't follow the dotnet test workflow so you need to change how you're doing the integration tests.