[Bug]: Build succeeds even though Errors were raised

[eerhardt]

Issue Description

As part of the dotnet/aspnetcore test build, we are publishing test apps and running them during the build. If the test app doesn't return the correct ExitCode, we fail the build.

However, we've run into situations where raising this Error doesn't fail the build, even though it should. This happens in CI, but we haven't found a case where it happens locally - if the test fails, the build fails locally.

Steps to Reproduce

See the binlog here: aspnetcore-succeeds-buterror.binlog

Here is the MSBuild code raising an error:

https://github.com/dotnet/aspnetcore/blob/81666eef98bd9e8c4d95c1478ebe4d4ba77e61a3/eng/testing/linker/trimmingTests.targets#L121-L125

Expected Behavior

The whole build should fail because an Error occurred during the build.

Actual Behavior

Analysis

No response

Versions & Configurations

No response

[JanKrivanek]

[5/23 9:01 PM] Andrew Casey

Here's a run where the error caused a build failure on mac but not on windows or linux

https://dev.azure.com/dnceng-public/public/_build/results?buildId=283737&view=results

[5/23 9:03 PM] Andrew Casey

binlogs are in MacOS_Test_Logs/Release/build.binlog and Linux_Test_logs/Release/build.binlog in the Artifacts view