Cannot test on Arm64 : Could not copy crossgen2_publish apphost

[a74nh]

With latest HEAD. On Arm64 Linux.

rm -fr artifacts/
./build.sh -rc' checked -lc release -s clr+libs
./src/tests/build.sh generatelayoutonly Checked

Errors with:

/home/alahay01/dotnet/runtime_base/.dotnet/sdk/9.0.100-preview.7.24407.12/Microsoft.Common.CurrentVersion.targets(5327,5): error MSB3030: Could not copy the file "/home/alahay01/dotnet/runtime_base/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/apphost" because it was not found. [/home/alahay01/dotnet/runtime_base/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]

Related to #106965 ??

This stops me from building and running the testsuite.

[dotnet-policy-service]

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

[a74nh]

Full generatelayoutonly log:

❯ ./src/tests/build.sh generatelayoutonly Checked
__OutputRid: linux-arm64
Building Tests...
__TargetOS: linux
__TargetArch: arm64
__BuildType: Checked
__TestIntermediatesDir: /home/alahay01/dotnet/runtime_base/artifacts/tests/coreclr/obj/linux.arm64.Checked
__NativeTestIntermediatesDir: /home/alahay01/dotnet/runtime_base/artifacts/tests/coreclr/obj/linux.arm64.Checked/Native
__ManagedTestIntermediatesDir: /home/alahay01/dotnet/runtime_base/artifacts/tests/coreclr/obj/linux.arm64.Checked/Managed
Creating TestBinDir: /home/alahay01/dotnet/runtime_base/artifacts/tests/coreclr/linux.arm64.Checked
Creating LogsDir: /home/alahay01/dotnet/runtime_base/artifacts/log
Creating MsbuildDebugLogsDir: /home/alahay01/dotnet/runtime_base/artifacts/log/MsbuildDebugLogs
Building tests via "/home/alahay01/dotnet/runtime_base/eng/common/msbuild.sh"  --warnAsError false /home/alahay01/dotnet/runtime_base/src/tests/build.proj /t:TestBuild /p:TargetArchitecture=arm64 /p:Configuration=Checked /p:TargetOS=linux /nodeReuse:false    /maxcpucount "/flp:Verbosity=normal;LogFile=/home/alahay01/dotnet/runtime_base/artifacts/log/TestBuild.linux.arm64.Checked.log" "/flp1:WarningsOnly;LogFile=/home/alahay01/dotnet/runtime_base/artifacts/log/TestBuild.linux.arm64.Checked.wrn" "/flp2:ErrorsOnly;LogFile=/home/alahay01/dotnet/runtime_base/artifacts/log/TestBuild.linux.arm64.Checked.err" "/bl:/home/alahay01/dotnet/runtime_base/artifacts/log/TestBuild.linux.arm64.Checked.binlog" /p:NUMBER_OF_PROCESSORS=80

  [11:52:08.96] Restoring all packages...

    Determining projects to restore...
    Restored /home/alahay01/dotnet/runtime_base/src/tests/Common/test_dependencies_fs/test_dependencies.fsproj (in 1.2 sec).

    Determining projects to restore...
    Restored /home/alahay01/dotnet/runtime_base/src/tests/Common/test_dependencies/test_dependencies.csproj (in 2.61 sec).

    Determining projects to restore...
    Restored /home/alahay01/dotnet/runtime_base/src/tests/Common/CoreCLRTestLibrary/CoreCLRTestLibrary.csproj (in 2.53 sec).

    Determining projects to restore...
    Restored /home/alahay01/dotnet/runtime_base/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_Arm.csproj (in 220 ms).

    Determining projects to restore...
    Restored /home/alahay01/dotnet/runtime_base/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_General.csproj (in 237 ms).

    Determining projects to restore...
    Restored /home/alahay01/dotnet/runtime_base/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_X86.csproj (in 215 ms).

    Determining projects to restore...
    Restored /home/alahay01/dotnet/runtime_base/src/tests/Common/XUnitLogChecker/XUnitLogChecker.csproj (in 1.77 sec).

    Determining projects to restore...
    Restored /home/alahay01/dotnet/runtime_base/src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.csproj (in 342 ms).

    Determining projects to restore...
    Restored /home/alahay01/dotnet/runtime_base/src/tests/Common/XUnitWrapperLibrary/XUnitWrapperLibrary.csproj (in 218 ms).

    Determining projects to restore...
    Restored /home/alahay01/dotnet/runtime_base/src/tests/Common/XHarnessRunnerLibrary/XHarnessRunnerLibrary.csproj (in 2.15 sec).
    1 of 2 projects are up-to-date for restore.

    Determining projects to restore...
    Restored /home/alahay01/dotnet/runtime_base/src/tests/Common/external/external.csproj (in 2.44 sec).

    Determining projects to restore...
    Restored /home/alahay01/dotnet/runtime_base/src/tests/Common/ilasm/ilasm.ilproj (in 2.28 sec).
  [11:53:02.30] Restoring all packages...Done.
  Building Targeting Pack
  external ->
  Determining projects to restore...
  Restored /home/alahay01/dotnet/runtime_base/src/tools/illink/src/linker/ref/Mono.Linker.csproj (in 430 ms).
  Restored /home/alahay01/dotnet/runtime_base/src/coreclr/tools/aot/ILCompiler.TypeSystem/ILCompiler.TypeSystem.csproj (in 430 ms).
  Restored /home/alahay01/dotnet/runtime_base/src/coreclr/tools/aot/ILCompiler.ReadyToRun/ILCompiler.ReadyToRun.csproj (in 431 ms).
  Restored /home/alahay01/dotnet/runtime_base/src/libraries/System.Runtime.InteropServices/gen/Microsoft.Interop.SourceGeneration/Microsoft.Interop.SourceGeneration.csproj (in 460 ms).
  Restored /home/alahay01/dotnet/runtime_base/src/coreclr/tools/aot/ILCompiler.Diagnostics/ILCompiler.Diagnostics.csproj (in 430 ms).
  Restored /home/alahay01/dotnet/runtime_base/src/coreclr/tools/aot/ILCompiler.DependencyAnalysisFramework/ILCompiler.DependencyAnalysisFramework.csproj (in 430 ms).
  Restored /home/alahay01/dotnet/runtime_base/src/tools/illink/src/ILLink.CodeFix/ILLink.CodeFixProvider.csproj (in 456 ms).
  Restored /home/alahay01/dotnet/runtime_base/src/tools/illink/src/linker/Mono.Linker.csproj (in 443 ms).
  Restored /home/alahay01/dotnet/runtime_base/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj (in 430 ms).
  Restored /home/alahay01/dotnet/runtime_base/src/libraries/System.Runtime.InteropServices/gen/DownlevelLibraryImportGenerator/DownlevelLibraryImportGenerator.csproj (in 450 ms).
  Restored /home/alahay01/dotnet/runtime_base/src/tools/illink/src/ILLink.RoslynAnalyzer/ILLink.RoslynAnalyzer.csproj (in 450 ms).
  Restored /home/alahay01/dotnet/runtime_base/src/tools/illink/src/ILLink.Tasks/ILLink.Tasks.csproj (in 489 ms).
  Restored /home/alahay01/dotnet/runtime_base/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj (in 1.99 sec).
  ILLink.RoslynAnalyzer -> /home/alahay01/dotnet/runtime_base/artifacts/bin/ILLink.RoslynAnalyzer/netstandard2.0/ILLink.RoslynAnalyzer.dll
  ILLink.CodeFixProvider -> /home/alahay01/dotnet/runtime_base/artifacts/bin/ILLink.CodeFixProvider/netstandard2.0/ILLink.CodeFixProvider.dll
  Mono.Linker -> /home/alahay01/dotnet/runtime_base/artifacts/bin/Mono.Linker/ref/net9.0/illink.dll
  Mono.Linker -> /home/alahay01/dotnet/runtime_base/artifacts/bin/Mono.Linker/net9.0/linux-arm64/illink.dll
  ILLink.Tasks -> /home/alahay01/dotnet/runtime_base/artifacts/bin/ILLink.Tasks/net9.0/linux-arm64/ILLink.Tasks.dll
  Mono.Linker -> /home/alahay01/dotnet/runtime_base/artifacts/bin/Mono.Linker/net9.0/illink.dll
  ILLink.Tasks -> /home/alahay01/dotnet/runtime_base/artifacts/bin/ILLink.Tasks/net9.0/ILLink.Tasks.dll
  ILCompiler.DependencyAnalysisFramework -> /home/alahay01/dotnet/runtime_base/artifacts/bin/ILCompiler.DependencyAnalysisFramework/arm64/Checked/ILCompiler.DependencyAnalysisFramework.dll
  ILCompiler.TypeSystem -> /home/alahay01/dotnet/runtime_base/artifacts/bin/ILCompiler.TypeSystem/arm64/Checked/ILCompiler.TypeSystem.dll
  ILCompiler.Diagnostics -> /home/alahay01/dotnet/runtime_base/artifacts/bin/ILCompiler.Diagnostics/arm64/Checked/ILCompiler.Diagnostics.dll
  ILCompiler.ReadyToRun -> /home/alahay01/dotnet/runtime_base/artifacts/bin/ILCompiler.ReadyToRun/arm64/Checked/ILCompiler.ReadyToRun.dll
/home/alahay01/dotnet/runtime_base/.dotnet/sdk/9.0.100-preview.7.24407.12/Microsoft.Common.CurrentVersion.targets(5327,5): error MSB3030: Could not copy the file "/home/alahay01/dotnet/runtime_base/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/apphost" because it was not found. [/home/alahay01/dotnet/runtime_base/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
  Determining projects to restore...
  All projects are up-to-date for restore.

Build FAILED.

/home/alahay01/dotnet/runtime_base/.dotnet/sdk/9.0.100-preview.7.24407.12/Microsoft.Common.CurrentVersion.targets(5327,5): error MSB3030: Could not copy the file "/home/alahay01/dotnet/runtime_base/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/apphost" because it was not found. [/home/alahay01/dotnet/runtime_base/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:01:16.16
Build failed with exit code 1. Check errors above.
Failed to build tests. See the build logs:
    /home/alahay01/dotnet/runtime_base/artifacts/log/TestBuild.linux.arm64.Checked.log
    /home/alahay01/dotnet/runtime_base/artifacts/log/TestBuild.linux.arm64.Checked.wrn
    /home/alahay01/dotnet/runtime_base/artifacts/log/TestBuild.linux.arm64.Checked.err
    /home/alahay01/dotnet/runtime_base/artifacts/log/TestBuild.linux.arm64.Checked.binlog

[am11]

./build.sh -rc' checked -lc release -s clr+libs

-lc should be -c.

Otherwise, it implies -hc Debug, i.e. host, libs and clr are in different configs which is not the intention, right?

[a74nh]

./build.sh -rc' checked -lc release -s clr+libs

-lc should be -c.

Otherwise, it implies -hc Debug, i.e. host, libs and clr are in different configs which is not the intention, right?

That seems to fix it for me.

I guess what I should really be using is:
./build.sh -rc checked -lc release -hc checked-s clr+libs

My question then is has something changed to why this used to work?

[am11]

My question then is has something changed to why this used to work?

We were not requiring apphost before, now it is required by some bogus copy target (which copies apphost just so it stays unused.. long story short; it's another infra bug which needs fixing).

The old and current behaviors are not necessarily wrong. The core of the problem is that build.sh and src/tests/build.sh have different defaults when it comes to configuration. We should streamline them, make an announcement for folks to update their local workflows (which only takes few seconds) and document which combinations are supported. On community-supported platforms, options are further constrained due to lack of official SDK.

e.g. the logical counterpart of ./build.sh -rc Checked -c Debug should be src/tests/build.sh Debug -p:RuntimeConfiguration=Checked, but it is src/tests/build.sh Checked -p:LibrariesConfiguration=Debug. The default of libraires configuration (or any configuration: runtime, tools, host, libraries) is Debug for the top-level build script, but test build's default configuration for libraries is Release. So we had to do extra mapping to streamline things in test infra.

In general, it's a good idea to use one of the workflows exercised by CI (200+ legs) and those are also documented. Those workflows are continuously tested.

[a74nh]

Thanks for the details. Closing this as I just need to update my workflow.