This is related to the use of the RestoreUseStaticGraphEvaluation option which was added in #6905.  This option does not appear to work well with the MSBuild.SourceBuild.slnf.  All of the projects in the main solution appear to get restored instead of only the ones in the filter.  Therefore msbuild has numerous prebuilts coming from various projects that are intended to be excluded from source-build.
source-build msbuild binlog
Related to dotnet/source-build#2626