Skip to content

Switch to objwriter from the dotnet/llvm-project repo #63177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Dec 31, 2021
Merged

Switch to objwriter from the dotnet/llvm-project repo #63177

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fe29344
Switch to objwriter from the dotnet/llvm-project repo
MichalStrehovsky Dec 28, 2021
b8d523f
Update ILCompiler.csproj
MichalStrehovsky Dec 28, 2021
b46c2ea
Update ILCompiler.csproj
MichalStrehovsky Dec 29, 2021
1844e3f
Use official build
MichalStrehovsky Dec 31, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions NuGet.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,6 @@
<add key="dotnet6-transport" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet6-transport/nuget/v3/index.json" />
<!-- Used for the Rich Navigation indexing task -->
<add key="richnav" value="https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-buildservices/nuget/v3/index.json" />
<!-- Used for NativeAOT ObjWriter. TODO: Delete once we publish the package from here. -->
<add key="dotnet-experimental" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-experimental/nuget/v3/index.json" />
</packageSources>
<disabledPackageSources>
<clear />
Expand Down
2 changes: 1 addition & 1 deletion docs/design/coreclr/botr/ilc-architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ While the object file format is highly target specific, the compiler represents
On a high level, the role of the object writer is to go over all the marked `ObjectNode`s in the graph, retrieve their data, defined symbols, and relocations to other symbols, and store them in the object file.

NativeAOT compiler contains multiple object writers:
* Native object writer (`src/coreclr/tools/aot/ObjWriter`) based on LLVM that is capable of producing Windows PE, Linux ELF, and macOS Mach-O file formats
* Native object writer based on LLVM that is capable of producing Windows PE, Linux ELF, and macOS Mach-O file formats
* Native object writer based on LLVM for WebAssembly
* Ready to run object writer that generates mixed CIL/native executables in the ready to run format for CoreCLR

Expand Down
2 changes: 1 addition & 1 deletion docs/workflow/building/coreclr/nativeaot.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ The executable looks like a native executable, in the sense that it can be debug
- [NOT PORTED OVER YET] The build will place the toolchain packages at `artifacts\packages\[Debug|Release]\Shipping`. To publish your project using these packages:
- [NOT PORTED OVER YET] Add the package directory to your `nuget.config` file. For example, replace `dotnet-experimental` line in `samples\HelloWorld\nuget.config` with `<add key="local" value="C:\runtimelab\artifacts\packages\Debug\Shipping" />`
- [NOT PORTED OVER YET] Run `dotnet publish --packages pkg -r [win-x64|linux-x64|osx-64] -c [Debug|Release]` to publish your project. `--packages pkg` option restores the package into a local directory that is easy to cleanup once you are done. It avoids polluting the global nuget cache with your locally built dev package.
- *Optional*. The ObjWriter component of the AOT compiler is not built by default. If you're working on ObjWriter or bringing up a new platform that doesn't have ObjWriter packages yet, as additional pre-requiresites you need to run `build[.cmd|.sh] clr.objwriter` from the repo root before building the product.
- The component that writes out object files (objwriter.dll/libobjwriter.so/libobjwriter.dylib) is based on LLVM and doesn't build in the runtime repo. It gets published as a NuGet package out of the dotnet/llvm-project repo (branch objwriter/12.x). If you're working on ObjWriter or bringing up a new platform that doesn't have ObjWriter packages yet, as additional pre-requiresites you need to build objwriter out of that repo and replace the file in the output.

## Visual Studio Solutions

Expand Down
2 changes: 1 addition & 1 deletion docs/workflow/debugging/coreclr/debugging-aot-compilers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ At the time of writing the current supported sets of valid arguments are:

- The ILC compiler typically compiles the whole program - it loosely corresponds to the composite mode of crossgen2. There is a multifile mode, where each managed assembly corresponds to a single object file, but this mode is not shipping.

- The object files generated by the ILC compiler are written out using an LLVM-based object writer (src\coreclr\tools\aot\ObjWriter). The object writer uses the LLVM assembler APIs (APIs meant to be used by tools that convert textual assembly into machine code) to emit object files in PE/ELF/Mach-O formats. Normally the object writer is not built as part of the repo, but gets downloaded through NuGet. If you need to debug the object writer, you can build it by specifying `clr.objwriter` subset to the root build script. It takes about 5 minutes to compile the object writer.
- The object files generated by the ILC compiler are written out using an LLVM-based object writer (consumed as a NuGet package built out of the dotnet/llvm-project repo, branch objwriter/12.x). The object writer uses the LLVM assembler APIs (APIs meant to be used by tools that convert textual assembly into machine code) to emit object files in PE/ELF/Mach-O formats.

Example of debugging a test application in Crossgen2
================================================
Expand Down
7 changes: 1 addition & 6 deletions eng/Subsets.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@

<PropertyGroup>
<!-- CLR NativeAot only builds in a subset of the matrix -->
<NativeAotSupported Condition="('$(TargetOS)' == 'windows' or '$(TargetOS)' == 'linux' or '$(TargetOS)' == 'OSX') and ('$(TargetArchitecture)' == 'x64' or '$(TargetArchitecture)' == 'arm64') and ('$(TargetOS)' != 'OSX' or '$(TargetArchitecture)' != 'arm64') and ('$(__DistroRid)' != 'linux-musl-arm64')">true</NativeAotSupported>
<NativeAotSupported Condition="('$(TargetOS)' == 'windows' or '$(TargetOS)' == 'linux' or '$(TargetOS)' == 'OSX') and ('$(TargetArchitecture)' == 'x64' or '$(TargetArchitecture)' == 'arm64') and ('$(TargetOS)' != 'OSX' or '$(TargetArchitecture)' != 'arm64')">true</NativeAotSupported>

<!-- If we're building clr.nativeaotlibs and not building the CLR runtime, compile libraries against NativeAOT CoreLib -->
<UseNativeAotCoreLib Condition="$(_subset.Contains('+clr.nativeaotlibs+')) and !$(_subset.Contains('+clr+')) and !$(_subset.Contains('+clr.runtime+'))">true</UseNativeAotCoreLib>
Expand Down Expand Up @@ -107,7 +107,6 @@
<SubsetName Include="Clr.Runtime" Description="The CoreCLR .NET runtime." />
<SubsetName Include="Clr.Native" Description="All CoreCLR native non-test components, including the runtime, jits, and other native tools." />
<SubsetName Include="Clr.NativeAotLibs" Description="The CoreCLR native AOT CoreLib, runtime, and other low level class libraries." />
<SubsetName Include="Clr.ObjWriter" Description="Object writer for the CoreCLR Native AOT compiler." />
<SubsetName Include="Clr.PalTests" OnDemand="true" Description="The CoreCLR PAL tests." />
<SubsetName Include="Clr.PalTestList" OnDemand="true" Description="Generate the list of the CoreCLR PAL tests. When using the command line, use Clr.PalTests instead." />
<SubsetName Include="Clr.Hosts" Description="The CoreCLR corerun test host." />
Expand Down Expand Up @@ -264,10 +263,6 @@
<ProjectToBuild Include="$(CoreClrProjectRoot)nativeaot\**\src\*.csproj" Category="clr" />
</ItemGroup>

<ItemGroup Condition="$(_subset.Contains('+clr.objwriter+'))">
<ProjectToBuild Include="$(CoreClrProjectRoot)tools\aot\ObjWriter\objwriter.proj" Category="clr" />
</ItemGroup>

<!-- Mono sets -->
<ItemGroup Condition="$(_subset.Contains('+mono.llvm+')) or $(_subset.Contains('+mono.aotcross+')) or '$(TargetOS)' == 'iOS' or '$(TargetOS)' == 'iOSSimulator' or '$(TargetOS)' == 'tvOS' or '$(TargetOS)' == 'tvOSSimulator' or '$(TargetOS)' == 'MacCatalyst' or '$(TargetOS)' == 'Android' or '$(TargetOS)' == 'Browser'">
<ProjectToBuild Include="$(MonoProjectRoot)llvm\llvm-init.proj" Category="mono" />
Expand Down
32 changes: 32 additions & 0 deletions eng/Version.Details.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,38 @@
<Uri>https://github.com/dotnet/wcf</Uri>
<Sha>7f504aabb1988e9a093c1e74d8040bd52feb2f01</Sha>
</Dependency>
<Dependency Name="runtime.linux-arm64.Microsoft.NETCore.Runtime.ObjWriter" Version="1.0.0-alpha.1.21630.1">
Copy link
Member Author

@MichalStrehovsky MichalStrehovsky Dec 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should have things set up for Arcade to auto update this. We'll see if it works next time we make an objwriter change.

Also, this would be prettier if there was a lineup package, but Mono's LLVM packages don't have lineup packages and I didn't spend time inventing how to do it with the build infra there.

<Uri>https://github.com/dotnet/llvm-project</Uri>
<Sha>ef5d90bc37b53288e32d78dbf34eae32cd0893d3</Sha>
</Dependency>
<Dependency Name="runtime.linux-x64.Microsoft.NETCore.Runtime.ObjWriter" Version="1.0.0-alpha.1.21630.1">
<Uri>https://github.com/dotnet/llvm-project</Uri>
<Sha>ef5d90bc37b53288e32d78dbf34eae32cd0893d3</Sha>
</Dependency>
<Dependency Name="runtime.linux-musl-arm64.Microsoft.NETCore.Runtime.ObjWriter" Version="1.0.0-alpha.1.21630.1">
<Uri>https://github.com/dotnet/llvm-project</Uri>
<Sha>ef5d90bc37b53288e32d78dbf34eae32cd0893d3</Sha>
</Dependency>
<Dependency Name="runtime.linux-musl-x64.Microsoft.NETCore.Runtime.ObjWriter" Version="1.0.0-alpha.1.21630.1">
<Uri>https://github.com/dotnet/llvm-project</Uri>
<Sha>ef5d90bc37b53288e32d78dbf34eae32cd0893d3</Sha>
</Dependency>
<Dependency Name="runtime.win-arm64.Microsoft.NETCore.Runtime.ObjWriter" Version="1.0.0-alpha.1.21630.1">
<Uri>https://github.com/dotnet/llvm-project</Uri>
<Sha>ef5d90bc37b53288e32d78dbf34eae32cd0893d3</Sha>
</Dependency>
<Dependency Name="runtime.win-x64.Microsoft.NETCore.Runtime.ObjWriter" Version="1.0.0-alpha.1.21630.1">
<Uri>https://github.com/dotnet/llvm-project</Uri>
<Sha>ef5d90bc37b53288e32d78dbf34eae32cd0893d3</Sha>
</Dependency>
<Dependency Name="runtime.osx.11.0-arm64.Microsoft.NETCore.Runtime.ObjWriter" Version="1.0.0-alpha.1.21630.1">
<Uri>https://github.com/dotnet/llvm-project</Uri>
<Sha>ef5d90bc37b53288e32d78dbf34eae32cd0893d3</Sha>
</Dependency>
<Dependency Name="runtime.osx.10.12-x64.Microsoft.NETCore.Runtime.ObjWriter" Version="1.0.0-alpha.1.21630.1">
<Uri>https://github.com/dotnet/llvm-project</Uri>
<Sha>ef5d90bc37b53288e32d78dbf34eae32cd0893d3</Sha>
</Dependency>
</ProductDependencies>
<ToolsetDependencies>
<Dependency Name="Microsoft.DotNet.Arcade.Sdk" Version="7.0.0-beta.21621.3">
Expand Down
8 changes: 8 additions & 0 deletions eng/Versions.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,14 @@
<MicrosoftExtensionsDependencyModelVersion>3.1.0</MicrosoftExtensionsDependencyModelVersion>
<!-- CoreClr dependencies -->
<MicrosoftNETCoreILAsmVersion>7.0.0-alpha.1.21620.1</MicrosoftNETCoreILAsmVersion>
<runtimelinuxarm64MicrosoftNETCoreRuntimeObjWriterVersion>1.0.0-alpha.1.21630.1</runtimelinuxarm64MicrosoftNETCoreRuntimeObjWriterVersion>
<runtimelinuxx64MicrosoftNETCoreRuntimeObjWriterVersion>1.0.0-alpha.1.21630.1</runtimelinuxx64MicrosoftNETCoreRuntimeObjWriterVersion>
<runtimelinuxmuslarm64MicrosoftNETCoreRuntimeObjWriterVersion>1.0.0-alpha.1.21630.1</runtimelinuxmuslarm64MicrosoftNETCoreRuntimeObjWriterVersion>
<runtimelinuxmuslx64MicrosoftNETCoreRuntimeObjWriterVersion>1.0.0-alpha.1.21630.1</runtimelinuxmuslx64MicrosoftNETCoreRuntimeObjWriterVersion>
<runtimewinarm64MicrosoftNETCoreRuntimeObjWriterVersion>1.0.0-alpha.1.21630.1</runtimewinarm64MicrosoftNETCoreRuntimeObjWriterVersion>
<runtimewinx64MicrosoftNETCoreRuntimeObjWriterVersion>1.0.0-alpha.1.21630.1</runtimewinx64MicrosoftNETCoreRuntimeObjWriterVersion>
<runtimeosx110arm64MicrosoftNETCoreRuntimeObjWriterVersion>1.0.0-alpha.1.21630.1</runtimeosx110arm64MicrosoftNETCoreRuntimeObjWriterVersion>
<runtimeosx1012x64MicrosoftNETCoreRuntimeObjWriterVersion>1.0.0-alpha.1.21630.1</runtimeosx1012x64MicrosoftNETCoreRuntimeObjWriterVersion>
<!-- Libraries dependencies -->
<MicrosoftBclAsyncInterfacesVersion>5.0.0</MicrosoftBclAsyncInterfacesVersion>
<MicrosoftWin32PrimitivesVersion>4.3.0</MicrosoftWin32PrimitivesVersion>
Expand Down
35 changes: 16 additions & 19 deletions src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,34 +31,31 @@
</PropertyGroup>

<PropertyGroup>
<ILCompilerVersion>7.0.0-alpha.1.21612.2</ILCompilerVersion>

<ObjWriterBuildType>$(Configuration)</ObjWriterBuildType>
<ObjWriterBuildType Condition="'$(ObjWriterBuildType)' == 'Checked'">Debug</ObjWriterBuildType>

<ObjWriterArtifactPath Condition="'$(TargetsWindows)' == 'true'">$(ArtifactsDir)llvm-project\llvm\build\$(TargetArchitecture)\$(ObjWriterBuildType)\bin\$(LibPrefix)objwriter$(LibSuffix)</ObjWriterArtifactPath>
<ObjWriterArtifactPath Condition="'$(TargetsWindows)' != 'true'">$(ArtifactsDir)llvm-project\llvm\build\$(TargetArchitecture)\lib\$(LibPrefix)objwriter$(LibSuffix)</ObjWriterArtifactPath>
<ObjWriterRid Condition="'$(ObjWriterRid)' == '' and '$(RuntimeIdentifier)' == 'osx-x64'">osx.10.12-x64</ObjWriterRid>
<ObjWriterRid Condition="'$(ObjWriterRid)' == '' and '$(RuntimeIdentifier)' == 'osx-arm64'">osx.11.0-arm64</ObjWriterRid>
<ObjWriterRid Condition="'$(ObjWriterRid)' == ''">$(RuntimeIdentifier)</ObjWriterRid>

<ObjWriterVersion Condition="'$(ObjWriterVersion)' == '' and '$(RuntimeIdentifier)' == 'linux-arm64'">$(runtimelinuxarm64MicrosoftNETCoreRuntimeObjWriterVersion)</ObjWriterVersion>
<ObjWriterVersion Condition="'$(ObjWriterVersion)' == '' and '$(RuntimeIdentifier)' == 'linux-x64'">$(runtimelinuxx64MicrosoftNETCoreRuntimeObjWriterVersion)</ObjWriterVersion>
<ObjWriterVersion Condition="'$(ObjWriterVersion)' == '' and '$(RuntimeIdentifier)' == 'linux-musl-arm64'">$(runtimelinuxmuslarm64MicrosoftNETCoreRuntimeObjWriterVersion)</ObjWriterVersion>
<ObjWriterVersion Condition="'$(ObjWriterVersion)' == '' and '$(RuntimeIdentifier)' == 'linux-musl-x64'">$(runtimelinuxmuslx64MicrosoftNETCoreRuntimeObjWriterVersion)</ObjWriterVersion>
<ObjWriterVersion Condition="'$(ObjWriterVersion)' == '' and '$(RuntimeIdentifier)' == 'win-arm64'">$(runtimewinarm64MicrosoftNETCoreRuntimeObjWriterVersion)</ObjWriterVersion>
<ObjWriterVersion Condition="'$(ObjWriterVersion)' == '' and '$(RuntimeIdentifier)' == 'win-x64'">$(runtimewinx64MicrosoftNETCoreRuntimeObjWriterVersion)</ObjWriterVersion>
<ObjWriterVersion Condition="'$(ObjWriterVersion)' == '' and '$(RuntimeIdentifier)' == 'osx-arm64'">$(runtimeosx110arm64MicrosoftNETCoreRuntimeObjWriterVersion)</ObjWriterVersion>
<ObjWriterVersion Condition="'$(ObjWriterVersion)' == '' and '$(RuntimeIdentifier)' == 'osx-x64'">$(runtimeosx1012x64MicrosoftNETCoreRuntimeObjWriterVersion)</ObjWriterVersion>

<!-- CoreDisTools are used in debugging visualizers. -->
<IncludeCoreDisTools Condition="'$(Configuration)' != 'Release'">true</IncludeCoreDisTools>
</PropertyGroup>

<Import Project="$(RepositoryEngineeringDir)coredistools.targets" Condition="'$(DotNetBuildFromSource)' != 'true' and '$(IncludeCoreDisTools)' == 'true'" />

<ItemGroup Condition="Exists('$(ObjWriterArtifactPath)')">
<Content Include="$(ObjWriterArtifactPath)">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Visible>false</Visible>
<Pack>false</Pack>
</Content>
</ItemGroup>

<ItemGroup Condition="!Exists('$(ObjWriterArtifactPath)')">
<PackageReference Include="Microsoft.DotNet.ILCompiler">
<Version>$(ILCompilerVersion)</Version>
<ItemGroup>
<PackageReference Include="runtime.$(ObjWriterRid).Microsoft.NETCore.Runtime.ObjWriter">
<Version>$(ObjWriterVersion)</Version>
</PackageReference>

<Content Include="$(NuGetPackageRoot)runtime.$(RuntimeIdentifier).microsoft.dotnet.ilcompiler\$(ILCompilerVersion)\tools\$(LibPrefix)objwriter$(LibSuffix)">
<Content Include="$(NuGetPackageRoot)runtime.$(ObjWriterRid).microsoft.netcore.runtime.objwriter\$(ObjWriterVersion)\runtimes\$(ObjWriterRid)\native\$(LibPrefix)objwriter$(LibSuffix)">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Visible>false</Visible>
<Pack>false</Pack>
Expand Down
3 changes: 0 additions & 3 deletions src/coreclr/tools/aot/ObjWriter/.editorconfig
View file
Open in desktop

This file was deleted.

26 changes: 0 additions & 26 deletions src/coreclr/tools/aot/ObjWriter/CMakeLists.txt
View file
Open in desktop

This file was deleted.

7 changes: 0 additions & 7 deletions src/coreclr/tools/aot/ObjWriter/README.md
View file
Open in desktop

This file was deleted.

113 changes: 0 additions & 113 deletions src/coreclr/tools/aot/ObjWriter/build.cmd
View file
Open in desktop

This file was deleted.

Loading