Add publish pipeline extension hooks for Xamarin #11073

akoeplinger · 2020-03-31T19:06:52Z

We'll need some hooks in the Publish pipeline for Xamarin workloads in .NET 5.
I added some extension points that the Xamarin SDKs can inject their targets into, these are the result of an ongoing investigation as part of our .NET 5 work.

The end goal is to be able to do for example dotnet publish -r ios-x64 --self-contained true -p:PublishTrimmed=true and have the tooling produce the final app bundle that can be deployed to the device.

Note: These names are an initial proposal but as I'm not familiar with the sdk codebase please feel free to suggest better names or other improvements :)

/cc @dsplaisted

marek-safar · 2020-03-31T19:24:16Z

/cc @rainersigwald

dsplaisted · 2020-03-31T20:49:07Z

I'm not sure we should have mono-specific targets / extension points here.

@swaroop-sridhar is probably the best person to give feedback on how iOS and Android should hook into this part of the build process.

swaroop-sridhar · 2020-03-31T22:43:42Z

src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Publish.targets

 
    The main publish entry point.
+
+    Extension hooks for injecting custom targets:


@akoeplinger Can you please provide more details about the architecture of the Mono extension?
Please post a pointer if there's a doc about it, thanks.

I have the following main questions:

Where do the Mono-specific targets live (ex: CoreMonoAOTCompilation)?

Why do we need the extensions in the SDK? How are they expected to be used by the Mono build?
Can it be handled via Before/AfterTargest in Mono-specific files?

swaroop-sridhar · 2020-03-31T22:45:15Z

src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Publish.targets

    -->
  <Target Name="CreateReadyToRunImages"
-          Condition="'$(_TargetFrameworkVersionWithoutV)' >= '3.0' And '$(PublishReadyToRun)' == 'true' And '$(TargetFrameworkIdentifier)' == '.NETCoreApp'"
+          Condition="'$(_TargetFrameworkVersionWithoutV)' >= '3.0' And '$(PublishReadyToRun)' == 'true' And '$(PublishMonoAOT)' != 'true' And '$(TargetFrameworkIdentifier)' == '.NETCoreApp'"


I don't think CreateReadyToRunImages should be conditioned on PublishMonoAOT settings (and fail silently).
I think there should be a check in the early validation to throw an error if CreateReadyToRunImages and PublishMonoAOT are both true.

swaroop-sridhar · 2020-03-31T22:45:54Z

src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Publish.targets

  </Target>

+  <Target Name="MonoAOTCompilation"
+          Condition="'$(_TargetFrameworkVersionWithoutV)' >= '5.0' And '$(PublishReadyToRun)' != 'true' And '$(PublishMonoAOT)' == 'true' And '$(TargetFrameworkIdentifier)' == '.NETCoreApp'"


Similar to the above comment, I don't think PublishReadyToRun should be checked here.

swaroop-sridhar · 2020-03-31T22:49:23Z

src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Publish.targets

+      - BeforeComputeFilesToPublishTargets
+      - AfterComputeFilesToPublishTargets
+      - BeforeILLinkTargets
+      - AfterILLinkTargets


If we add public extension hooks, it could be used for any arbitrary targets by users.
We should have good testing to ensure this case is well supported.

dsplaisted · 2020-04-01T00:01:19Z

@swaroop-sridhar Here is some background info on how iOS and Android will ship:

So the Mono specific build logic won't be imported as a package into the SDK repo. Rather it will be an optional workload, and will be imported at build time by the .NET SDK for projects where the TargetFramework specifies iOS or Android.

swaroop-sridhar · 2020-04-01T06:59:11Z

@swaroop-sridhar Here is some background info on how iOS and Android will ship:

.NET SDK workloads overview spec

Workload resolution in MSBuild

So the Mono specific build logic won't be imported as a package into the SDK repo. Rather it will be an optional workload, and will be imported at build time by the .NET SDK for projects where the TargetFramework specifies iOS or Android.

Thanks a lot for sharing the links @dsplaisted. I'll go through the specs in more detail tomorrow, but I understand the motivation for this change now. Please disregard my comments about importing the Mono targets as a package.

There are currently two "verbs" we are aiming to get working in Xamarin.Android: dotnet build dotnet publish Currently in .NET Core (aka .NET 5), `dotnet publish` is where all the work to produce a self-contained "app" happens: * The linker via the `<IlLink/>` MSBuild task * .NET Core's version of AOT, named "ReadyToRun" https://docs.microsoft.com/en-us/dotnet/core/whats-new/dotnet-core-3-0#readytorun-images This means Xamarin.Android would run the following during `dotnet build`: * Run `aapt` to generate `Resource.designer.cs` and potentially emit build errors for issues in `@(AndroidResource)` files. * Compile C# code Almost everything else happens during `donet publish`: * Generate java stubs, `AndroidManifest.xml`, etc. This must happen after the linker. * Compile java code via `javac` * Convert java code to `.dex` via d8/r8 * Create an `.apk` and sign it This is highly subject to change, however. To get everything working: * I moved any `$(FooDependsOn)` properties that determine MSBuild target ordering for "legacy" projects to `Xamarin.Android.Legacy.targets`. * The equivalent for .NET 5 will be in `Xamarin.Android.Sdk.BuildOrder.targets`. * I modified the private `$(_CorePublishTargets)` property to run Xamarin.Android-specific targets after `<IlLink/>`. This will change after this PR lands: dotnet/sdk#11073 * I moved the existing `_ResolveAssemblies` MSBuild target to `Xamarin.Android.Legacy.targets`. * The .NET 5 version of `_ResolveAssemblies` should be able to make use of item groups provided by MSBuild. This file lives in `Xamarin.Android.Sdk.AssemblyResolution.targets`. It will eventually change when we have Mono runtime packs for .NET 5. Future changes: * We will need to use different item groups in the `_ResolveAssemblies` target once a Mono runtime pack for .NET 5 is available. * `dotnet publish` still uses the current Mono linker. We will need to refactor and integrate the new linker documented at: https://github.com/mono/linker/tree/ec2cbdb894e11065f42b1223f664d9688b749323/src/linker#adding-custom-steps-to-the-linker

swaroop-sridhar · 2020-04-02T01:15:51Z

@akoeplinger I agree with @dsplaisted that we shouldn't have mono-specific targets in this file. We could have generic extensions that manipulate the computation of files to publish, that are used by the Mono workload. Once you answer some of the questions in the above review, it'll help determine the exact extensions necessary. Thanks.

There are currently two "verbs" we are aiming to get working in Xamarin.Android: dotnet build dotnet publish Currently in .NET Core (aka .NET 5), `dotnet publish` is where all the work to produce a self-contained "app" happens: * The linker via the `<IlLink/>` MSBuild task * .NET Core's version of AOT, named "ReadyToRun" https://docs.microsoft.com/en-us/dotnet/core/whats-new/dotnet-core-3-0#readytorun-images This means Xamarin.Android would run the following during `dotnet build`: * Run `aapt` to generate `Resource.designer.cs` and potentially emit build errors for issues in `@(AndroidResource)` files. * Compile C# code Almost everything else happens during `donet publish`: * Generate java stubs, `AndroidManifest.xml`, etc. This must happen after the linker. * Compile java code via `javac` * Convert java code to `.dex` via d8/r8 * Create an `.apk` and sign it This is highly subject to change, however. To get everything working: * I moved any `$(FooDependsOn)` properties that determine MSBuild target ordering for "legacy" projects to `Xamarin.Android.Legacy.targets`. * The equivalent for .NET 5 will be in `Xamarin.Android.Sdk.BuildOrder.targets`. * I modified the private `$(_CorePublishTargets)` property to run Xamarin.Android-specific targets after `<IlLink/>`. This will change after this PR lands: dotnet/sdk#11073 * I moved the existing `_ResolveAssemblies` MSBuild target to `Xamarin.Android.Legacy.targets`. * The .NET 5 version of `_ResolveAssemblies` should be able to make use of item groups provided by MSBuild. This file lives in `Xamarin.Android.Sdk.AssemblyResolution.targets`. It will eventually change when we have Mono runtime packs for .NET 5. Future changes: * We will need to use different item groups in the `_ResolveAssemblies` target once a Mono runtime pack for .NET 5 is available. * `dotnet publish` still uses the current Mono linker. We will need to refactor and integrate the new linker documented at: https://github.com/mono/linker/tree/ec2cbdb894e11065f42b1223f664d9688b749323/src/linker#adding-custom-steps-to-the-linker

akoeplinger · 2020-04-02T16:50:37Z

This needs some more work based on recent discussions with @dsplaisted and others, I'll come back later when that is ready, thanks!

There are currently two "verbs" we are aiming to get working in Xamarin.Android: dotnet build dotnet publish Currently in .NET Core (aka .NET 5), `dotnet publish` is where all the work to produce a self-contained "app" happens: * The linker via the `<IlLink/>` MSBuild task * .NET Core's version of AOT, named "ReadyToRun" https://docs.microsoft.com/en-us/dotnet/core/whats-new/dotnet-core-3-0#readytorun-images This means Xamarin.Android would run the following during `dotnet build`: * Run `aapt` to generate `Resource.designer.cs` and potentially emit build errors for issues in `@(AndroidResource)` files. * Compile C# code Almost everything else happens during `donet publish`: * Generate java stubs, `AndroidManifest.xml`, etc. This must happen after the linker. * Compile java code via `javac` * Convert java code to `.dex` via d8/r8 * Create an `.apk` and sign it This is highly subject to change, however. To get everything working: * I moved any `$(FooDependsOn)` properties that determine MSBuild target ordering for "legacy" projects to `Xamarin.Android.Legacy.targets`. * The equivalent for .NET 5 will be in `Xamarin.Android.Sdk.BuildOrder.targets`. * I modified the private `$(_CorePublishTargets)` property to run Xamarin.Android-specific targets after `<IlLink/>`. This will change after this PR lands: dotnet/sdk#11073 * I moved the existing `_ResolveAssemblies` MSBuild target to `Xamarin.Android.Legacy.targets`. * The .NET 5 version of `_ResolveAssemblies` should be able to make use of item groups provided by MSBuild. This file lives in `Xamarin.Android.Sdk.AssemblyResolution.targets`. It will eventually change when we have Mono runtime packs for .NET 5. Future changes: * We will need to use different item groups in the `_ResolveAssemblies` target once a Mono runtime pack for .NET 5 is available. * `dotnet publish` still uses the current Mono linker. We will need to refactor and integrate the new linker documented at: https://github.com/mono/linker/tree/ec2cbdb894e11065f42b1223f664d9688b749323/src/linker#adding-custom-steps-to-the-linker

Add publish pipeline extension hooks

b10a49a

swaroop-sridhar reviewed Mar 31, 2020

View reviewed changes

jonathanpeppers mentioned this pull request Apr 1, 2020

[.NET 5] initial implementation of "dotnet publish" dotnet/android#4498

Merged

akoeplinger closed this Apr 2, 2020

akoeplinger deleted the publish-target-hooks branch November 14, 2022 13:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add publish pipeline extension hooks for Xamarin #11073

Add publish pipeline extension hooks for Xamarin #11073

Uh oh!

akoeplinger commented Mar 31, 2020

Uh oh!

marek-safar commented Mar 31, 2020

Uh oh!

dsplaisted commented Mar 31, 2020

Uh oh!

swaroop-sridhar Mar 31, 2020

Uh oh!

swaroop-sridhar Mar 31, 2020

Uh oh!

swaroop-sridhar Mar 31, 2020

Uh oh!

swaroop-sridhar Mar 31, 2020

Uh oh!

dsplaisted commented Apr 1, 2020

Uh oh!

swaroop-sridhar commented Apr 1, 2020

Uh oh!

swaroop-sridhar commented Apr 2, 2020

Uh oh!

akoeplinger commented Apr 2, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		The main publish entry point.

		Extension hooks for injecting custom targets:

Add publish pipeline extension hooks for Xamarin #11073

Add publish pipeline extension hooks for Xamarin #11073

Uh oh!

Conversation

akoeplinger commented Mar 31, 2020

Uh oh!

marek-safar commented Mar 31, 2020

Uh oh!

dsplaisted commented Mar 31, 2020

Uh oh!

swaroop-sridhar Mar 31, 2020

Choose a reason for hiding this comment

Uh oh!

swaroop-sridhar Mar 31, 2020

Choose a reason for hiding this comment

Uh oh!

swaroop-sridhar Mar 31, 2020

Choose a reason for hiding this comment

Uh oh!

swaroop-sridhar Mar 31, 2020

Choose a reason for hiding this comment

Uh oh!

dsplaisted commented Apr 1, 2020

Uh oh!

swaroop-sridhar commented Apr 1, 2020

Uh oh!

swaroop-sridhar commented Apr 2, 2020

Uh oh!

akoeplinger commented Apr 2, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants