Use Swift SDKs to generate symbols for Linux targets and armv7 (using swift-sdk-generator)

[xtremekforever]

Swift SDK solution for adding armv7 symbols for swift-crypto using the swift-sdk-generator.

Motivation:

As outlined in #223, the necessary symbols for armv7 are missing from swift-crypto, and we'd like to add them in some way that could be accepted by the team. It was discussed that using Swift SDKs generated from the swift-sdk-generator could be a solution that could be easily vetted, since the swift-sdk-generator is a component maintained by Apple and the swiftlang organization.

Modifications:

• Added a new ./scripts/generate-linux-sdks.sh script to clone & build the swift-sdk-generator, and create Swift SDKs for x86_64, aarch64, and armv7.
• Modified the ./scripts/vendor-boringssl.sh script to use these installed Swift SDKs to cross-compile for Linux architectures in the script.
• Ran the scripts against the exact revision of BoringSSL to avoid source updates, which adds the missing symbols for armv7.

Result:

Swift SDKs are used to generate symbols for Linux architectures instead of Docker, and missing symbols for armv7 are added as well.

I have a few pros and cons for this solution:

• Pros
  • Swift SDKs are used to cross-compile for Linux instead of Docker, which performs better than Rosetta 2 emulation. Docker does not have to be used at all for this solution.
  • Switching to using Swift SDKs is the more "correct" way to cross-compile for Linux targets, and much of the ecosystem for cross-compilation of Swift packages is moving to using Swift SDKs.
• Cons
  • Lots of downloads need to happen since the swift-sdk-generator needs to download the host macOS toolchain for Swift 5.10, plus Ubuntu packages and target toolchains for each architecture (x86_64, aarch64, armv7). This results in ~15GB of artifacts in the /tmp directory for just the generator artifacts. However, this only needs to be run once before running the vendoring script.

Let me know what you think @Lukasa.

[Lukasa]

Shiny, looking really good. A few quick notes.

[Lukasa]

This is very nice, thanks so much @xtremekforever!

[MaxDesiatov]

@xtremekforever what's the plan for upstreaming build scripts that produce these tarballs? IMO these belong to swiftlang/swift with a corresponding preset that potentially could be running on CI to track regressions

[xtremekforever]

I wonder if the best solution could be to provide a separate build.sh script that can be run from the Apple CIs? I would LOVE to upstream this but don't know what steps are the right ones to take. I will need some help.

@MaxDesiatov is this something we could discuss, perhaps on Slack or even in a Discord call? I just don't know how to get this "into" swiftlang/swift without monumental effort.