[Alpine Linux] `s390x` platform support

[ayakael]

Description

Dotnet6 on Fedora supports s390x. Dotnet6 on Alpine should do so, too.

Parallel thread on Alpine Gitlab

Major roadblocks

As there are no dotnet6 prebuilt bootstraps release for s390x, we have to build our own by crosscompiling from x86_64 to s390x. Crosscompilation scripts was kindly provided by the Fedora team, and added to a git repo on Apine's Gitlab instance for further work.

Hypothetical steps

1. clone aports, and cd aports/community/dotnet6-stage0
2. abuild deps to install build dependencies of dotnet
3. apk add quilt mono dotnet6-sdk
4. git clone https://gitlab.alpinelinux.org/ayakael/dotnet6-s390x (original version here)
5. cd dotnet6-s390x && ./dotnet-s390x-prepare
6. ./dotnet-s390x-build

cc. @nekopsykose @uweigand @omajid

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ayakael]

Current issue: As is currently the case, crosscompilation of runtime fails due to clang13 not having elf64_s390 emulation. How to get this working? Having no experience of crosscompilation, I'd appreciate if anyone has pointers for this.

[ayakael]

I made another attempt. Considering no s390x support from lld, I set LD=s390x-linux-gnu-ld and tried making a binutils-cross package for s390x by modifying binutils-cross-embedded aport to target s390x-linux-gnu. New errros:

  Invoking "/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/eng/native/gen-buildsys.sh" "/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/src/coreclr" "/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release" s390x clang "" "0" Release ""  -DCLR_CMAKE_PGO_INSTRUMENT=0 -DCLR_CMAKE_OPTDATA_PATH= -DCLR_CMAKE_PGO_OPTIMIZE=1 -DFEATURE_DISTRO_AGNOSTIC_SSL=1 
  WARN: Specific version of clang not found, falling back to use the one in PATH.
  ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/src/coreclr
  Not searching for unused variables given on the command line.
  loading initial cache file /var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/eng/native/tryrun.cmake
  -- The C compiler identification is Clang 13.0.1
  -- The CXX compiler identification is Clang 13.0.1
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - failed
  -- Check for working C compiler: /usr/bin/clang
  -- Check for working C compiler: /usr/bin/clang - broken
  CMake Error at /usr/share/cmake/Modules/CMakeTestCCompiler.cmake:69 (message):
    The C compiler
  
      "/usr/bin/clang"
  
    is not able to compile a simple test program.
  
    It fails with the following output:
  
      Change Dir: /var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeTmp
      
      Run Build Command(s):/usr/bin/make -f Makefile cmTC_30712/fast && /usr/bin/make  -f CMakeFiles/cmTC_30712.dir/build.make CMakeFiles/cmTC_30712.dir/build
      make[1]: Entering directory '/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeTmp'
      Building C object CMakeFiles/cmTC_30712.dir/testCCompiler.c.o
      /usr/bin/clang --target=s390x-linux-gnu --gcc-toolchain=//usr --sysroot=/    -MD -MT CMakeFiles/cmTC_30712.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_30712.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_30712.dir/testCCompiler.c.o -c /var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeTmp/testCCompiler.c
      Linking C executable cmTC_30712
      /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_30712.dir/link.txt --verbose=1
      /usr/bin/clang --target=s390x-linux-gnu --gcc-toolchain=//usr --sysroot=/ -Wl,--rpath-link=//lib/s390x-linux-gnu -Wl,--rpath-link=//usr/lib/s390x-linux-gnu -Wl,--rpath-link=//lib/s390x-linux-gnu -Wl,--rpath-link=//usr/lib/s390x-linux-gnu  CMakeFiles/cmTC_30712.dir/testCCompiler.c.o -o cmTC_30712 
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)
      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: error adding symbols: file in wrong format
      clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
      make[1]: *** [CMakeFiles/cmTC_30712.dir/build.make:100: cmTC_30712] Error 1
      make[1]: Leaving directory '/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeTmp'
      make: *** [Makefile:127: cmTC_30712/fast] Error 2
      
      
  
    
  
    CMake will not be able to correctly generate this project.
  Call Stack (most recent call first):
    CMakeLists.txt:16 (project)
  
  
  -- Configuring incomplete, errors occurred!
  See also "/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeOutput.log".
  See also "/var/build/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release/CMakeFiles/CMakeError.log".
  ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/src/coreclr
  ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/artifacts/obj/coreclr/Linux.s390x.Release ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/src/coreclr
  Executing make   iltools  -j 24
  make: *** No rule to make target 'iltools'.  Stop.
  ~/dotnet6-s390x/community/dotnet6-stage0/dotnet-s390x/runtime/src/coreclr
  Failed to build "CoreCLR component".

[omajid]

cc @BahaVv

[uweigand]

This seems to be a problem with the cross-compile environment.

Specifically:

      /usr/bin/s390x-linux-gnu-ld: /usr/bin/../lib/crt1.o: Relocations in generic ELF (EM: 62)

EM 62 is EM_X86_64, which the s390x specific ld doesn't understand. This is probably a problem with the cross-compiler configuration, since the compiler apparently attempts to pull in the host version of crt1.o into a cross-link, instead of the appropriate target version.

crt1.o is part of the libc package, so the question is whether/where you've installed the target libc. In my original Ubuntu-based scripts, multiple architecture versions of libc (and other libraries) can be installed into the same root directory, because of Debian's multi-architecture file system layout. This doesn't work on most other distros, so there you usually have to use something like a sysroot environment.

I see in the repo you've linked above that in dotnet-s390x-build you have this line:

ROOTFS_DIR=/ ./build.sh -arch s390x -c ${RUNTIME_CONF} -cross -clang

ROOTFS_DIR=/ is correct on Ubuntu because of the multi-arch file system layout mentioned above. On other distros, this is nearly certain to be incorrect. Instead, you'll have to create a sysroot directory containing target libraries somewhere, and then point ROOTFS_DIR to that directory.

[ayakael]

That strikes me as making total sense. I'll explore this further and come back to you.

[ayakael]

Okay, finally found time to setup a crossbuild environement on Alpine. I've managed to get parts of runtime built, but musl is apparently utterly unhappy with dotnet's mono. Many errors such as function-like macro '__GLIBC_PREREQ' is not defined and variable 'is_enum' set but not used [-Wunused-but-set-variable]. I assume I need to go through the build and turn all Werrors off, as I don't see any deal breaker erros.

[uweigand]

Okay, finally found time to setup a crossbuild environement on Alpine. I've managed to get parts of runtime built, but musl is apparently utterly unhappy with dotnet's mono. Many errors such as function-like macro '__GLIBC_PREREQ' is not defined and variable 'is_enum' set but not used [-Wunused-but-set-variable]. I assume I need to go through the build and turn all Werrors off, as I don't see any deal breaker erros.

Yes, it's never been built against musl, so there may well be some issues. The __GLIBC_PREREQ most likely comes from this:

#elif defined(__s390x__)

#define MONO_ARCH_HAS_MONO_CONTEXT 1

#include <sys/ucontext.h>

#if __GLIBC_PREREQ(2, 26)
typedef ucontext_t MonoContext;
#else
typedef struct ucontext MonoContext;
#endif

so this needs to be fixed for musl.