Android code doesn't use emulated TLS

[jmrico01]

Zig Version

0.14.1

Steps to Reproduce and Observed Behavior

It seems like threadlocal variables when compiled for aarch64-linux-android produce code that uses TPIDR_EL0 to access thread local storage, when I believe it should be using emulated TLS via __emutls_get_address like Clang/LLVM does. This is fine for newer Android devices, but causes crashes on earlier Android devices, like a Samsung Galaxy S8 running Android 9 or a Moto G71 running Android 12 (Moto isn't 100% confirmed to be the same root cause). I see TLS emulation already in Zig with code to export the relevant functions for Android - not sure what's missing.

The following minimal snippet reproduces the issue:

threadlocal var tl: u32 = 0;

export fn getValue() u32
{
    return tl;
}

export fn setValue(v: u32) void
{
    tl = v;
}

I'm compiling this with zigup run 0.14.1 build-lib .\sample.zig -target aarch64-linux-android -dynamic -O ReleaseFast. I'm using ReleaseFast to keep the output assembly short, though it still happens on Debug builds. This is the output assembly:

000000000001038c <getValue>:
   1038c:	a9bf7bfd 	stp	x29, x30, [sp,#-16]!
   10390:	910003fd 	mov	x29, sp
   10394:	90000080 	adrp	x0, 20000 <setValue+0xfc4c>
   10398:	f9425801 	ldr	x1, [x0,#1200]
   1039c:	9112c000 	add	x0, x0, #0x4b0
   103a0:	d63f0020 	blr	x1
   103a4:	d53bd048 	mrs	x8, tpidr_el0
   103a8:	b8606900 	ldr	w0, [x8,x0]
   103ac:	a8c17bfd 	ldp	x29, x30, [sp],#16
   103b0:	d65f03c0 	ret

00000000000103b4 <setValue>:
   103b4:	a9bf7bfd 	stp	x29, x30, [sp,#-16]!
   103b8:	910003fd 	mov	x29, sp
   103bc:	2a0003e8 	mov	w8, w0
   103c0:	90000080 	adrp	x0, 20000 <setValue+0xfc4c>
   103c4:	f9425801 	ldr	x1, [x0,#1200]
   103c8:	9112c000 	add	x0, x0, #0x4b0
   103cc:	d63f0020 	blr	x1
   103d0:	d53bd049 	mrs	x9, tpidr_el0
   103d4:	b8206928 	str	w8, [x9,x0]
   103d8:	a8c17bfd 	ldp	x29, x30, [sp],#16
   103dc:	d65f03c0 	ret

Notice the use of tpidr_el0. My current workaround is just to avoid threadlocal variables, which is hard because it includes some stuff in the Zig standard library.

EDIT: I've also tried creating a libc.txt that points to the Android NDK, passing --libc libc.txt to the build-lib command. Also tried ways to tell Zig "compile for Android API 21" but not sure where to do that. Clearly emulated TLS was happening at some point per #5921, maybe even from Zig stdlib code. Not sure what changed.

Expected Behavior

Compare the output above with the equivalent C++ snippet:

thread_local int tl = 0;

extern "C" int getValue()
{
    return tl;
}

extern "C" int setValue(int v)
{
    tl = v;
}

Compiled similarly with zigup run 0.14.1 build-lib .\sample.cpp -target aarch64-linux-android -dynamic -O ReleaseFast, which produces the following more backwards-compatible assembly:

00000000000103f4 <getValue>:
   103f4:	a9bf7bfd 	stp	x29, x30, [sp,#-16]!
   103f8:	910003fd 	mov	x29, sp
   103fc:	90000080 	adrp	x0, 20000 <__emutls_get_address@plt+0xfbc0>
   10400:	f942b000 	ldr	x0, [x0,#1376]
   10404:	9400000f 	bl	10440 <__emutls_get_address@plt>
   10408:	b9400000 	ldr	w0, [x0]
   1040c:	a8c17bfd 	ldp	x29, x30, [sp],#16
   10410:	d65f03c0 	ret

Disassembly of section .plt:

0000000000010420 <__emutls_get_address@plt-0x20>:
   10420:	a9bf7bf0 	stp	x16, x30, [sp,#-16]!
   10424:	90000090 	adrp	x16, 20000 <__emutls_get_address@plt+0xfbc0>
   10428:	f942be11 	ldr	x17, [x16,#1400]
   1042c:	9115e210 	add	x16, x16, #0x578
   10430:	d61f0220 	br	x17
   10434:	d503201f 	nop
   10438:	d503201f 	nop
   1043c:	d503201f 	nop

0000000000010440 <__emutls_get_address@plt>:
   10440:	90000090 	adrp	x16, 20000 <__emutls_get_address@plt+0xfbc0>
   10444:	f942c211 	ldr	x17, [x16,#1408]
   10448:	91160210 	add	x16, x16, #0x580
   1044c:	d61f0220 	br	x17

I'd expect Zig to generate code that uses __emutls_get_address, since that's more backward-compatible and it will cause crashes on real Android devices if you ship an app with native Zig code.

[jmrico01]

I feel like this could be as simple as telling LLVM "use emulated TLS" somehow, at least for LLVM code-gen backend

[alexrp]

It's indeed just a TargetMachine option. Fairly easy to wire up.

[alexrp]

However: https://github.com/llvm/llvm-project/blob/58b939abe5085a750be844cc3c681b40afe98454/llvm/include/llvm/TargetParser/Triple.h#L1150-L1156

This means LLVM should already know to use emulated TLS here. The only way I see that not happening is if we (for some reason) don't lower the Android API level in the LLVM target triple string? And it sure seems like we do:

zig/src/codegen/llvm.zig

Lines 293 to 307 in 75d0ec9

	switch (target.os.versionRange()) {
	.none,
	.semver,
	.windows,
	=> {},
	inline .hurd, .linux => |ver| if (target.abi.isGnu()) {
	try llvm_triple.writer().print("{d}.{d}.{d}", .{
	ver.glibc.major,
	ver.glibc.minor,
	ver.glibc.patch,
	});
	} else if (@TypeOf(ver) == std.Target.Os.LinuxVersionRange and target.abi.isAndroid()) {
	try llvm_triple.writer().print("{d}", .{ver.android});
	},
	}

[jmrico01]

Checking in Godbolt - zig 0.11.0 produces the correct code with __emutls_get_address and 0.12.0 doesn't. Maybe some default Android API level changed, in Zig or in LLVM? I saw the LLVM version-passing code you linked, but I'm not sure how to explicitly pass an Android version down there - wasn't super clear from just the std.Target.Query in the build system.

EDIT: Ooh! I just found 3c1ccbd. My full Android app project with a proper build.zig is still on zig 0.13.0, so I didn't find this new build setting for Android API level. Gonna try it out...

[jmrico01]

No luck. Tried setting .android_api_level = 21 explicitly in my build.zig, and lot of other API level values. Also tried via CLI with -target aarch64-linux-android.X. Something maybe noteworthy is that in the LLVM IR (which you can helpfully inspect through Godbolt), in 0.14.1 I see the LLVM triple when using default linux and Android versions as
target triple = "aarch64-unknown-linux4.19.0-android14"
but in zig 0.11.0 it's
target triple = "aarch64-unknown-linux-android"
which omits both the linux kernel and Android versions.