x/mobile: gomobile bind: Crash on android: dlopen failed: TLS symbol "(null)" in dlopened x86_64/libgojni.so using IE access model

[marius-enlock]

Go version

go version go1.21.0 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/marius/.cache/go-build'
GOENV='/home/marius/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/marius/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/marius/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/marius/work/project/go.mod'
GOWORK='/home/marius/work/go.work'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build238633052=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I am creating a go library for Android that I use in React Native. The go library uses CGO (and has a as a dependency a static library that is built against android ndk llvm libc++ ), and also uses gRPC for communication with the server (However in the gRPC package I do not rely on TLS - I have a custom connection protocol, nor anywhere in the library I rely on TLS)

CC_FOR_TARGET=/home/marius/work/android-ndk-r26b/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/x86_64-linux-android34-clang gomobile bind -target android/amd64 -androidapi 34 -o redactedd.aar ./mobile/

And then I load the .aar archive into a react native project in the respective android project, and created the library data bindings in Kotlin. The project compiles.

What did you see happen?

My Android Kotlin binding that loads the library compiles and I can build it from react native, but when loading it into the Android emulator (Pixel 3a API 34 x86_64) I get the following Crash:

--------- beginning of crash
08-28 18:23:13.711 11376 11425 E AndroidRuntime: FATAL EXCEPTION: create_react_context
08-28 18:23:13.711 11376 11425 E AndroidRuntime: Process: com.redactedmobile, PID: 11376
08-28 18:23:13.711 11376 11425 E AndroidRuntime: java.lang.UnsatisfiedLinkError: dlopen failed: TLS symbol "(null)" in dlopened "/data/app/~~DwBiQaHdfiu5-dwMAj1Ulw==/com.redactedmobile-6XrWB7ekbVezJa8jWGp8QQ==/base.apk!/lib/x86_64/libgojni.so" referenced from "/data/app/~~DwBiQaHdfiu5-dwMAj1Ulw==/com.redactedmobile-6XrWB7ekbVezJa8jWGp8QQ==/base.apk!/lib/x86_64/libgojni.so" using IE access model
08-28 18:23:13.711 11376 11425 E AndroidRuntime: 	at java.lang.Runtime.loadLibrary0(Runtime.java:1082)
08-28 18:23:13.711 11376 11425 E AndroidRuntime: 	at java.lang.Runtime.loadLibrary0(Runtime.java:1003)
08-28 18:23:13.711 11376 11425 E AndroidRuntime: 	at java.lang.System.loadLibrary(System.java:1661)
08-28 18:23:13.711 11376 11425 E AndroidRuntime: 	at go.Seq.<clinit>(Seq.java:37)
08-28 18:23:13.711 11376 11425 E AndroidRuntime: 	at mobile.Mobile.<clinit>(Mobile.java:12)
08-28 18:23:13.711 11376 11425 E AndroidRuntime: 	at mobile.API.<clinit>(API.java:11)
08-28 18:23:13.711 11376 11425 E AndroidRuntime: 	at com.readactedmobile.Service.<init>(Service.kt:38)
08-28 18:23:13.711 11376 11425 E AndroidRuntime: 	at com.readactedmobile.ServicePackage.createNativeModules(Service.kt:27)
08-28 18:23:13.711 11376 11425 E AndroidRuntime: 	at com.facebook.react.ReactPackageHelper.getNativeModuleIterator(ReactPackageHelper.java:35)
08-28 18:23:13.711 11376 11425 E AndroidRuntime: 	at com.facebook.react.NativeModuleRegistryBuilder.processPackage(NativeModuleRegistryBuilder.java:40)
08-28 18:23:13.711 11376 11425 E AndroidRuntime: 	at com.facebook.react.ReactInstanceManager.processPackage(ReactInstanceManager.java:1510)
08-28 18:23:13.711 11376 11425 E AndroidRuntime: 	at com.facebook.react.ReactInstanceManager.processPackages(ReactInstanceManager.java:1481)
08-28 18:23:13.711 11376 11425 E AndroidRuntime: 	at com.facebook.react.ReactInstanceManager.createReactContext(ReactInstanceManager.java:1392)
08-28 18:23:13.711 11376 11425 E AndroidRuntime: 	at com.facebook.react.ReactInstanceManager.lambda$runCreateReactContextOnNewThread$2(ReactInstanceManager.java:1161)
08-28 18:23:13.711 11376 11425 E AndroidRuntime: 	at com.facebook.react.ReactInstanceManager.$r8$lambda$PrBhihCbbAFk4ZReAALGanVLCyc(Unknown Source:0)
08-28 18:23:13.711 11376 11425 E AndroidRuntime: 	at com.facebook.react.ReactInstanceManager$$ExternalSyntheticLambda1.run(D8$$SyntheticClass:0)
08-28 18:23:13.711 11376 11425 E AndroidRuntime: 	at java.lang.Thread.run(Thread.java:1012)
08-28 18:23:14.543   418   418 E BpTransactionCompletedListener: Failed to transact (-32)
08-28 18:23:15.144   585  1658 E TaskPersister: File error accessing recents directory (directory doesn't exist?).

(I think a relevant document is this one: https://android.googlesource.com/platform/bionic/+/HEAD/docs/elf-tls.md)

What did you expect to see?

I was expecting for the react native program to not crash.

[marius-enlock]

I have also tried with go version 1.23.0, the exact same problem.

[prattmic]

cc @hyangah

[gabyhelp]

Related Issues and Documentation

• x/mobile: build doesn't include libc++_shared.so to produced .apk and app immediately crashes upon start #44897
• x/mobile/bind: Unable to import bind package into java file #18960 (closed)
• x/mobile: bind with go1.22.5,1.22.6 leads to: fatal error: runtime: stack split at bad time #68760
• cmd/cgo: building fully statically linked executable for Android fails when cgo is enabled #59942
• x/mobile: Java to Go bindings crashes with java.lang.UnsatisfiedLinkError #12951 (closed)
• gomobile always get『# runtime/cgo ld: error: duplicate symbol: x_cgo_inittls』 #43399 (closed)
• x/mobile: proguard strips generated proxy classes from AAR #30251 (closed)
• x/mobile: gomobile bind triggers a CC compilation with the wrong compiler #49502 (closed)
• x/mobile: Crash gomobile-app with gomobile bind #25293 (closed)
• x/mobile: ndk-build finds wrong library path for dlopen #17807

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[marius-enlock]

I have went through all the above links and they do not solve this problem.
The NDK build tools and the CC must be the above ones due to React Native version 0.75.2.

The exact same crash error messages occurs in this Issue: mjansson/rpmalloc#332
For my project I think it comes from a different symbol, not the TLS_MODEL

I have checked the symbols from the libgojni.so with nm -j and the following ones mention TLS

crypto/tls.aesgcmCiphers
crypto/tls.CipherSuites
crypto/tls.(*ConnectionState).ExportKeyingMaterial
crypto/tls._CurveID_index_0
crypto/tls.(*CurveID).String
crypto/tls.CurveID.String
crypto/tls.disabledCipherSuites
crypto/tls.globalCertCache
crypto/tls.hasAESGCMHardwareSupport
crypto/tls.hasGCMAsmAMD64
crypto/tls.hasGCMAsmARM64
crypto/tls.hasGCMAsmS390X
crypto/tls.init
crypto/tls..inittask
crypto/tls.InsecureCipherSuites
crypto/tls..stmp_1
crypto/tls..stmp_2
crypto/tls..stmp_3
crypto/tls..stmp_4
crypto/tls.supportedOnlyTLS12
crypto/tls.supportedOnlyTLS13
crypto/tls.supportedUpToTLS12
crypto/tls.tdesCiphers
__emutls_unregister_key
go:itab.*google.golang.org/grpc/credentials.TLSChannelzSecurityValue,google.golang.org/grpc/credentials.ChannelzSecurityValue
go:itab.google.golang.org/grpc/credentials.TLSInfo,google.golang.org/grpc/credentials.AuthInfo
google.golang.org/grpc/credentials.(*TLSInfo).AuthType
google.golang.org/grpc/credentials.TLSInfo.AuthType
google.golang.org/grpc/credentials.(*TLSInfo).GetCommonAuthInfo
google.golang.org/grpc/credentials.TLSInfo.GetCommonAuthInfo
google.golang.org/grpc/credentials.(*TLSInfo).GetSecurityValue
google.golang.org/grpc/credentials.TLSInfo.GetSecurityValue
inittls
__libc_tls_generation_copy
runtime.settls.abi0
runtime.tls_g
__set_tls
__tls_get_addr
x_cgo_inittls
_Z17__init_static_tlsPv
_Z18__free_dynamic_tlsP10bionic_tcb
_Z22__free_temp_bionic_tlsP10bionic_tls
_Z22__init_bionic_tls_ptrsP10bionic_tcbP10bionic_tls
_Z24__bionic_get_tls_segmentPK10elf64_phdrmyP10TlsSegment
_Z26__allocate_temp_bionic_tlsv
_Z28__bionic_check_tls_alignmentPm
_Z28__set_stack_and_tls_vma_nameb
_ZL22tls_get_addr_slow_pathPK8TlsIndex
_ZN15StaticTlsLayout13finish_layoutEv
_ZN15StaticTlsLayout18reserve_bionic_tlsEv
_ZN15StaticTlsLayout27reserve_exe_segment_and_tcbEPK10TlsSegmentPKc
_ZN15StaticTlsLayout28round_up_with_overflow_checkEmm
_ZN15StaticTlsLayout7reserveEmm
_ZNK15StaticTlsLayout21offset_thread_pointerEv
_ZZL17layout_static_tlsR19KernelArgumentBlockE3mod

I suspect (without proof or good reasoning) it might be from __tls_get_addr() (I might have read a mention in an issue which I can't find),
or crypto/tls.init (I might have read a mention in an issue of an init function which I can't find),
or x_cgo_inittls (based on one of the above metioned links)

In the C library I link with CGO, I do not have any mentions of TLS symbols, nor dependencies on it.

I use this imports that might produce some of the above symbols in my go project:

"crypto/aes"
"crypto/cipher"
"crypto/rand"	

[marius-enlock]

I have checked the crypto library, and the grpc library, and it doesn't seem to use CGO to load any TLS library, I think the issue is not from there.

[marius-enlock]

ldd libgojni.so

	linux-vdso.so.1 (0x00007ffd5cd7a000)
	liblog.so => not found
	libandroid.so => not found
	libc++_shared.so => not found

I checked liblog and libandroid with readelf -a and there are no TLS symbols,
I suspect the culprit is libc++_shared.so as there are the TLS symbols
The exact library is this one:

android-ndk-r26b/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/x86_64-linux-android/libc++_shared.so

Where android-ndk-r26b is extracted from the linux archive from here: https://github.com/android/ndk/releases/tag/r26b

I think the issue might originate from here:
android/ndk#1679: Clang will now automatically enable ELF TLS for
minSdkVersion 29 or higher.

[marius-enlock]

In the file using CGO, I have tried passing the following flags:

-Wl,-plugin-opt=-emulated-tls=0

And in my gomobile bind command this flags:

 CGO_CFLAGS="-fno-emulated-tls" 

And it doesn't solve the issue, it is the same crash and error.

[xaionaro]

I don't know if this is of any help, but I've also stumbled upon basically the exact same error message (but as user of fyne), and this is the difference I see between a working APK and non-working APK:

Working APK

xaionaro@void:/tmp/2/lib/arm64-v8a$ readelf -d libstreampanel.so

Dynamic section at offset 0x3716758 contains 31 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libandroid.so]
 0x0000000000000001 (NEEDED)             Shared library: [liblog.so]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so]
 0x0000000000000001 (NEEDED)             Shared library: [libEGL.so]
 0x0000000000000001 (NEEDED)             Shared library: [libGLESv2.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x000000000000001e (FLAGS)              SYMBOLIC BIND_NOW
 0x000000006ffffffb (FLAGS_1)            Flags: NOW NODELETE
 0x0000000000000007 (RELA)               0xf448
 0x0000000000000008 (RELASZ)             5716704 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffff9 (RELACOUNT)          238195
 0x0000000000000017 (JMPREL)             0x582f28
 0x0000000000000002 (PLTRELSZ)           3960 (bytes)
 0x0000000000000003 (PLTGOT)             0x3718f50
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000006 (SYMTAB)             0x370
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000005 (STRTAB)             0xa36c
 0x000000000000000a (STRSZ)              20695 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x6b00
 0x0000000000000004 (HASH)               0x83a4
 0x0000000000000019 (INIT_ARRAY)         0x3718750
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x3718740
 0x000000000000001c (FINI_ARRAYSZ)       16 (bytes)
 0x000000006ffffff0 (VERSYM)             0x62b0
 0x000000006ffffffe (VERNEED)            0x6aa0
 0x000000006fffffff (VERNEEDNUM)         3
 0x0000000000000000 (NULL)               0x0

Non-working APK

root@void:/tmp# readelf -d libstreampanel.so

Dynamic section at offset 0x64f2538 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libandroid.so]
 0x0000000000000001 (NEEDED)             Shared library: [liblog.so]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so]
 0x0000000000000001 (NEEDED)             Shared library: [libEGL.so]
 0x0000000000000001 (NEEDED)             Shared library: [libGLESv2.so]
 0x000000000000001e (FLAGS)              SYMBOLIC BIND_NOW STATIC_TLS
 0x000000006ffffffb (FLAGS_1)            Flags: NOW NODELETE
 0x0000000000000007 (RELA)               0x1dfe68
 0x0000000000000008 (RELASZ)             7756656 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffff9 (RELACOUNT)          322942
 0x0000000000000017 (JMPREL)             0x9459d8
 0x0000000000000002 (PLTRELSZ)           12648 (bytes)
 0x0000000000000003 (PLTGOT)             0x6501f30
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000006 (SYMTAB)             0x3a8
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000005 (STRTAB)             0x11db08
 0x000000000000000a (STRSZ)              795485 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0xb46a8
 0x0000000000000004 (HASH)               0xe63f8
 0x0000000000000019 (INIT_ARRAY)         0x64f44a0
 0x000000000000001b (INIT_ARRAYSZ)       152 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x64f4480
 0x000000000000001c (FINI_ARRAYSZ)       32 (bytes)
 0x000000006ffffff0 (VERSYM)             0xa68c0
 0x000000006ffffffe (VERNEED)            0xb4684
 0x000000006fffffff (VERNEEDNUM)         1
 0x0000000000000000 (NULL)               0x0

Hypothesis

It seems like STATIC_TLS breaks the thing. I also do not have libdl.so and libc.so in the broken APK, but it feels like a separate problem.

---

UPDATE

Yes. I've managed to get rid of all those static library dependencies which were introducing the STATIC_TLS (which were only 2 of dozens static libs) and manually added -ldl -lc and now it runs in my case.
Though, this is obviously not a solution to the problem; just may be somebody will find this as an useful hint.