llvm-symbolizer can fail due to asan-compiled dependency libraries in LD_LIBRARY_PATH; SIGPIPE exit results

[greghudson]

This issue only affects developers of libraries in the dependency chain of llvm-symbolizer, and it might be tough to fix.

I maintain MIT krb5. Our make check runs various binaries with LD_LIBRARY_PATH set to a subdirectory of the build directory containing the built versions of libgssapi_krb5/libkrb5/etc.. If we have built everything with -fsanitizer=address and asan wants to report a problem with a stack trace when one of the binaries exits, llvm-symbolizer gets run to convert addresses into symbol names. When llvm-symbolizer starts, the dynamic linker sees a bunch of asan-compiled krb5 libraries in LD_LIBRARY_PATH. Since llvm-symbolizer itself wasn't built with asan, one of these libraries will cause a dynamic linker error like:

/usr/bin/llvm-symbolizer-18: symbol lookup error: /home/runner/work/krb5/krb5/src/./lib/libkrb5support.so.0: undefined symbol: __asan_option_detect_stack_use_after_return

The program then attempts to write to the llvm-symbolizer pipe and gets a SIGPIPE because the subprocess has exited. If SIGPIPE isn't ignored, the process exits immediately without displaying the un-symbolized stack trace.

[llvmbot]

@llvm/issue-subscribers-tools-llvm-symbolizer

Author: Greg Hudson (greghudson)

This issue only affects developers of libraries in the dependency chain of llvm-symbolizer, and it might be tough to fix.

I maintain MIT krb5. Our make check runs various binaries with LD_LIBRARY_PATH set to a subdirectory of the build directory containing the built versions of libgssapi_krb5/libkrb5/etc.. If we have built everything with -fsanitizer=address and asan wants to report a problem with a stack trace when one of the binaries exits, llvm-symbolizer gets run to convert addresses into symbol names. When llvm-symbolizer starts, the dynamic linker sees a bunch of asan-compiled krb5 libraries in LD_LIBRARY_PATH. Since llvm-symbolizer itself wasn't built with asan, one of these libraries will cause a dynamic linker error like:

/usr/bin/llvm-symbolizer-18: symbol lookup error: /home/runner/work/krb5/krb5/src/./lib/libkrb5support.so.0: undefined symbol: __asan_option_detect_stack_use_after_return

The program then attempts to write to the llvm-symbolizer pipe and gets a SIGPIPE because the subprocess has exited. If SIGPIPE isn't ignored, the process exits immediately without displaying the un-symbolized stack trace.

[jh7370]

@greghudson, do you have any specific suggestions as to what you'd like see fixed? It sounds to me like the problem is with the libraries the dynamic loader is trying to load for llvm-symbolizer, namely that it should try to load the non-ASAN versions. This would presumably require LD_LIBRARY_PATH to be different for the llvm-symbolizer call to that of the host process.

[greghudson]

Mitigating the SIGPIPE consequence would help. As the process is about to exit, I think there would be no harm in ignoring SIGPIPE after invoking the symbolizer and before writing to it. If this were fixed, I'd see un-symbolized stack traces, but I wouldn't see unexpected process exit statuses.

For the fundamental problem, an asan option to clear LD_LIBRARY_PATH before running the symbolizer would provide a workaround. Right now the only workaround is to avoid using LD_LIBRARY_PATH for make check, which requires a significant change to the build system design. [ETA: I guess I do have another workaround currently: I can create an llvm-symbolizer wrapper that unsets LD_LIBRARY_PATH, and point asan at that via the path or ASAN_SYMBOLIZER_PATH.]

Changing the asan design not to require a sub-process invocation to symbolize stack traces would of course be ideal from my point of view, but that sounds extremely high-effort.

[jh7370]

Thanks @greghudson, I've added the compiler-rt:asan label to this ticket, as I think it fits better under that category, by the sounds of it. I'm not sure there's much the llvm-symbolizer itself can do, since the problem occurs before the tool itself gets up and running.

[bernd-edlinger]

FWIW exactly the same happens with openssl-3.x: when an memory sanitizer error is to be reported this happens:
/usr/bin/llvm-symbolizer-18: symbol lookup error: ../../util/../util/../libcrypto.so.3: undefined symbol: __msan_retval_tls
The only workaround I have found is building the openssl statically with memory sanitizer enabled.
However also other environment variables are able to influence the openssl library after it is loaded,
e.g. OPENSSL_CONF which is used in the openssl test suite to override the system openssl.cnf file
or OPENSSL_ENGINES which overrides the system engines directory
or OPENSSL_MODULES which overrides the system providers directory to name just a few.
So in case the symbolizer needs to use the loaded openssl those can still cause trouble.

[greghudson]

It might help to have a facility to print the symbolizer pathname asan will select, making it easier to create wrapper scripts that unset variables. Right now I'm having to replicate the logic in ChooseExternalSymbolizer(), plus having to look for the versioned llvm-symbolizer installed/chosen on Debian-derived systems.