Skip to content

Commit 673476d

Browse files
bd1976brisbd1976bris
authored
[DTLTO][LLD][ELF] Support bitcode members of thin archives (#149425)
This patch adds support for bitcode members of thin archives to DTLTO (https://llvm.org/docs/DTLTO.html) in ELF LLD. For DTLTO, bitcode identifiers must be valid paths to bitcode files on disk. Clang does not support archive inputs for ThinLTO backend compilations. This patch adjusts the identifier for bitcode members of thin archives in DTLTO links so that it is the path to the member file on disk, allowing such members to be supported in DTLTO. This patch is sufficient to allow for self-hosting an LLVM build with DTLTO when thin archives are used. Note: Bitcode members of non-thin archives remain unsupported. This will be addressed in a future change. Testing: - LLD lit test coverage has been added to check that the identifier is adjusted appropriately. - A cross-project lit test has been added to show that a DTLTO link can succeed when linking bitcode members of thin archives. For the design discussion of the DTLTO feature, see: #126654.
1 parent 965231c commit 673476d

File tree

5 files changed

+216
-12
lines changed

5 files changed

+216
-12
lines changed

‎cross-project-tests/CMakeLists.txt‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,12 @@ set(CROSS_PROJECT_TEST_DEPS
1919
FileCheck
2020
check-gdb-llvm-support
2121
count
22-
llvm-dwarfdump
22+
llvm-ar
2323
llvm-config
24+
llvm-dwarfdump
2425
llvm-objdump
25-
split-file
2626
not
27+
split-file
2728
)
2829

2930
if ("clang" IN_LIST LLVM_ENABLE_PROJECTS)

‎cross-project-tests/dtlto/ld-archive-thin.test‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
REQUIRES: ld.lld,llvm-ar
2+
3+
## Test that a DTLTO link succeeds and outputs the expected set of files
4+
## correctly when thin archives are present.
5+
6+
RUN: rm -rf %t && split-file %s %t && cd %t
7+
8+
## Compile bitcode. -O2 is required for cross-module importing.
9+
RUN: %clang -O2 --target=x86_64-linux-gnu -flto=thin -c \
10+
RUN: foo.c bar.c dog.c cat.c start.c
11+
12+
## Generate thin archives.
13+
RUN: llvm-ar rcs foo.a foo.o --thin
14+
## Create this bitcode thin archive in a subdirectory to test the expansion of
15+
## the path to a bitcode file that is referenced using "..", e.g., in this case
16+
## "../bar.o".
17+
RUN: mkdir lib
18+
RUN: llvm-ar rcs lib/bar.a bar.o --thin
19+
## Create this bitcode thin archive with an absolute path entry containing "..".
20+
RUN: llvm-ar rcs dog.a %t/lib/../dog.o --thin
21+
## The bitcode member of cat.a will not be used in the link.
22+
RUN: llvm-ar rcs cat.a cat.o --thin
23+
RUN: llvm-ar rcs start.a start.o --thin
24+
25+
## Link from a different directory to ensure that thin archive member paths are
26+
## resolved correctly relative to the archive locations.
27+
RUN: mkdir %t/out && cd %t/out
28+
29+
RUN: %clang --target=x86_64-linux-gnu -flto=thin -fuse-ld=lld %t/foo.a %t/lib/bar.a ../start.a %t/cat.a \
30+
RUN: -Wl,--whole-archive ../dog.a \
31+
RUN: -fthinlto-distributor=%python \
32+
RUN: -Xthinlto-distributor=%llvm_src_root/utils/dtlto/local.py \
33+
RUN: -Wl,--save-temps -nostdlib -Werror
34+
35+
## Check that the required output files have been created.
36+
RUN: ls | sort | FileCheck %s
37+
38+
## No files are expected before.
39+
CHECK-NOT: {{.}}
40+
41+
## JSON jobs description.
42+
CHECK: {{^}}a.[[PID:[a-zA-Z0-9_]+]].dist-file.json{{$}}
43+
44+
## Native output object files and individual summary index files.
45+
CHECK: {{^}}bar.3.[[PID]].native.o{{$}}
46+
CHECK: {{^}}bar.3.[[PID]].native.o.thinlto.bc{{$}}
47+
CHECK: {{^}}dog.1.[[PID]].native.o{{$}}
48+
CHECK: {{^}}dog.1.[[PID]].native.o.thinlto.bc{{$}}
49+
CHECK: {{^}}foo.2.[[PID]].native.o{{$}}
50+
CHECK: {{^}}foo.2.[[PID]].native.o.thinlto.bc{{$}}
51+
CHECK: {{^}}start.4.[[PID]].native.o{{$}}
52+
CHECK: {{^}}start.4.[[PID]].native.o.thinlto.bc{{$}}
53+
54+
## No files are expected after.
55+
CHECK-NOT: {{.}}
56+
57+
58+
## It is important that cross-module inlining occurs for this test to show that Clang can
59+
## successfully load the bitcode file dependencies recorded in the summary indices.
60+
## Explicitly check that the expected importing has occurred.
61+
62+
RUN: llvm-dis start.4.*.native.o.thinlto.bc -o - | \
63+
RUN: FileCheck %s --check-prefixes=FOO,BAR,START
64+
65+
RUN: llvm-dis dog.1.*.native.o.thinlto.bc -o - | \
66+
RUN: FileCheck %s --check-prefixes=FOO,BAR,DOG,START
67+
68+
RUN: llvm-dis foo.2.*.native.o.thinlto.bc -o - | \
69+
RUN: FileCheck %s --check-prefixes=FOO,BAR,START
70+
71+
RUN: llvm-dis bar.3.*.native.o.thinlto.bc -o - | \
72+
RUN: FileCheck %s --check-prefixes=FOO,BAR,START
73+
74+
FOO-DAG: foo.o
75+
BAR-DAG: bar.o
76+
DOG-DAG: dog.o
77+
START-DAG: start.o
78+
79+
80+
#--- foo.c
81+
extern int bar(int), _start(int);
82+
__attribute__((retain)) int foo(int x) { return x + bar(x) + _start(x); }
83+
84+
#--- bar.c
85+
extern int foo(int), _start(int);
86+
__attribute__((retain)) int bar(int x) { return x + foo(x) + _start(x); }
87+
88+
#--- dog.c
89+
extern int foo(int), bar(int), _start(int);
90+
__attribute__((retain)) int dog(int x) { return x + foo(x) + bar(x) + _start(x); }
91+
92+
#--- cat.c
93+
__attribute__((retain)) void cat(int x) {}
94+
95+
#--- start.c
96+
extern int foo(int), bar(int);
97+
__attribute__((retain)) int _start(int x) { return x + foo(x) + bar(x); }

‎cross-project-tests/lit.cfg.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,8 @@ def get_required_attr(config, attr_name):
107107
if lldb_path is not None:
108108
config.available_features.add("lldb")
109109

110+
if llvm_config.use_llvm_tool("llvm-ar"):
111+
config.available_features.add("llvm-ar")
110112

111113
def configure_dexter_substitutions():
112114
"""Configure substitutions for host platform and return list of dependencies"""

‎lld/ELF/InputFiles.cpp‎

Lines changed: 49 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
#include "llvm/ADT/CachedHashString.h"
2121
#include "llvm/ADT/STLExtras.h"
2222
#include "llvm/LTO/LTO.h"
23+
#include "llvm/Object/Archive.h"
2324
#include "llvm/Object/IRObjectFile.h"
2425
#include "llvm/Support/AArch64AttributeParser.h"
2526
#include "llvm/Support/ARMAttributeParser.h"
@@ -1811,6 +1812,39 @@ static uint8_t getOsAbi(const Triple &t) {
18111812
}
18121813
}
18131814

1815+
// For DTLTO, bitcode member names must be valid paths to files on disk.
1816+
// For thin archives, resolve `memberPath` relative to the archive's location.
1817+
// Returns true if adjusted; false otherwise. Non-thin archives are unsupported.
1818+
static bool dtltoAdjustMemberPathIfThinArchive(Ctx &ctx, StringRef archivePath,
1819+
std::string &memberPath) {
1820+
assert(!archivePath.empty());
1821+
1822+
if (ctx.arg.dtltoDistributor.empty())
1823+
return false;
1824+
1825+
// Read the archive header to determine if it's a thin archive.
1826+
auto bufferOrErr =
1827+
MemoryBuffer::getFileSlice(archivePath, sizeof(ThinArchiveMagic) - 1, 0);
1828+
if (std::error_code ec = bufferOrErr.getError()) {
1829+
ErrAlways(ctx) << "cannot open " << archivePath << ": " << ec.message();
1830+
return false;
1831+
}
1832+
1833+
if (!bufferOrErr->get()->getBuffer().starts_with(ThinArchiveMagic))
1834+
return false;
1835+
1836+
SmallString<128> resolvedPath;
1837+
if (path::is_relative(memberPath)) {
1838+
resolvedPath = path::parent_path(archivePath);
1839+
path::append(resolvedPath, memberPath);
1840+
} else
1841+
resolvedPath = memberPath;
1842+
1843+
path::remove_dots(resolvedPath, /*remove_dot_dot=*/true);
1844+
memberPath = resolvedPath.str();
1845+
return true;
1846+
}
1847+
18141848
BitcodeFile::BitcodeFile(Ctx &ctx, MemoryBufferRef mb, StringRef archiveName,
18151849
uint64_t offsetInArchive, bool lazy)
18161850
: InputFile(ctx, BitcodeKind, mb) {
@@ -1821,17 +1855,22 @@ BitcodeFile::BitcodeFile(Ctx &ctx, MemoryBufferRef mb, StringRef archiveName,
18211855
if (ctx.arg.thinLTOIndexOnly)
18221856
path = replaceThinLTOSuffix(ctx, mb.getBufferIdentifier());
18231857

1824-
// ThinLTO assumes that all MemoryBufferRefs given to it have a unique
1825-
// name. If two archives define two members with the same name, this
1826-
// causes a collision which result in only one of the objects being taken
1827-
// into consideration at LTO time (which very likely causes undefined
1828-
// symbols later in the link stage). So we append file offset to make
1829-
// filename unique.
18301858
StringSaver &ss = ctx.saver;
1831-
StringRef name = archiveName.empty()
1832-
? ss.save(path)
1833-
: ss.save(archiveName + "(" + path::filename(path) +
1834-
" at " + utostr(offsetInArchive) + ")");
1859+
StringRef name;
1860+
if (archiveName.empty() ||
1861+
dtltoAdjustMemberPathIfThinArchive(ctx, archiveName, path)) {
1862+
name = ss.save(path);
1863+
} else {
1864+
// ThinLTO assumes that all MemoryBufferRefs given to it have a unique
1865+
// name. If two archives define two members with the same name, this
1866+
// causes a collision which result in only one of the objects being taken
1867+
// into consideration at LTO time (which very likely causes undefined
1868+
// symbols later in the link stage). So we append file offset to make
1869+
// filename unique.
1870+
name = ss.save(archiveName + "(" + path::filename(path) + " at " +
1871+
utostr(offsetInArchive) + ")");
1872+
}
1873+
18351874
MemoryBufferRef mbref(mb.getBuffer(), name);
18361875

18371876
obj = CHECK2(lto::InputFile::create(mbref), this);

‎lld/test/ELF/dtlto/archive-thin.test‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
REQUIRES: x86
2+
3+
## Test that a DTLTO link assigns Module IDs to thin archive members as expected.
4+
5+
RUN: rm -rf %t && split-file %s %t && cd %t
6+
7+
RUN: sed 's/@t1/@t2/g' t1.ll > t2.ll
8+
RUN: sed 's/@t1/@t3/g' t1.ll > t3.ll
9+
10+
RUN: opt -thinlto-bc t1.ll -o t1.bc
11+
RUN: opt -thinlto-bc t2.ll -o t2.bc
12+
RUN: opt -thinlto-bc t3.ll -o t3.bc
13+
14+
RUN: llvm-ar rcs t1.a t1.bc --thin
15+
## Create this bitcode thin archive in a subdirectory to test the expansion of
16+
## the path to a bitcode file that is referenced using "..", e.g., in this case
17+
## "../t2.bc".
18+
RUN: mkdir lib
19+
RUN: llvm-ar rcs lib/t2.a t2.bc --thin
20+
## Create this bitcode thin archive with an absolute path entry containing "..".
21+
RUN: llvm-ar rcs t3.a %t/lib/../t3.bc --thin
22+
23+
## Link from a different directory to ensure that thin archive member paths are
24+
## resolved correctly relative to the archive locations.
25+
RUN: mkdir %t/out && cd %t/out
26+
27+
## Build a response file to share common linking arguments.
28+
## Note: validate.py does not perform any compilation. Instead, it validates the
29+
## received JSON, pretty-prints the JSON and the supplied arguments, and then
30+
## exits with an error. This allows FileCheck directives to verify the
31+
## distributor inputs.
32+
RUN: echo '%t/t1.a %t/lib/t2.a ../t3.a \
33+
RUN: --thinlto-distributor="%python" \
34+
RUN: --thinlto-distributor-arg="%llvm_src_root/utils/dtlto/validate.py"' > rsp
35+
36+
## Link thin archives using -u/--undefined.
37+
RUN: not ld.lld @rsp -u t1 -u t2 -u t3 2>&1 | FileCheck %s
38+
39+
## Link thin archives using --whole-archive.
40+
RUN: not ld.lld --whole-archive @rsp 2>&1 | FileCheck %s
41+
42+
## Check the module IDs in the JSON jobs description.
43+
CHECK: "jobs": [
44+
CHECK: "inputs": [
45+
CHECK-NEXT: "{{([a-zA-Z]:)|/}}
46+
CHECK-SAME: {{/|\\\\}}archive-thin.test.tmp{{/|\\\\}}t1.bc"
47+
48+
CHECK: "inputs": [
49+
CHECK-NEXT: "{{([a-zA-Z]\:)|/}}
50+
CHECK-SAME: {{/|\\\\}}archive-thin.test.tmp{{/|\\\\}}t2.bc"
51+
52+
CHECK: "inputs": [
53+
CHECK-NEXT: "{{([a-zA-Z]:)|/}}
54+
CHECK-SAME: {{/|\\\\}}archive-thin.test.tmp{{/|\\\\}}t3.bc"
55+
56+
## Ensure backend compilation fails as expected (due to validate.py dummy behavior).
57+
CHECK: error: DTLTO backend compilation: cannot open native object file:
58+
59+
#--- t1.ll
60+
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
61+
target triple = "x86_64-unknown-linux-gnu"
62+
63+
define void @t1() {
64+
ret void
65+
}

0 commit comments

Comments
 (0)