Skip to content

Commit 3a43f71

Browse files
jbachorikjbachorik
committed
Add arch-doc for CallTraceStorage
1 parent 7e18119 commit 3a43f71

File tree

1 file changed

+348
-0
lines changed

1 file changed

+348
-0
lines changed

‎docs/architecture/CallTraceStorage.md‎

Lines changed: 348 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,348 @@
1+
# CallTraceStorage Triple-Buffer Architecture
2+
3+
## Overview
4+
5+
The CallTraceStorage system implements a sophisticated triple-buffered architecture designed for lock-free, signal-handler-safe profiling data collection. This design enables concurrent trace collection from signal handlers while allowing safe background processing for JFR (Java Flight Recorder) serialization.
6+
7+
Each collected call trace receives a globally unique 64-bit identifier composed of a 32-bit instance epoch ID and a 32-bit slot index. This dual-component design ensures collision-free trace identification across buffer rotations and supports stable JFR constant pool references.
8+
9+
## Core Design Principles
10+
11+
1. **Signal Handler Safety**: All operations in signal handlers use lock-free atomic operations
12+
2. **Globally Unique Trace IDs**: 64-bit identifiers (instance epoch + slot index) prevent collisions across buffer rotations
13+
3. **Memory Continuity**: Traces can be preserved across collection cycles for liveness tracking
14+
4. **Zero-Copy Collection**: Uses atomic pointer swapping instead of data copying
15+
5. **ABA Protection**: Generation counters and hazard pointers prevent use-after-free
16+
6. **Lock-Free Concurrency**: Multiple threads can collect traces without blocking each other
17+
18+
## Triple-Buffer States
19+
20+
The system maintains three `CallTraceHashTable` instances with distinct roles:
21+
22+
```
23+
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
24+
│ ACTIVE │ │ STANDBY │ │ SCRATCH │
25+
│ │ │ │ │ │
26+
│ New traces │ │ Preserved │ │ Processing │
27+
│ from signal │ │ traces from │ │ old traces │
28+
│ handlers │ │ prev cycle │ │ before clear│
29+
└─────────────┘ └─────────────┘ └─────────────┘
30+
```
31+
32+
### Buffer Roles
33+
34+
- **ACTIVE**: Receives new traces from signal handlers (lock-free puts)
35+
- **STANDBY**: Contains preserved traces from the previous collection cycle
36+
- **SCRATCH**: Temporary storage during rotation, gets cleared after processing
37+
38+
## Triple-Buffer Rotation Algorithm
39+
40+
The rotation follows a carefully orchestrated 6-step sequence:
41+
42+
### Phase Diagram
43+
44+
```
45+
BEFORE ROTATION:
46+
┌─────────────────────────────────────────────────────────────┐
47+
│ Thread A (Signal Handler) │ Thread B (JFR Processing) │
48+
├─────────────────────────────────────────────────────────────┤
49+
│ │ │
50+
│ put() → ACTIVE │ processTraces() │
51+
│ ↓ │ ↓ │
52+
│ [New Traces] │ Step 1: Collect STANDBY │
53+
│ │ Step 2: Clear STANDBY │
54+
│ │ Step 3: ATOMIC SWAP │
55+
└─────────────────────────────────────────────────────────────┘
56+
57+
DURING ROTATION (Atomic Swap):
58+
┌─────────────────────────────────────────────────────────────┐
59+
│ OLD STATE │ ATOMIC SWAP │ NEW STATE │
60+
├─────────────────────────────────────────────────────────────┤
61+
│ ACTIVE = A │ │ ACTIVE = B │
62+
│ STANDBY = B │ ──── SWAP ────→ │ STANDBY = C │
63+
│ SCRATCH = C │ │ SCRATCH = A │
64+
└─────────────────────────────────────────────────────────────┘
65+
66+
AFTER ROTATION:
67+
┌────────────────────────────────────────────────────────────┐
68+
│ put() → NEW ACTIVE (B) │ Step 4: Collect SCRATCH │
69+
│ │ Step 5: Process All │
70+
│ [Safe to continue] │ Step 6: Preserve & Clear │
71+
└────────────────────────────────────────────────────────────┘
72+
```
73+
74+
### Detailed Steps
75+
76+
```cpp
77+
void processTraces() {
78+
// PHASE 1: Liveness Analysis
79+
// Determine which traces need preservation
80+
81+
// PHASE 2: Collection Sequence
82+
83+
// Step 1: Collect from STANDBY (preserved traces)
84+
current_standby->collect(standby_traces);
85+
86+
// Step 2: Clear STANDBY, prepare for new role as ACTIVE
87+
current_standby->clear();
88+
current_standby->setInstanceId(new_instance_id);
89+
90+
// Step 3: ATOMIC ROTATION
91+
// STANDBY (empty) → ACTIVE (receives new traces)
92+
old_active = _active_storage.exchange(current_standby);
93+
94+
// ACTIVE (full) → SCRATCH (for processing)
95+
old_scratch = _scratch_storage.exchange(old_active);
96+
97+
// SCRATCH (processed) → STANDBY (for next cycle)
98+
_standby_storage.store(old_scratch);
99+
100+
// Step 4: Collect from SCRATCH (old active, now read-only)
101+
old_active->collect(active_traces);
102+
103+
// Step 5: Process combined traces
104+
all_traces = standby_traces ∪ active_traces;
105+
processor(all_traces);
106+
107+
// Step 6: Preserve traces for next cycle
108+
old_scratch->clear();
109+
for (trace : preserved_traces) {
110+
old_scratch->putWithExistingIdLockFree(trace);
111+
}
112+
}
113+
```
114+
115+
## Memory Safety Mechanisms
116+
117+
### Hazard Pointers
118+
119+
Signal handlers use hazard pointers to prevent tables from being deleted during access:
120+
121+
```
122+
Signal Handler Thread JFR Processing Thread
123+
───────────────────── ──────────────────────
124+
1. Load active table
125+
2. Register hazard pointer ──→ 1. Check hazard pointers
126+
3. Verify table still active 2. Wait if hazards exist
127+
4. Use table safely 3. Safe to delete/clear
128+
5. Clear hazard pointer 4. Continue processing
129+
```
130+
131+
### ABA Protection
132+
133+
Generation counters prevent the ABA problem during concurrent access:
134+
135+
```cpp
136+
// Each storage operation includes generation check
137+
u64 generation = _generation_counter.load();
138+
CallTraceHashTable* table = _active_storage.load();
139+
140+
if (_generation_counter.load() != generation) {
141+
// Storage was rotated, retry or abort
142+
}
143+
```
144+
145+
## Thread-Local Collections
146+
147+
Each thread maintains pre-allocated collections to avoid malloc/free in hot paths:
148+
149+
```
150+
Thread A Thread B Thread N
151+
──────── ──────── ────────
152+
ThreadLocalCollections ThreadLocalCollections ThreadLocalCollections
153+
├─ traces_buffer ├─ traces_buffer ├─ traces_buffer
154+
├─ standby_traces ├─ standby_traces ├─ standby_traces
155+
├─ active_traces ├─ active_traces ├─ active_traces
156+
├─ preserve_set ├─ preserve_set ├─ preserve_set
157+
└─ traces_to_preserve └─ traces_to_preserve └─ traces_to_preserve
158+
```
159+
160+
## Liveness Preservation
161+
162+
The system supports pluggable liveness checkers to determine which traces to preserve:
163+
164+
```cpp
165+
// Liveness checker interface
166+
typedef std::function<void(std::unordered_set<u64>&)> LivenessChecker;
167+
168+
// Example: JFR constant pool preservation
169+
registerLivenessChecker([](std::unordered_set<u64>& preserve_set) {
170+
// Add trace IDs that appear in active JFR recordings
171+
preserve_set.insert(active_jfr_traces.begin(), active_jfr_traces.end());
172+
});
173+
```
174+
175+
## 64-Bit Trace ID Architecture
176+
177+
The system uses a sophisticated 64-bit trace ID scheme that combines collision avoidance with instance tracking to ensure globally unique, stable trace identifiers across buffer rotations.
178+
179+
### Trace ID Structure
180+
181+
```
182+
┌─────────────────────────────────────────────────────────────────────┐
183+
│ 64-bit Trace ID │
184+
├──────────────────────────────┬──────────────────────────────────────┤
185+
│ Upper 32 bits │ Lower 32 bits │
186+
│ Instance Epoch ID │ Hash Table Slot Index │
187+
│ │ │
188+
│ Unique per active rotation │ Position in hash table │
189+
│ Prevents collision across │ (0 to capacity-1) │
190+
│ buffer swaps │ │
191+
└──────────────────────────────┴──────────────────────────────────────┘
192+
```
193+
194+
### Instance Epoch ID Generation
195+
196+
Each time a `CallTraceHashTable` transitions from STANDBY to ACTIVE during buffer rotation, it receives a new instance epoch ID:
197+
198+
```cpp
199+
// During rotation - Step 2
200+
current_standby->clear();
201+
u64 new_instance_id = getNextInstanceId(); // Atomic increment
202+
current_standby->setInstanceId(new_instance_id);
203+
204+
// Later during trace creation
205+
u64 trace_id = (instance_id << 32) | slot_index;
206+
```
207+
208+
### Collision Prevention Across Rotations
209+
210+
The instance epoch prevents trace ID collisions when the same hash table slot is reused across different active periods:
211+
212+
```
213+
Timeline Example:
214+
─────────────────────────────────────────────────────────────────────
215+
216+
Rotation 1: Instance ID = 0x00000001
217+
┌─────────────────┐
218+
│ ACTIVE Table A │ Slot 100 → Trace ID: 0x0000000100000064
219+
│ Instance: 001 │ Slot 200 → Trace ID: 0x00000001000000C8
220+
└─────────────────┘
221+
222+
Rotation 2: Instance ID = 0x00000002
223+
┌─────────────────┐
224+
│ ACTIVE Table A │ Slot 100 → Trace ID: 0x0000000200000064
225+
│ Instance: 002 │ Slot 200 → Trace ID: 0x00000002000000C8
226+
│ (same table, │
227+
│ different ID) │
228+
└─────────────────┘
229+
```
230+
231+
### JFR Constant Pool Stability
232+
233+
The trace ID scheme provides crucial benefits for JFR serialization:
234+
235+
1. **Stable References**: Trace IDs remain consistent during the active period
236+
2. **Unique Across Cycles**: Even if the same slot is reused, the trace ID differs
237+
3. **Collision Avoidance**: 32-bit instance space prevents ID conflicts
238+
4. **Liveness Tracking**: Preserved traces maintain their original IDs
239+
240+
### Implementation Details
241+
242+
```cpp
243+
class CallTraceHashTable {
244+
std::atomic<u64> _instance_id; // Set when becoming active
245+
246+
u64 put(int num_frames, ASGCT_CallFrame* frames, bool truncated, u64 weight) {
247+
// ... hash table logic ...
248+
249+
// Generate unique trace ID
250+
u64 instance_id = _instance_id.load(std::memory_order_acquire);
251+
u64 trace_id = (instance_id << 32) | slot;
252+
253+
CallTrace* trace = storeCallTrace(num_frames, frames, truncated, trace_id);
254+
return trace->trace_id;
255+
}
256+
};
257+
```
258+
259+
### Instance ID Generation
260+
261+
```cpp
262+
class CallTraceStorage {
263+
static std::atomic<u64> _next_instance_id; // Global counter
264+
265+
static u64 getNextInstanceId() {
266+
return _next_instance_id.fetch_add(1, std::memory_order_relaxed);
267+
}
268+
269+
void processTraces() {
270+
// During rotation - assign new instance ID
271+
u64 new_instance_id = getNextInstanceId();
272+
current_standby->setInstanceId(new_instance_id);
273+
274+
// Atomic swap: standby becomes new active with fresh instance ID
275+
_active_storage.exchange(current_standby, std::memory_order_acq_rel);
276+
}
277+
};
278+
```
279+
280+
### Reserved ID Space
281+
282+
The system reserves trace IDs with upper 32 bits = 0 for special purposes:
283+
284+
```cpp
285+
// Reserved for dropped samples (contention/allocation failures)
286+
static const u64 DROPPED_TRACE_ID = 1ULL;
287+
288+
// Real trace IDs always have instance_id >= 1
289+
// Format: (instance_id << 32) | slot where instance_id starts from 1
290+
// This guarantees no collision with reserved IDs
291+
```
292+
293+
### Benefits of This Architecture
294+
295+
1. **Collision Immunity**: Same slot across rotations generates different trace IDs
296+
2. **JFR Compatibility**: 64-bit IDs work seamlessly with JFR constant pool indices
297+
3. **Liveness Support**: Preserved traces maintain stable IDs across collection cycles
298+
4. **Debug Capability**: Instance ID in trace ID aids in debugging buffer rotation issues
299+
5. **Scalability**: 32-bit instance space supports ~4 billion rotations before wraparound
300+
301+
This trace ID design ensures that each call trace has a globally unique, stable identifier that survives the complex buffer rotation lifecycle while providing essential metadata about its origin and timing.
302+
303+
## Performance Characteristics
304+
305+
### Lock-Free Operations
306+
- **put()**: O(1) average, lock-free with hazard pointer protection
307+
- **processTraces()**: Lock-free table swapping, O(n) collection where n = trace count
308+
309+
### Memory Efficiency
310+
- **Zero-Copy Rotation**: Only atomic pointer swaps, no data copying
311+
- **Pre-allocated Collections**: Thread-local collections prevent malloc/free cycles
312+
- **Trace Deduplication**: Hash tables prevent duplicate trace storage
313+
314+
### Concurrency Benefits
315+
- **Signal Handler Safe**: No blocking operations in signal context
316+
- **Multi-threaded Collection**: Multiple threads can process traces concurrently
317+
- **Contention-Free**: Atomic operations eliminate lock contention
318+
319+
## Usage Example
320+
321+
```cpp
322+
// Setup
323+
CallTraceStorage storage;
324+
storage.registerLivenessChecker([](auto& preserve_set) {
325+
// Add traces to preserve
326+
});
327+
328+
// Signal handler (lock-free)
329+
u64 trace_id = storage.put(num_frames, frames, truncated, weight);
330+
331+
// Background processing
332+
storage.processTraces([](const std::unordered_set<CallTrace*>& traces) {
333+
// Serialize to JFR format
334+
for (CallTrace* trace : traces) {
335+
writeToJFR(trace);
336+
}
337+
});
338+
```
339+
340+
## Key Architectural Benefits
341+
342+
1. **Scalability**: Lock-free design scales linearly with thread count
343+
2. **Reliability**: Hazard pointers prevent memory safety issues
344+
3. **Flexibility**: Pluggable liveness checkers support different use cases
345+
4. **Performance**: Zero-copy operations minimize overhead
346+
5. **Safety**: Signal-handler safe operations prevent deadlocks
347+
348+
This architecture enables high-performance, concurrent profiling data collection suitable for production environments with minimal impact on application performance.

0 commit comments

Comments
 (0)