Fix benchmark flakiness #7790

NachoEchevarria · 2025-11-06T16:05:53Z

Summary of changes

This PR addresses benchmark flakiness detected by eliminating static constructors and moving all initialization to [GlobalSetup] methods, which BenchmarkDotNet explicitly excludes from timing measurements. Additionally, adds forced GC
collection in CharSliceBenchmark to reduce variance from unpredictable garbage collection timing.

Benchmarks fixed:

CIVisibilityProtocolWriterBenchmark (151% CV - highest variance)
CharSliceBenchmark.OriginalCharSlice (63.9% CV)
AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack (43.2% CV throughput, 34.1% CV execution)
AgentWriterBenchmark.WriteAndFlushEnrichedTraces (28-31% CV)
AspNetCoreBenchmark.SendRequest (26-27% CV)
AppSecBodyBenchmark (11-24% CV across scenarios)
StringAspectsBenchmark.StringConcatAspectBenchmark (33.3% CV)
Log4netBenchmark.EnrichedLog (5.8% CV)
AppSecEncoderBenchmark.EncodeLegacyArgs (5.3% CV)

Reason for change

Benchmark variance analysis revealed high Coefficients of Variation (CV) ranging from 5.8% to 151%, indicating unstable and unreliable performance measurements. Investigation identified two root causes:

1. Static Constructor Timing Issues

Static constructors run at CLR-controlled times, not BenchmarkDotNet-controlled times. This creates several problems:

Unpredictable initialization timing - Static constructors may execute during BenchmarkDotNet's pilot phase, warmup, or actual measurement, introducing timing inconsistencies
GC side effects - Memory allocated in static constructors increases the Large Object Heap (LOH) budget, causing subsequent Gen 2 garbage collections to take significantly longer
No warmup control - BenchmarkDotNet cannot ensure JIT compilation and warmup happen before measurement begins

2. Unmanaged Memory Allocation Without GC Control

CharSliceBenchmark allocates/frees unmanaged memory 10,000 times per iteration using Marshal.AllocHGlobal/FreeHGlobal. While unmanaged memory doesn't trigger GC directly, the managed byte arrays created during encoding do.
Without explicit GC control, garbage collection can occur at unpredictable times during measurement.

Implementation details

1. Eliminated All Static Constructors and Field Initializers

Benefits:

BenchmarkDotNet runs [GlobalSetup] explicitly before starting timers
All allocation overhead excluded from measurements
GC state stabilizes before measurement begins
JIT compilation happens during warmup, not during measurement

2. Added Forced GC in CharSliceBenchmark

Why only CharSliceBenchmark?

Only benchmark with significant unmanaged memory allocation during measurement
Allocates/frees 10,000 times per iteration
Other benchmarks reuse pre-allocated data (no need for iteration-level GC)
[IterationSetup] overhead justified by eliminating GC timing variance

3. Moved Static Field Initializers to GlobalSetup

Applied to:

AppSecEncoderBenchmark: Moved MakeNestedMap(20) allocation (~40 dictionary/list allocations)
AppSecWafBenchmark: Moved 4 MakeRealisticNestedMapStage*() calls (significant nested dictionary allocations)

4. Added Benchmark Guidelines Documentation

Design decisions:

Did not modify non-flaky benchmarks - 11 other benchmark files have static constructors but weren't in the flakiness report. Left unchanged to keep PR focused and minimize risk. Guidelines document provides pattern for future
standardization if desired.
Did not add [IterationSetup] to other benchmarks - Only CharSliceBenchmark has significant allocation during measurement. Other benchmarks reuse pre-allocated data.
Trusted BenchmarkDotNet's adaptive algorithm - Did not add iteration count overrides ([WarmupCount], [IterationCount]). Per BenchmarkDotNet documentation: "Usually, you shouldn't specify characteristics like WarmupCount or
IterationCount because BenchmarkDotNet has a smart algorithm to choose these values automatically."
Static vs instance fields - Kept fields as static where data is immutable and reused (spans, dictionaries).

Test coverage

Other details

dd-trace-dotnet-ci-bot · 2025-11-06T17:14:17Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (7790) and master.

✅ No regressions detected - check the details below

Full Metrics Comparison

FakeDbCommand

Metric	Master (Mean ± 95% CI)	Current (Mean ± 95% CI)	Change	Status
.NET Framework 4.8 - Baseline
duration	75.76 ± (75.75 - 76.44) ms	76.07 ± (76.01 - 76.83) ms	+0.4%	✅⬆️
.NET Framework 4.8 - Bailout
duration	79.19 ± (79.45 - 80.50) ms	79.16 ± (78.89 - 79.61) ms	-0.0%	✅
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration	1121.95 ± (1119.71 - 1133.13) ms	1119.62 ± (1125.30 - 1137.46) ms	-0.2%	✅
.NET Core 3.1 - Baseline
process.internal_duration_ms	22.89 ± (22.82 - 22.97) ms	22.89 ± (22.82 - 22.97) ms	+0.0%	✅⬆️
process.time_to_main_ms	87.21 ± (86.86 - 87.57) ms	86.66 ± (86.37 - 86.95) ms	-0.6%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.90 ± (10.89 - 10.90) MB	10.92 ± (10.92 - 10.93) MB	+0.2%	✅⬆️
runtime.dotnet.threads.count	12 ± (12 - 12)	12 ± (12 - 12)	+0.0%	✅
.NET Core 3.1 - Bailout
process.internal_duration_ms	22.92 ± (22.84 - 22.99) ms	22.75 ± (22.68 - 22.82) ms	-0.7%	✅
process.time_to_main_ms	89.94 ± (89.55 - 90.33) ms	88.68 ± (88.32 - 89.04) ms	-1.4%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.94 ± (10.94 - 10.95) MB	10.97 ± (10.96 - 10.97) MB	+0.2%	✅⬆️
runtime.dotnet.threads.count	13 ± (13 - 13)	13 ± (13 - 13)	+0.0%	✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms	219.34 ± (217.91 - 220.76) ms	219.02 ± (217.66 - 220.38) ms	-0.1%	✅
process.time_to_main_ms	548.40 ± (547.08 - 549.72) ms	545.07 ± (544.00 - 546.15) ms	-0.6%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	52.58 ± (52.56 - 52.60) MB	52.54 ± (52.52 - 52.56) MB	-0.1%	✅
runtime.dotnet.threads.count	28 ± (28 - 28)	28 ± (28 - 28)	+0.1%	✅⬆️
.NET 6 - Baseline
process.internal_duration_ms	22.05 ± (21.98 - 22.13) ms	21.74 ± (21.66 - 21.82) ms	-1.4%	✅
process.time_to_main_ms	78.32 ± (77.91 - 78.73) ms	76.62 ± (76.23 - 77.00) ms	-2.2%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.59 ± (10.59 - 10.59) MB	10.65 ± (10.65 - 10.65) MB	+0.5%	✅⬆️
runtime.dotnet.threads.count	10 ± (10 - 10)	10 ± (10 - 10)	+0.0%	✅
.NET 6 - Bailout
process.internal_duration_ms	21.80 ± (21.74 - 21.87) ms	21.81 ± (21.74 - 21.88) ms	+0.0%	✅⬆️
process.time_to_main_ms	78.70 ± (78.36 - 79.04) ms	77.92 ± (77.60 - 78.23) ms	-1.0%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.66 ± (10.66 - 10.66) MB	10.68 ± (10.67 - 10.68) MB	+0.2%	✅⬆️
runtime.dotnet.threads.count	11 ± (11 - 11)	11 ± (11 - 11)	+0.0%	✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms	206.85 ± (205.60 - 208.11) ms	207.46 ± (205.88 - 209.04) ms	+0.3%	✅⬆️
process.time_to_main_ms	517.87 ± (516.78 - 518.97) ms	511.98 ± (510.99 - 512.97) ms	-1.1%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	51.83 ± (51.80 - 51.86) MB	51.61 ± (51.59 - 51.64) MB	-0.4%	✅
runtime.dotnet.threads.count	28 ± (28 - 28)	28 ± (28 - 28)	-0.1%	✅
.NET 8 - Baseline
process.internal_duration_ms	19.94 ± (19.87 - 20.00) ms	20.25 ± (20.18 - 20.32) ms	+1.6%	✅⬆️
process.time_to_main_ms	76.13 ± (75.80 - 76.46) ms	77.51 ± (77.21 - 77.81) ms	+1.8%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	7.64 ± (7.63 - 7.65) MB	7.67 ± (7.66 - 7.67) MB	+0.4%	✅⬆️
runtime.dotnet.threads.count	10 ± (10 - 10)	10 ± (10 - 10)	+0.0%	✅
.NET 8 - Bailout
process.internal_duration_ms	19.86 ± (19.80 - 19.92) ms	19.99 ± (19.93 - 20.06) ms	+0.7%	✅⬆️
process.time_to_main_ms	76.54 ± (76.24 - 76.85) ms	77.23 ± (76.86 - 77.60) ms	+0.9%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	7.70 ± (7.69 - 7.70) MB	7.73 ± (7.73 - 7.74) MB	+0.5%	✅⬆️
runtime.dotnet.threads.count	11 ± (11 - 11)	11 ± (11 - 11)	+0.0%	✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms	191.89 ± (190.98 - 192.81) ms	193.98 ± (192.96 - 195.01) ms	+1.1%	✅⬆️
process.time_to_main_ms	495.38 ± (494.22 - 496.54) ms	490.12 ± (489.14 - 491.11) ms	-1.1%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	39.03 ± (38.99 - 39.07) MB	39.01 ± (38.97 - 39.05) MB	-0.0%	✅
runtime.dotnet.threads.count	27 ± (27 - 27)	27 ± (27 - 27)	+0.0%	✅⬆️

HttpMessageHandler

Metric	Master (Mean ± 95% CI)	Current (Mean ± 95% CI)	Change	Status
.NET Framework 4.8 - Baseline
duration	193.65 ± (193.42 - 194.25) ms	194.02 ± (194.21 - 195.33) ms	+0.2%	✅⬆️
.NET Framework 4.8 - Bailout
duration	197.45 ± (197.24 - 198.16) ms	198.91 ± (198.65 - 199.60) ms	+0.7%	✅⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration	1165.94 ± (1171.13 - 1180.86) ms	1182.35 ± (1182.18 - 1191.97) ms	+1.4%	✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms	188.26 ± (187.87 - 188.64) ms	189.09 ± (188.69 - 189.49) ms	+0.4%	✅⬆️
process.time_to_main_ms	80.86 ± (80.58 - 81.13) ms	81.04 ± (80.84 - 81.24) ms	+0.2%	✅⬆️
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	16.03 ± (16.01 - 16.06) MB	16.13 ± (16.11 - 16.16) MB	+0.6%	✅⬆️
runtime.dotnet.threads.count	20 ± (19 - 20)	20 ± (20 - 20)	+0.2%	✅⬆️
.NET Core 3.1 - Bailout
process.internal_duration_ms	187.78 ± (187.39 - 188.17) ms	188.17 ± (187.74 - 188.60) ms	+0.2%	✅⬆️
process.time_to_main_ms	81.80 ± (81.65 - 81.95) ms	82.28 ± (82.07 - 82.49) ms	+0.6%	✅⬆️
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	16.11 ± (16.08 - 16.14) MB	16.20 ± (16.17 - 16.23) MB	+0.6%	✅⬆️
runtime.dotnet.threads.count	21 ± (20 - 21)	21 ± (20 - 21)	-0.0%	✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms	394.74 ± (392.15 - 397.34) ms	397.77 ± (395.29 - 400.25) ms	+0.8%	✅⬆️
process.time_to_main_ms	517.34 ± (516.69 - 517.98) ms	521.58 ± (520.73 - 522.44) ms	+0.8%	✅⬆️
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	62.87 ± (62.72 - 63.03) MB	62.88 ± (62.73 - 63.03) MB	+0.0%	✅⬆️
runtime.dotnet.threads.count	29 ± (29 - 29)	29 ± (29 - 29)	+0.0%	✅⬆️
.NET 6 - Baseline
process.internal_duration_ms	192.96 ± (192.47 - 193.45) ms	193.47 ± (193.04 - 193.89) ms	+0.3%	✅⬆️
process.time_to_main_ms	70.13 ± (69.93 - 70.34) ms	70.34 ± (70.15 - 70.54) ms	+0.3%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	16.32 ± (16.27 - 16.37) MB	16.37 ± (16.32 - 16.41) MB	+0.3%	✅⬆️
runtime.dotnet.threads.count	19 ± (19 - 19)	19 ± (19 - 19)	-0.3%	✅
.NET 6 - Bailout
process.internal_duration_ms	191.18 ± (190.90 - 191.46) ms	191.65 ± (191.31 - 191.98) ms	+0.2%	✅⬆️
process.time_to_main_ms	70.83 ± (70.72 - 70.94) ms	70.87 ± (70.75 - 70.98) ms	+0.1%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	16.01 ± (15.85 - 16.16) MB	16.16 ± (16.03 - 16.30) MB	+1.0%	✅⬆️
runtime.dotnet.threads.count	19 ± (19 - 19)	20 ± (19 - 20)	+2.1%	✅⬆️
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms	413.14 ± (410.11 - 416.18) ms	412.05 ± (408.92 - 415.18) ms	-0.3%	✅
process.time_to_main_ms	490.90 ± (490.13 - 491.66) ms	484.36 ± (483.75 - 484.98) ms	-1.3%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	62.22 ± (62.08 - 62.36) MB	61.83 ± (61.69 - 61.97) MB	-0.6%	✅
runtime.dotnet.threads.count	29 ± (29 - 29)	29 ± (29 - 29)	+0.0%	✅⬆️
.NET 8 - Baseline
process.internal_duration_ms	193.74 ± (193.15 - 194.34) ms	190.47 ± (190.08 - 190.87) ms	-1.7%	✅
process.time_to_main_ms	70.75 ± (70.46 - 71.04) ms	69.59 ± (69.39 - 69.79) ms	-1.6%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	11.63 ± (11.62 - 11.65) MB	11.75 ± (11.72 - 11.78) MB	+1.0%	✅⬆️
runtime.dotnet.threads.count	18 ± (18 - 18)	18 ± (18 - 18)	-1.3%	✅
.NET 8 - Bailout
process.internal_duration_ms	192.13 ± (191.68 - 192.57) ms	190.39 ± (190.06 - 190.72) ms	-0.9%	✅
process.time_to_main_ms	71.17 ± (70.97 - 71.38) ms	70.53 ± (70.42 - 70.65) ms	-0.9%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	11.72 ± (11.70 - 11.75) MB	11.69 ± (11.61 - 11.77) MB	-0.3%	✅
runtime.dotnet.threads.count	19 ± (19 - 19)	19 ± (18 - 19)	-2.8%	✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms	366.75 ± (365.45 - 368.06) ms	365.03 ± (363.55 - 366.52) ms	-0.5%	✅
process.time_to_main_ms	471.67 ± (470.97 - 472.38) ms	461.64 ± (460.99 - 462.29) ms	-2.1%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	50.51 ± (50.48 - 50.54) MB	50.32 ± (50.29 - 50.34) MB	-0.4%	✅
runtime.dotnet.threads.count	29 ± (29 - 29)	29 ± (29 - 29)	+0.8%	✅⬆️

Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts

FakeDbCommand (.NET Framework 4.8)

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7790) - mean (76ms)  : 70, 83
    master - mean (76ms)  : 71, 81

    section Bailout
    This PR (7790) - mean (79ms)  : 74, 85
    master - mean (80ms)  : 74, 86

    section CallTarget+Inlining+NGEN
    This PR (7790) - mean (1,131ms)  : 1041, 1222
    master - mean (1,126ms)  : 1057, 1196

FakeDbCommand (.NET Core 3.1)

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7790) - mean (117ms)  : 111, 123
    master - mean (118ms)  : 111, 124

    section Bailout
    This PR (7790) - mean (119ms)  : 111, 127
    master - mean (120ms)  : 112, 128

    section CallTarget+Inlining+NGEN
    This PR (7790) - mean (803ms)  : 769, 837
    master - mean (804ms)  : 757, 851

FakeDbCommand (.NET 6)

gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7790) - mean (105ms)  : 99, 112
    master - mean (108ms)  : 100, 115

    section Bailout
    This PR (7790) - mean (107ms)  : 100, 114
    master - mean (108ms)  : 100, 115

    section CallTarget+Inlining+NGEN
    This PR (7790) - mean (751ms)  : 710, 792
    master - mean (756ms)  : 722, 789

FakeDbCommand (.NET 8)

gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7790) - mean (106ms)  : 100, 113
    master - mean (104ms)  : 97, 112

    section Bailout
    This PR (7790) - mean (106ms)  : 98, 113
    master - mean (105ms)  : 98, 112

    section CallTarget+Inlining+NGEN
    This PR (7790) - mean (717ms)  : 683, 751
    master - mean (719ms)  : 688, 750

HttpMessageHandler (.NET Framework 4.8)

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7790) - mean (195ms)  : 187, 202
    master - mean (194ms)  : 189, 199

    section Bailout
    This PR (7790) - mean (199ms)  : 194, 204
    master - mean (198ms)  : 193, 202

    section CallTarget+Inlining+NGEN
    This PR (7790) - mean (1,187ms)  : 1114, 1260
    master - mean (1,176ms)  : 1103, 1249

HttpMessageHandler (.NET Core 3.1)

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7790) - mean (278ms)  : 271, 286
    master - mean (277ms)  : 272, 283

    section Bailout
    This PR (7790) - mean (278ms)  : 273, 284
    master - mean (278ms)  : 273, 282

    section CallTarget+Inlining+NGEN
    This PR (7790) - mean (955ms)  : 908, 1002
    master - mean (942ms)  : 905, 980

HttpMessageHandler (.NET 6)

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7790) - mean (272ms)  : 264, 280
    master - mean (271ms)  : 265, 277

    section Bailout
    This PR (7790) - mean (270ms)  : 266, 274
    master - mean (270ms)  : 267, 273

    section CallTarget+Inlining+NGEN
    This PR (7790) - mean (929ms)  : 874, 984
    master - mean (937ms)  : 888, 986

HttpMessageHandler (.NET 8)

gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7790) - mean (270ms)  : 264, 277
    master - mean (274ms)  : 261, 288

    section Bailout
    This PR (7790) - mean (270ms)  : 266, 275
    master - mean (273ms)  : 266, 280

    section CallTarget+Inlining+NGEN
    This PR (7790) - mean (857ms)  : 825, 889
    master - mean (870ms)  : 846, 895

NachoEchevarria · 2025-11-07T11:44:27Z

tracer/test/benchmarks/Benchmarks.Trace/README_GUIDELINES.md

@@ -0,0 +1,109 @@
+# Benchmark Guidelines


Guide generated with Claude code.

igoragoli · 2025-11-12T16:00:16Z

Thanks for working on this, @NachoEchevarria!

Seeing errors like these on the CI:

 [ERR] RunBenchmarks: C:\dd-trace-dotnet\tracer\test\benchmarks\Benchmarks.Trace\bin\Release\net6.0\2b73268d-661e-4034-861a-8c6b24014cbc\2b73268d-661e-4034-861a-8c6b24014cbc.notcs(592,33): error CS0103: The name 'GlobalSetup' does not exist in the current context [C:\dd-trace-dotnet\tracer\test\benchmarks\Benchmarks.Trace\bin\Release\net6.0\2b73268d-661e-4034-861a-8c6b24014cbc\BenchmarkDotNet.Autogenerated.csproj]

The benchmarks completes successfully, though.

Because of a rate limiting issue, the script failed to run the PR commenter¹ and failed to upload results to the BP UI (which was done after the PR comment). I'm rue-running the benchmarks so we can get results on the BP UI and I can run the /flakiness endpoint.

That said, looks good to me, excited to see the results.

I took note of that for our team, this shouldn't happen. ↩

pr-commenter · 2025-11-12T16:43:17Z

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmarks for #7790 compared to master:

1 benchmarks are faster, with geometric mean 2.170
9 benchmarks are slower, with geometric mean 1.423
10 benchmarks have fewer allocations
10 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7790

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑net472	6.12 KB	6 KB	-123 B	-2.01%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`StartStopWithChild`	net6.0	10.7μs	58.9ns	353ns	0	0	5.52 KB
master	`StartStopWithChild`	netcoreapp3.1	13.7μs	63.4ns	237ns	0	0	5.71 KB
master	`StartStopWithChild`	net472	22μs	119ns	662ns	0.97	0.323	6.12 KB
#7790	`StartStopWithChild`	net6.0	10.7μs	59.9ns	379ns	0	0	5.51 KB
#7790	`StartStopWithChild`	netcoreapp3.1	13.8μs	70.8ns	339ns	0	0	5.7 KB
#7790	`StartStopWithChild`	net472	21.7μs	114ns	556ns	0.879	0.22	6 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	924μs	96.6ns	374ns	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	1.03ms	3.56μs	13.8μs	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	1.22ms	541ns	2.1μs	3.31 KB
#7790	`WriteAndFlushEnrichedTraces`	net6.0	910μs	43ns	155ns	2.71 KB
#7790	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	1.06ms	213ns	823ns	2.7 KB
#7790	`WriteAndFlushEnrichedTraces`	net472	1.23ms	85.4ns	320ns	3.31 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`AllCycleSimpleBody`	net6.0	1.06μs	6.13ns	52ns	0	1.22 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	1.42μs	6.87ns	28.3ns	0	1.2 KB
master	`AllCycleSimpleBody`	net472	1.03μs	0.253ns	0.945ns	0.195	1.23 KB
master	`AllCycleMoreComplexBody`	net6.0	7.12μs	23.4ns	90.6ns	0	4.72 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	8.89μs	36.8ns	143ns	0	4.62 KB
master	`AllCycleMoreComplexBody`	net472	7.65μs	2.02ns	7.84ns	0.73	4.74 KB
master	`ObjectExtractorSimpleBody`	net6.0	321ns	1.74ns	9.05ns	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	393ns	2.23ns	14.9ns	0	272 B
master	`ObjectExtractorSimpleBody`	net472	293ns	0.0253ns	0.0946ns	0.0432	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	6.39μs	18.5ns	71.7ns	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	7.8μs	35.7ns	143ns	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	6.76μs	9.88ns	38.3ns	0.572	3.8 KB
#7790	`AllCycleSimpleBody`	net6.0	1.1μs	0.76ns	2.94ns	0	1.22 KB
#7790	`AllCycleSimpleBody`	netcoreapp3.1	1.35μs	7.72ns	59.3ns	0	1.2 KB
#7790	`AllCycleSimpleBody`	net472	1.04μs	0.944ns	3.66ns	0.192	1.23 KB
#7790	`AllCycleMoreComplexBody`	net6.0	7.44μs	38.7ns	197ns	0	4.72 KB
#7790	`AllCycleMoreComplexBody`	netcoreapp3.1	8.86μs	46.4ns	223ns	0	4.62 KB
#7790	`AllCycleMoreComplexBody`	net472	7.6μs	4.63ns	17.9ns	0.724	4.74 KB
#7790	`ObjectExtractorSimpleBody`	net6.0	343ns	1.74ns	7.57ns	0	280 B
#7790	`ObjectExtractorSimpleBody`	netcoreapp3.1	394ns	2.14ns	12.1ns	0	272 B
#7790	`ObjectExtractorSimpleBody`	net472	295ns	0.124ns	0.48ns	0.0434	281 B
#7790	`ObjectExtractorMoreComplexBody`	net6.0	6.24μs	31.4ns	151ns	0	3.78 KB
#7790	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	7.75μs	35.9ns	143ns	0	3.69 KB
#7790	`ObjectExtractorMoreComplexBody`	net472	6.7μs	2.25ns	8.43ns	0.572	3.8 KB

Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EncodeArgs`	net6.0	76.9μs	292ns	1.13μs	0	32.4 KB
master	`EncodeArgs`	netcoreapp3.1	96.3μs	253ns	982ns	0	32.4 KB
master	`EncodeArgs`	net472	111μs	16.6ns	64.4ns	5.01	32.5 KB
master	`EncodeLegacyArgs`	net6.0	142μs	58ns	217ns	0	2.15 KB
master	`EncodeLegacyArgs`	netcoreapp3.1	199μs	246ns	951ns	0	2.14 KB
master	`EncodeLegacyArgs`	net472	264μs	28.5ns	107ns	0	2.17 KB
#7790	`EncodeArgs`	net6.0	77.1μs	22.4ns	83.7ns	0	32.4 KB
#7790	`EncodeArgs`	netcoreapp3.1	98.9μs	201ns	779ns	0	32.4 KB
#7790	`EncodeArgs`	net472	110μs	67.7ns	262ns	4.98	32.51 KB
#7790	`EncodeLegacyArgs`	net6.0	144μs	26.2ns	98ns	0	2.15 KB
#7790	`EncodeLegacyArgs`	netcoreapp3.1	197μs	94.2ns	340ns	0	2.14 KB
#7790	`EncodeLegacyArgs`	net472	260μs	30.3ns	117ns	0	2.17 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Slower ⚠️ More allocations ⚠️

Slower ⚠️ in #7790

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack‑net472	1.211	313,166.25	379,300.00
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark‑net472	1.194	427,072.40	510,100.00
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack‑net6.0	1.132	283,117.43	320,350.00

Faster 🎉 in #7790

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack‑netcoreapp3.1	2.170	715,761.46	329,900.00

More allocations ⚠️ in #7790

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack‑net6.0	2.24 KB	2.54 KB	303 B	13.52%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark‑net6.0	4.55 KB	4.86 KB	302 B	6.63%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack‑netcoreapp3.1	2.22 KB	2.32 KB	104 B	4.69%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark‑netcoreapp3.1	4.48 KB	4.58 KB	104 B	2.32%

Fewer allocations 🎉 in #7790

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark‑net472	4.66 KB	0 b	-4.66 KB	-100.00%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack‑net472	2.29 KB	0 b	-2.29 KB	-100.00%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`RunWafRealisticBenchmark`	net6.0	396μs	90.5ns	339ns	4.55 KB
master	`RunWafRealisticBenchmark`	netcoreapp3.1	409μs	114ns	428ns	4.48 KB
master	`RunWafRealisticBenchmark`	net472	427μs	48.2ns	187ns	4.66 KB
master	`RunWafRealisticBenchmarkWithAttack`	net6.0	283μs	45ns	168ns	2.24 KB
master	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	708μs	3.76μs	19.5μs	2.22 KB
master	`RunWafRealisticBenchmarkWithAttack`	net472	313μs	27.6ns	107ns	2.29 KB
#7790	`RunWafRealisticBenchmark`	net6.0	445μs	1.92μs	6.93μs	4.86 KB
#7790	`RunWafRealisticBenchmark`	netcoreapp3.1	462μs	2.71μs	24.6μs	4.58 KB
#7790	`RunWafRealisticBenchmark`	net472	510μs	249ns	932ns	0 b
#7790	`RunWafRealisticBenchmarkWithAttack`	net6.0	324μs	1.65μs	7.55μs	2.54 KB
#7790	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	331μs	1.29μs	4.66μs	2.32 KB
#7790	`RunWafRealisticBenchmarkWithAttack`	net472	382μs	1.22μs	4.72μs	0 b

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`SendRequest`	net6.0	61.5μs	97ns	350ns	14.52 KB
master	`SendRequest`	netcoreapp3.1	72.1μs	120ns	434ns	17.42 KB
master	`SendRequest`	net472	0.00444ns	0.00159ns	0.00618ns	0 b
#7790	`SendRequest`	net6.0	61.7μs	153ns	591ns	14.52 KB
#7790	`SendRequest`	netcoreapp3.1	72.1μs	205ns	740ns	17.42 KB
#7790	`SendRequest`	net472	0.164ns	0.00397ns	0.0154ns	0 b

Benchmarks.Trace.CharSliceBenchmark - Slower ⚠️ More allocations ⚠️

Slower ⚠️ in #7790

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑netcoreapp3.1	2.296	812,986.16	1,867,000.00
Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice‑netcoreapp3.1	1.886	2,086,749.22	3,935,850.00
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑netcoreapp3.1	1.700	1,656,978.75	2,816,450.00
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net6.0	1.376	839,533.01	1,154,800.00
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑net6.0	1.255	1,341,038.89	1,682,550.00

More allocations ⚠️ in #7790

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net6.0	2 B	640 B	638 B	31,900.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑netcoreapp3.1	1 B	104 B	103 B	10,300.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑netcoreapp3.1	1 B	104 B	103 B	10,300.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑net6.0	7 B	640 B	633 B	9,042.86%
Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice‑net472	641.95 KB	647.17 KB	5.21 KB	0.81%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`OriginalCharSlice`	net6.0	1.88ms	4.78μs	18.5μs	0	640.01 KB
master	`OriginalCharSlice`	netcoreapp3.1	2.08ms	5.69μs	22.1μs	0	640 KB
master	`OriginalCharSlice`	net472	2.58ms	73ns	273ns	100	641.95 KB
master	`OptimizedCharSlice`	net6.0	1.34ms	239ns	926ns	0	7 B
master	`OptimizedCharSlice`	netcoreapp3.1	1.66ms	411ns	1.59μs	0	1 B
master	`OptimizedCharSlice`	net472	2.01ms	222ns	830ns	0	0 b
master	`OptimizedCharSliceWithPool`	net6.0	840μs	77.1ns	289ns	0	2 B
master	`OptimizedCharSliceWithPool`	netcoreapp3.1	813μs	53.5ns	200ns	0	1 B
master	`OptimizedCharSliceWithPool`	net472	1.13ms	90.9ns	352ns	0	0 b
#7790	`OriginalCharSlice`	net6.0	1.92ms	1.67μs	6.48μs	0	640.64 KB
#7790	`OriginalCharSlice`	netcoreapp3.1	3.94ms	1.02μs	3.82μs	0	640.1 KB
#7790	`OriginalCharSlice`	net472	2.8ms	684ns	2.47μs	0	647.17 KB
#7790	`OptimizedCharSlice`	net6.0	1.68ms	605ns	2.18μs	0	640 B
#7790	`OptimizedCharSlice`	netcoreapp3.1	2.82ms	478ns	1.79μs	0	104 B
#7790	`OptimizedCharSlice`	net472	2.03ms	734ns	2.84μs	0	0 b
#7790	`OptimizedCharSliceWithPool`	net6.0	1.15ms	210ns	757ns	0	640 B
#7790	`OptimizedCharSliceWithPool`	netcoreapp3.1	1.87ms	1.73μs	6.47μs	0	104 B
#7790	`OptimizedCharSliceWithPool`	net472	1.15ms	497ns	1.92μs	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Slower ⚠️ More allocations ⚠️

Slower ⚠️ in #7790

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net472	1.147	886,784.58	1,017,373.75

More allocations ⚠️ in #7790

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1	41.86 KB	42.29 KB	432 B	1.03%

Fewer allocations 🎉 in #7790

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0	42.32 KB	41.79 KB	-533 B	-1.26%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	705μs	1.68μs	7.86μs	0	42.32 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	782μs	1.09μs	4.21μs	0	41.86 KB
master	`WriteAndFlushEnrichedTraces`	net472	888μs	2.64μs	10.2μs	8.33	56.29 KB
#7790	`WriteAndFlushEnrichedTraces`	net6.0	700μs	883ns	3.06μs	0	41.79 KB
#7790	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	687μs	4.03μs	39.7μs	0	42.29 KB
#7790	`WriteAndFlushEnrichedTraces`	net472	1ms	5.85μs	52μs	8.33	56.49 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.97μs	2.62ns	10.1ns	0	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	2.64μs	9.62ns	34.7ns	0	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.91μs	8.4ns	32.5ns	0.146	0.0146	987 B
#7790	`ExecuteNonQuery`	net6.0	1.93μs	9.29ns	36ns	0	0	1.02 KB
#7790	`ExecuteNonQuery`	netcoreapp3.1	2.59μs	11.8ns	45.8ns	0	0	1.02 KB
#7790	`ExecuteNonQuery`	net472	2.9μs	5.02ns	19.4ns	0.146	0.0146	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.69μs	8.88ns	43.5ns	0	1.03 KB
master	`CallElasticsearch`	netcoreapp3.1	2.35μs	9.14ns	35.4ns	0	1.03 KB
master	`CallElasticsearch`	net472	3.51μs	2.12ns	8.19ns	0.158	1.04 KB
master	`CallElasticsearchAsync`	net6.0	1.8μs	8.94ns	36.9ns	0	1.01 KB
master	`CallElasticsearchAsync`	netcoreapp3.1	2.35μs	12.3ns	59.1ns	0	1.08 KB
master	`CallElasticsearchAsync`	net472	3.66μs	2.06ns	7.99ns	0.164	1.1 KB
#7790	`CallElasticsearch`	net6.0	1.7μs	9.06ns	45.3ns	0	1.03 KB
#7790	`CallElasticsearch`	netcoreapp3.1	2.24μs	9.62ns	37.3ns	0	1.03 KB
#7790	`CallElasticsearch`	net472	3.51μs	1.19ns	4.62ns	0.157	1.04 KB
#7790	`CallElasticsearchAsync`	net6.0	1.88μs	3.73ns	14.4ns	0	1.01 KB
#7790	`CallElasticsearchAsync`	netcoreapp3.1	2.36μs	3.95ns	15.3ns	0	1.08 KB
#7790	`CallElasticsearchAsync`	net472	3.62μs	2.18ns	8.16ns	0.163	1.1 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	2μs	7.29ns	28.2ns	0	952 B
master	`ExecuteAsync`	netcoreapp3.1	2.43μs	6.52ns	25.2ns	0	952 B
master	`ExecuteAsync`	net472	2.63μs	2.07ns	8.03ns	0.145	915 B
#7790	`ExecuteAsync`	net6.0	1.87μs	4.75ns	18.4ns	0	952 B
#7790	`ExecuteAsync`	netcoreapp3.1	2.39μs	7.99ns	31ns	0	952 B
#7790	`ExecuteAsync`	net472	2.52μs	0.724ns	2.71ns	0.139	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	6.8μs	18ns	67.5ns	0	2.36 KB
master	`SendAsync`	netcoreapp3.1	8.49μs	7.13ns	27.6ns	0	2.9 KB
master	`SendAsync`	net472	12.4μs	10.8ns	41.9ns	0.495	3.18 KB
#7790	`SendAsync`	net6.0	6.98μs	5.23ns	18.9ns	0	2.36 KB
#7790	`SendAsync`	netcoreapp3.1	8.53μs	19.6ns	73.2ns	0	2.9 KB
#7790	`SendAsync`	net472	12.2μs	9.23ns	35.8ns	0.489	3.18 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Unknown 🤷 Fewer allocations 🎉

Fewer allocations 🎉 in #7790

Benchmark	Base Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net6.0	43.44 KB	-43.44 KB	-100.00%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑netcoreapp3.1	42.8 KB	-42.8 KB	-100.00%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472	57.34 KB	-57.34 KB	-100.00%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0	276.95 KB	-276.95 KB	-100.00%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1	255.84 KB	-255.84 KB	-100.00%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net472	278.53 KB	-278.53 KB	-100.00%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StringConcatBenchmark`	net6.0	42.1μs	202ns	834ns	0	0	0	43.44 KB
master	`StringConcatBenchmark`	netcoreapp3.1	51.8μs	395ns	3.77μs	0	0	0	42.8 KB
master	`StringConcatBenchmark`	net472	56.9μs	281ns	1.29μs	0	0	0	57.34 KB
master	`StringConcatAspectBenchmark`	net6.0	499μs	1.52μs	5.49μs	0	0	0	276.95 KB
master	`StringConcatAspectBenchmark`	netcoreapp3.1	510μs	2.23μs	8.34μs	0	0	0	255.84 KB
master	`StringConcatAspectBenchmark`	net472	417μs	2.33μs	14.7μs	0	0	0	278.53 KB
#7790	`StringConcatBenchmark`	net6.0	N/A	N/A	N/A	NaN	NaN	NaN	0 b
#7790	`StringConcatBenchmark`	netcoreapp3.1	N/A	N/A	N/A	NaN	NaN	NaN	0 b
#7790	`StringConcatBenchmark`	net472	N/A	N/A	N/A	NaN	NaN	NaN	0 b
#7790	`StringConcatAspectBenchmark`	net6.0	N/A	N/A	N/A	NaN	NaN	NaN	0 b
#7790	`StringConcatAspectBenchmark`	netcoreapp3.1	N/A	N/A	N/A	NaN	NaN	NaN	0 b
#7790	`StringConcatAspectBenchmark`	net472	N/A	N/A	N/A	NaN	NaN	NaN	0 b

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.65μs	13.8ns	67.8ns	0	1.7 KB
master	`EnrichedLog`	netcoreapp3.1	3.57μs	16.5ns	66ns	0	1.7 KB
master	`EnrichedLog`	net472	3.85μs	3.42ns	13.2ns	0.249	1.64 KB
#7790	`EnrichedLog`	net6.0	2.59μs	13.4ns	65.4ns	0	1.7 KB
#7790	`EnrichedLog`	netcoreapp3.1	3.62μs	2.96ns	11.5ns	0	1.7 KB
#7790	`EnrichedLog`	net472	4.09μs	7.24ns	28ns	0.244	1.64 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`EnrichedLog`	net6.0	128μs	240ns	928ns	4.31 KB
master	`EnrichedLog`	netcoreapp3.1	133μs	275ns	953ns	4.31 KB
master	`EnrichedLog`	net472	173μs	412ns	1.59μs	4.52 KB
#7790	`EnrichedLog`	net6.0	124μs	58.3ns	202ns	4.31 KB
#7790	`EnrichedLog`	netcoreapp3.1	128μs	93.2ns	349ns	4.31 KB
#7790	`EnrichedLog`	net472	169μs	175ns	678ns	4.52 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	5.3μs	24.1ns	96.5ns	0	2.26 KB
master	`EnrichedLog`	netcoreapp3.1	6.78μs	18.5ns	71.8ns	0	2.26 KB
master	`EnrichedLog`	net472	7.56μs	7.28ns	28.2ns	0.304	2.08 KB
#7790	`EnrichedLog`	net6.0	5.16μs	9.81ns	38ns	0	2.26 KB
#7790	`EnrichedLog`	netcoreapp3.1	6.77μs	25.2ns	97.4ns	0	2.26 KB
#7790	`EnrichedLog`	net472	7.56μs	3.96ns	15.3ns	0.302	2.08 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	2.08μs	9.78ns	40.3ns	0	1.2 KB
master	`SendReceive`	netcoreapp3.1	2.69μs	10.3ns	39.7ns	0	1.2 KB
master	`SendReceive`	net472	3.03μs	3.68ns	14.2ns	0.181	1.2 KB
#7790	`SendReceive`	net6.0	1.95μs	10.5ns	55.7ns	0	1.2 KB
#7790	`SendReceive`	netcoreapp3.1	2.67μs	12.7ns	54ns	0	1.2 KB
#7790	`SendReceive`	net472	3.09μs	4.35ns	16.9ns	0.185	1.2 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	4.46μs	6.8ns	25.4ns	0	1.58 KB
master	`EnrichedLog`	netcoreapp3.1	5.71μs	13.8ns	53.4ns	0	1.63 KB
master	`EnrichedLog`	net472	6.49μs	6.2ns	24ns	0.321	2.03 KB
#7790	`EnrichedLog`	net6.0	4.33μs	7.55ns	28.2ns	0	1.58 KB
#7790	`EnrichedLog`	netcoreapp3.1	5.71μs	13.3ns	51.5ns	0	1.63 KB
#7790	`EnrichedLog`	net472	6.64μs	3.55ns	12.8ns	0.3	2.03 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	756ns	4.12ns	22.6ns	0	576 B
master	`StartFinishSpan`	netcoreapp3.1	958ns	4.63ns	18.5ns	0	576 B
master	`StartFinishSpan`	net472	957ns	0.413ns	1.6ns	0.0913	578 B
master	`StartFinishScope`	net6.0	972ns	5.06ns	23.8ns	0	696 B
master	`StartFinishScope`	netcoreapp3.1	1.2μs	0.445ns	1.73ns	0	696 B
master	`StartFinishScope`	net472	1.12μs	1.08ns	4.18ns	0.1	658 B
#7790	`StartFinishSpan`	net6.0	762ns	0.43ns	1.67ns	0	576 B
#7790	`StartFinishSpan`	netcoreapp3.1	985ns	5.34ns	28.8ns	0	576 B
#7790	`StartFinishSpan`	net472	925ns	0.789ns	3.06ns	0.0877	578 B
#7790	`StartFinishScope`	net6.0	922ns	4.36ns	17.4ns	0	696 B
#7790	`StartFinishScope`	netcoreapp3.1	1.22μs	5.38ns	20.8ns	0	696 B
#7790	`StartFinishScope`	net472	1.12μs	0.84ns	3.14ns	0.1	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	1.06μs	5.29ns	26.5ns	0	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	1.46μs	3.98ns	15.4ns	0	696 B
master	`RunOnMethodBegin`	net472	1.47μs	1.73ns	6.71ns	0.103	658 B
#7790	`RunOnMethodBegin`	net6.0	1.07μs	4.99ns	20ns	0	696 B
#7790	`RunOnMethodBegin`	netcoreapp3.1	1.43μs	6.76ns	26.2ns	0	696 B
#7790	`RunOnMethodBegin`	net472	1.46μs	1.01ns	3.91ns	0.102	658 B

NachoEchevarria · 2025-11-12T17:38:49Z

Thanks for working on this, @NachoEchevarria!

Seeing errors like these on the CI:
 [ERR] RunBenchmarks: C:\dd-trace-dotnet\tracer\test\benchmarks\Benchmarks.Trace\bin\Release\net6.0\2b73268d-661e-4034-861a-8c6b24014cbc\2b73268d-661e-4034-861a-8c6b24014cbc.notcs(592,33): error CS0103: The name 'GlobalSetup' does not exist in the current context [C:\dd-trace-dotnet\tracer\test\benchmarks\Benchmarks.Trace\bin\Release\net6.0\2b73268d-661e-4034-861a-8c6b24014cbc\BenchmarkDotNet.Autogenerated.csproj]
The benchmarks completes successfully, though.

Because of a rate limiting issue, the script failed to run the PR commenter1 and failed to upload results to the BP UI (which was done after the PR comment). I'm rue-running the benchmarks so we can get results on the BP UI and I can run the /flakiness endpoint.

That said, looks good to me, excited to see the results.

Footnotes

I took note of that for our team, this shouldn't happen. ↩

Thanks! I will check the cause of the errors.

igoragoli · 2025-11-13T11:14:46Z

I ran the flakiness check again for 0f5d7f6, looks like there were some big improvements!

E.g., Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net6.0 went from 0.478733 to 0.004474.

Full check: https://gist.github.com/igoragoli/806c821ba255e1a8fcad6538307ae323

igoragoli · 2025-11-14T18:02:14Z

I simplified the /flakiness endpoint, so tests are slightly different.

Full check comparing this branch at 82bfad against master at 8603e6d: https://docs.google.com/spreadsheets/d/1LKKYSYefti9l4AVA1imZSssPes0aDuqkSUD7zfxlONw/edit?usp=sharing

igoragoli

Tests are a bit inconclusive (there's still a bit flakiness and many benchmarks on this branch weren't run), but approving since changes look good to me!

NachoEchevarria added 2 commits November 6, 2025 16:47

Rely on [GlobalSetup] instead of constructors

3e55742

Benchmark guidelines

90e6e7c

github-actions bot added the area:tests unit tests, integration tests label Nov 6, 2025

Move to globalSetup

0f5d7f6

NachoEchevarria commented Nov 7, 2025

View reviewed changes

igoragoli self-requested a review November 12, 2025 13:21

NachoEchevarria added 3 commits November 13, 2025 11:20

Fix GlobalSetup Error

9682e42

Fix StringAspectBenchmark

9ff09f3

Fix possible null exception

58412b0

This comment has been minimized.

Sign in to view

Try to improve flakyness in flaky tests

82bfad7

igoragoli approved these changes Nov 14, 2025

View reviewed changes

NachoEchevarria changed the title ~~Fix benchmark flakyness~~ Fix benchmark flakiness Nov 17, 2025

NachoEchevarria added 2 commits November 17, 2025 18:04

make static some instance fields

9418a8e

Restore IterationSetup

d129961

Fix benchmark flakiness #7790

Are you sure you want to change the base?

Fix benchmark flakiness #7790

Uh oh!

Conversation

NachoEchevarria commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Reason for change

1. Static Constructor Timing Issues

2. Unmanaged Memory Allocation Without GC Control

Implementation details

1. Eliminated All Static Constructors and Field Initializers

2. Added Forced GC in CharSliceBenchmark

3. Moved Static Field Initializers to GlobalSetup

4. Added Benchmark Guidelines Documentation

Design decisions:

Test coverage

Other details

Uh oh!

dd-trace-dotnet-ci-bot bot commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Execution-Time Benchmarks Report ⏱️

FakeDbCommand

HttpMessageHandler

Uh oh!

NachoEchevarria Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

igoragoli commented Nov 12, 2025

Footnotes

Uh oh!

pr-commenter bot commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmark details

Fewer allocations 🎉 in #7790

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #7790

Faster 🎉 in #7790

More allocations ⚠️ in #7790

Fewer allocations 🎉 in #7790

Raw results

Raw results

Slower ⚠️ in #7790

More allocations ⚠️ in #7790

Raw results

Slower ⚠️ in #7790

More allocations ⚠️ in #7790

Fewer allocations 🎉 in #7790

Raw results

Raw results

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #7790

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Uh oh!

NachoEchevarria commented Nov 12, 2025

Footnotes

Uh oh!

igoragoli commented Nov 13, 2025

Uh oh!

This comment has been minimized.

igoragoli commented Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

igoragoli left a comment

Choose a reason for hiding this comment

NachoEchevarria commented Nov 6, 2025 •

edited

Loading

dd-trace-dotnet-ci-bot bot commented Nov 6, 2025 •

edited

Loading

pr-commenter bot commented Nov 12, 2025 •

edited

Loading

igoragoli commented Nov 14, 2025 •

edited

Loading