🪞 9757 - Avoid pending queue wedge if tracer flare is generated multiple times #9840
+56
−16
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In DumpDrain, the collectTraces method replaces the 'data' field with an empty ArrayList, but at the same time, it does not also reset the 'index' field. If another dump is performed later, this leads the get method reaching the 'return null' statement, and as the comment states, this can (and does) break the queue. This change does a few things: - Resets the index in collectTraces when the data field is replaced (and marks the index field as volatile). This should prevent the above situation from happening. - In case the situation still happens, a stand-in CommandElement is returned to avoid returning null. A warning message is also logged. - The existing "testing tracer flare dump with multiple traces" test case is expanded to exercise problem. Here is an example stack trace when the hang happens: "dd-trace-monitor" #38 daemon prio=5 os_prio=31 tid=0x0000000110e6e000 nid=0x7617 runnable [0x0000000171032000] java.lang.Thread.State: RUNNABLE at org.jctools.queues.MpscBlockingConsumerArrayQueue.spinWaitForElement(MpscBlockingConsumerArrayQueue.java:634) at org.jctools.queues.MpscBlockingConsumerArrayQueue.parkUntilNext(MpscBlockingConsumerArrayQueue.java:566) at org.jctools.queues.MpscBlockingConsumerArrayQueue.take(MpscBlockingConsumerArrayQueue.java:482) at datadog.trace.core.PendingTraceBuffer$DelayingPendingTraceBuffer$Worker.run(PendingTraceBuffer.java:317) at java.lang.Thread.run(Thread.java:750)
mhlidd
added
type: enhancement
|
🎯 Code Coverage 🔗 Commit SHA: 3f35463 | Docs | Was this helpful? Give us feedback! |
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 53 metrics, 12 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.55.0-SNAPSHOT~3f35463cf8, baseline=1.55.0-SNAPSHOT~cdebbd8dc3
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.03 s) : 0, 1030003
Total [baseline] (10.781 s) : 0, 10781266
Agent [candidate] (1.031 s) : 0, 1030590
Total [candidate] (10.82 s) : 0, 10819923
section appsec
Agent [baseline] (1.207 s) : 0, 1206713
Total [baseline] (10.992 s) : 0, 10992385
Agent [candidate] (1.202 s) : 0, 1201854
Total [candidate] (10.868 s) : 0, 10867882
section iast
Agent [baseline] (1.157 s) : 0, 1157115
Total [baseline] (11.06 s) : 0, 11059895
Agent [candidate] (1.162 s) : 0, 1162339
Total [candidate] (11.012 s) : 0, 11012005
section profiling
Agent [baseline] (1.179 s) : 0, 1179158
Total [baseline] (10.841 s) : 0, 10840897
Agent [candidate] (1.177 s) : 0, 1176957
Total [candidate] (10.897 s) : 0, 10897277
gantt
title petclinic - break down per module: candidate=1.55.0-SNAPSHOT~3f35463cf8, baseline=1.55.0-SNAPSHOT~cdebbd8dc3
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.46 ms) : 0, 1460
crashtracking [candidate] (1.462 ms) : 0, 1462
BytebuddyAgent [baseline] (702.215 ms) : 0, 702215
BytebuddyAgent [candidate] (703.502 ms) : 0, 703502
GlobalTracer [baseline] (244.89 ms) : 0, 244890
GlobalTracer [candidate] (244.471 ms) : 0, 244471
AppSec [baseline] (32.344 ms) : 0, 32344
AppSec [candidate] (32.158 ms) : 0, 32158
Debugger [baseline] (6.37 ms) : 0, 6370
Debugger [candidate] (6.386 ms) : 0, 6386
Remote Config [baseline] (674.824 µs) : 0, 675
Remote Config [candidate] (668.752 µs) : 0, 669
Telemetry [baseline] (14.431 ms) : 0, 14431
Telemetry [candidate] (15.098 ms) : 0, 15098
Flare Poller [baseline] (6.427 ms) : 0, 6427
Flare Poller [candidate] (5.72 ms) : 0, 5720
section appsec
crashtracking [baseline] (1.474 ms) : 0, 1474
crashtracking [candidate] (1.469 ms) : 0, 1469
BytebuddyAgent [baseline] (728.886 ms) : 0, 728886
BytebuddyAgent [candidate] (725.169 ms) : 0, 725169
GlobalTracer [baseline] (236.954 ms) : 0, 236954
GlobalTracer [candidate] (235.768 ms) : 0, 235768
AppSec [baseline] (174.226 ms) : 0, 174226
AppSec [candidate] (174.45 ms) : 0, 174450
Debugger [baseline] (5.877 ms) : 0, 5877
Debugger [candidate] (5.954 ms) : 0, 5954
Remote Config [baseline] (622.459 µs) : 0, 622
Remote Config [candidate] (635.833 µs) : 0, 636
Telemetry [baseline] (8.433 ms) : 0, 8433
Telemetry [candidate] (8.393 ms) : 0, 8393
Flare Poller [baseline] (3.854 ms) : 0, 3854
Flare Poller [candidate] (3.958 ms) : 0, 3958
IAST [baseline] (25.088 ms) : 0, 25088
IAST [candidate] (24.923 ms) : 0, 24923
section iast
crashtracking [baseline] (1.464 ms) : 0, 1464
crashtracking [candidate] (1.463 ms) : 0, 1463
BytebuddyAgent [baseline] (820.683 ms) : 0, 820683
BytebuddyAgent [candidate] (824.293 ms) : 0, 824293
GlobalTracer [baseline] (233.326 ms) : 0, 233326
GlobalTracer [candidate] (234.343 ms) : 0, 234343
AppSec [baseline] (27.968 ms) : 0, 27968
AppSec [candidate] (29.642 ms) : 0, 29642
Debugger [baseline] (6.077 ms) : 0, 6077
Debugger [candidate] (6.191 ms) : 0, 6191
Remote Config [baseline] (599.1 µs) : 0, 599
Remote Config [candidate] (606.014 µs) : 0, 606
Telemetry [baseline] (8.431 ms) : 0, 8431
Telemetry [candidate] (8.467 ms) : 0, 8467
Flare Poller [baseline] (4.167 ms) : 0, 4167
Flare Poller [candidate] (4.114 ms) : 0, 4114
IAST [baseline] (33.108 ms) : 0, 33108
IAST [candidate] (31.876 ms) : 0, 31876
section profiling
crashtracking [baseline] (1.45 ms) : 0, 1450
crashtracking [candidate] (1.446 ms) : 0, 1446
BytebuddyAgent [baseline] (730.268 ms) : 0, 730268
BytebuddyAgent [candidate] (729.663 ms) : 0, 729663
GlobalTracer [baseline] (221.388 ms) : 0, 221388
GlobalTracer [candidate] (220.238 ms) : 0, 220238
AppSec [baseline] (33.386 ms) : 0, 33386
AppSec [candidate] (32.473 ms) : 0, 32473
Debugger [baseline] (12.381 ms) : 0, 12381
Debugger [candidate] (13.089 ms) : 0, 13089
Remote Config [baseline] (706.2 µs) : 0, 706
Remote Config [candidate] (704.485 µs) : 0, 704
Telemetry [baseline] (9.669 ms) : 0, 9669
Telemetry [candidate] (9.9 ms) : 0, 9900
Flare Poller [baseline] (4.175 ms) : 0, 4175
Flare Poller [candidate] (4.178 ms) : 0, 4178
ProfilingAgent [baseline] (110.533 ms) : 0, 110533
ProfilingAgent [candidate] (109.95 ms) : 0, 109950
Profiling [baseline] (111.123 ms) : 0, 111123
Profiling [candidate] (110.562 ms) : 0, 110562
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.55.0-SNAPSHOT~3f35463cf8, baseline=1.55.0-SNAPSHOT~cdebbd8dc3
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.03 s) : 0, 1029615
Total [baseline] (8.66 s) : 0, 8660042
Agent [candidate] (1.037 s) : 0, 1036711
Total [candidate] (8.697 s) : 0, 8696731
section iast
Agent [baseline] (1.168 s) : 0, 1167948
Total [baseline] (9.354 s) : 0, 9354083
Agent [candidate] (1.158 s) : 0, 1158372
Total [candidate] (9.34 s) : 0, 9340041
gantt
title insecure-bank - break down per module: candidate=1.55.0-SNAPSHOT~3f35463cf8, baseline=1.55.0-SNAPSHOT~cdebbd8dc3
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.47 ms) : 0, 1470
crashtracking [candidate] (1.485 ms) : 0, 1485
BytebuddyAgent [baseline] (702.899 ms) : 0, 702899
BytebuddyAgent [candidate] (706.825 ms) : 0, 706825
GlobalTracer [baseline] (244.228 ms) : 0, 244228
GlobalTracer [candidate] (246.118 ms) : 0, 246118
AppSec [baseline] (32.431 ms) : 0, 32431
AppSec [candidate] (32.71 ms) : 0, 32710
Debugger [baseline] (6.387 ms) : 0, 6387
Debugger [candidate] (6.486 ms) : 0, 6486
Remote Config [baseline] (691.989 µs) : 0, 692
Remote Config [candidate] (682.855 µs) : 0, 683
Telemetry [baseline] (12.959 ms) : 0, 12959
Telemetry [candidate] (14.475 ms) : 0, 14475
Flare Poller [baseline] (7.242 ms) : 0, 7242
Flare Poller [candidate] (6.654 ms) : 0, 6654
section iast
crashtracking [baseline] (1.467 ms) : 0, 1467
crashtracking [candidate] (1.457 ms) : 0, 1457
BytebuddyAgent [baseline] (828.466 ms) : 0, 828466
BytebuddyAgent [candidate] (822.125 ms) : 0, 822125
GlobalTracer [baseline] (234.791 ms) : 0, 234791
GlobalTracer [candidate] (233.343 ms) : 0, 233343
AppSec [baseline] (30.285 ms) : 0, 30285
AppSec [candidate] (27.817 ms) : 0, 27817
Debugger [baseline] (6.221 ms) : 0, 6221
Debugger [candidate] (6.057 ms) : 0, 6057
Remote Config [baseline] (631.182 µs) : 0, 631
Remote Config [candidate] (592.462 µs) : 0, 592
Telemetry [baseline] (8.645 ms) : 0, 8645
Telemetry [candidate] (8.389 ms) : 0, 8389
Flare Poller [baseline] (4.199 ms) : 0, 4199
Flare Poller [candidate] (4.104 ms) : 0, 4104
IAST [baseline] (31.812 ms) : 0, 31812
IAST [candidate] (33.345 ms) : 0, 33345
LoadParameters
See matching parameters
SummaryFound 3 performance improvements and 1 performance regressions! Performance is the same for 8 metrics, 12 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.55.0-SNAPSHOT~3f35463cf8, baseline=1.55.0-SNAPSHOT~cdebbd8dc3
dateFormat X
axisFormat %s
section baseline
no_agent (37.468 ms) : 37166, 37770
. : milestone, 37468,
appsec (47.585 ms) : 47162, 48008
. : milestone, 47585,
code_origins (42.425 ms) : 42070, 42779
. : milestone, 42425,
iast (43.522 ms) : 43144, 43901
. : milestone, 43522,
profiling (47.124 ms) : 46650, 47597
. : milestone, 47124,
tracing (44.591 ms) : 44209, 44973
. : milestone, 44591,
section candidate
no_agent (37.796 ms) : 37494, 38097
. : milestone, 37796,
appsec (49.39 ms) : 48950, 49830
. : milestone, 49390,
code_origins (43.135 ms) : 42771, 43499
. : milestone, 43135,
iast (44.67 ms) : 44296, 45045
. : milestone, 44670,
profiling (47.756 ms) : 47293, 48219
. : milestone, 47756,
tracing (42.553 ms) : 42209, 42896
. : milestone, 42553,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.55.0-SNAPSHOT~3f35463cf8, baseline=1.55.0-SNAPSHOT~cdebbd8dc3
dateFormat X
axisFormat %s
section baseline
no_agent (4.444 ms) : 4388, 4500
. : milestone, 4444,
iast (9.541 ms) : 9374, 9708
. : milestone, 9541,
iast_FULL (14.208 ms) : 13924, 14492
. : milestone, 14208,
iast_GLOBAL (10.939 ms) : 10742, 11135
. : milestone, 10939,
profiling (9.077 ms) : 8935, 9219
. : milestone, 9077,
tracing (8.001 ms) : 7878, 8124
. : milestone, 8001,
section candidate
no_agent (4.551 ms) : 4496, 4605
. : milestone, 4551,
iast (9.402 ms) : 9243, 9561
. : milestone, 9402,
iast_FULL (13.849 ms) : 13570, 14128
. : milestone, 13849,
iast_GLOBAL (11.001 ms) : 10804, 11198
. : milestone, 11001,
profiling (8.633 ms) : 8496, 8770
. : milestone, 8633,
tracing (7.633 ms) : 7520, 7746
. : milestone, 7633,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.55.0-SNAPSHOT~3f35463cf8, baseline=1.55.0-SNAPSHOT~cdebbd8dc3
dateFormat X
axisFormat %s
section baseline
no_agent (1.487 ms) : 1476, 1499
. : milestone, 1487,
appsec (3.725 ms) : 3505, 3944
. : milestone, 3725,
iast (2.232 ms) : 2168, 2295
. : milestone, 2232,
iast_GLOBAL (2.267 ms) : 2203, 2331
. : milestone, 2267,
profiling (2.058 ms) : 2007, 2109
. : milestone, 2058,
tracing (2.054 ms) : 2004, 2104
. : milestone, 2054,
section candidate
no_agent (1.487 ms) : 1475, 1498
. : milestone, 1487,
appsec (3.768 ms) : 3548, 3987
. : milestone, 3768,
iast (2.231 ms) : 2166, 2295
. : milestone, 2231,
iast_GLOBAL (2.288 ms) : 2224, 2353
. : milestone, 2288,
profiling (2.503 ms) : 2328, 2678
. : milestone, 2503,
tracing (2.046 ms) : 1996, 2096
. : milestone, 2046,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.55.0-SNAPSHOT~3f35463cf8, baseline=1.55.0-SNAPSHOT~cdebbd8dc3
dateFormat X
axisFormat %s
section baseline
no_agent (14.844 s) : 14844000, 14844000
. : milestone, 14844000,
appsec (14.823 s) : 14823000, 14823000
. : milestone, 14823000,
iast (18.382 s) : 18382000, 18382000
. : milestone, 18382000,
iast_GLOBAL (18.396 s) : 18396000, 18396000
. : milestone, 18396000,
profiling (14.96 s) : 14960000, 14960000
. : milestone, 14960000,
tracing (15.255 s) : 15255000, 15255000
. : milestone, 15255000,
section candidate
no_agent (14.986 s) : 14986000, 14986000
. : milestone, 14986000,
appsec (14.799 s) : 14799000, 14799000
. : milestone, 14799000,
iast (18.242 s) : 18242000, 18242000
. : milestone, 18242000,
iast_GLOBAL (18.179 s) : 18179000, 18179000
. : milestone, 18179000,
profiling (15.196 s) : 15196000, 15196000
. : milestone, 15196000,
tracing (15.485 s) : 15485000, 15485000
. : milestone, 15485000,
|
bric3
approved these changes
Oct 23, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR mirrors the changes from the original community contribution to enable CI testing with maintainer privileges.
Original PR: #9757
Original Author: @deejgregor
Original Branch: deejgregor/dd-trace-java:pending-traces-multiple-dump-fix
Closes #9757
This is an automated mirror created to run CI checks. See tooling/mirror-community-pull-request.sh for details.