Implement dogstatsd, add timestamp support, fix flushing #648

nhulston · 2025-05-07T14:28:12Z

What does this PR do?

Removes hot-shots
Implements dogstatsd ourselves
Updates dogstatsd implementation to support timestamps
Therefore, if the user has extension enabled and sends a metric with a timestamp, we can now use dogstatsd instead of directly using the API
This PR also implements a working flush. The previous implementation just closed the UDP socket on "flush" which actually just drops any unsent UDP packets, leading to lost metrics.
Just use one socket, no need to keep opening/closing sockets on every invocation

Motivation

hot-shots did not support metrics with timestamps, so originally we just sent metrics directly using the Datadog API
However, the DD API is not FIPs compliant. This seemed like the easiest solution
dogstatsd is a very simple protocol so might as well do it ourselves. Now, our performance should be slightly faster and our package size should be slightly smaller!

Testing Guidelines

Tested manually with a Lambda, calling sendDistributionMetric and sendDistributionMetricWithDate. Both work!
Unit tests

Additional Notes

This should be non-breaking, but this is a big change, so we should be careful and test a lot with self-monitoring before the next release.
Will deal with FIPs compliance stuff in a separate PR after this

Types of Changes

Bug fix
New feature
Breaking change
Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

This PR's description is comprehensive
This PR contains breaking changes that are documented in the description
This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
This PR impacts documentation, and it has been updated (or a ticket has been logged)
This PR's changes are covered by the automated tests
This PR collects user input/sensitive content into Datadog
This PR passes the integration tests (ask a Datadog member to run the tests)

https://datadoghq.atlassian.net/browse/SVLS-6785

duncanista · 2025-05-07T19:50:35Z

src/metrics/dogstatsd.ts

+import { logDebug } from "../utils";
+
+export class LambdaDogStatsD {
+  private static readonly HOST = "localhost";


Maybe we set this to 127.0.0.1 instead of localhost, for safety

duncanista · 2025-05-07T19:51:58Z

src/metrics/dogstatsd.ts

+      return;
+    }
+    const serializedTags = tags && tags.length ? `|#${this.normalizeTags(tags).join(",")}` : "";
+    const timeStampPart = timestamp != null ? `|T${timestamp}` : "";


Suggested change

const timeStampPart = timestamp != null ? `|T${timestamp}` : "";

const timestampPart = timestamp != null ? `|T${timestamp}` : "";

src/metrics/listener.ts

duncanista · 2025-05-07T19:56:37Z

src/metrics/listener.ts

      }
+
+      const secondsSinceEpoch = Math.floor(metricTime.getTime() / 1000);
+      this.statsDClient?.distribution(name, value, secondsSinceEpoch, tags);


Suggested change

this.statsDClient?.distribution(name, value, secondsSinceEpoch, tags);

this.statsDClient.distribution(name, value, secondsSinceEpoch, tags);

duncanista

Left some comments – would like to see if we can do millisecond/nanosecond resolution in Extension/Forwarder, just a nit

nhulston · 2025-05-08T15:04:22Z

Left some comments – would like to see if we can do millisecond/nanosecond resolution in Extension/Forwarder, just a nit

We discussed in Slack but replying here for visibility:

The extension rounds metrics to the nearest 10 second bucket, so rounding here is fine
Also, timestamps with decimals are currently broken (this is a regression), but I opened a fix here: Fix parsing metrics with decimal timestamp serverless-components#17
In case customers in the future use old versions of the extension and new versions of datadog-lambda-js, I will make sure to round here for backwards compatibility

duncanista · 2025-05-08T15:23:42Z

src/metrics/dogstatsd.ts

+        logDebug(`Socket send buffer increased to ${LambdaDogStatsD.MIN_SEND_BUFFER_SIZE / 1024}kb`);
+      }
+    } catch {
+      // ignore


Added a debug log

nhulston · 2025-05-08T15:46:58Z

src/index.spec.ts

    expect(mockedIncrementInvocations).toBeCalledTimes(1);
    expect(mockedIncrementInvocations).toBeCalledWith(expect.anything(), mockContext);
-    expect(logger.debug).toHaveBeenCalledTimes(8);
+    expect(logger.debug).toHaveBeenCalledTimes(9);


we get another debug log for increasing buffer send size

… size... no way to set min send size in Node

nhulston added 6 commits May 7, 2025 12:45

create dogstatsd class and constructor

630019a

create report method to build dogstatsd payload

567b508

implement send UDP datagram

2dde47a

replace hot-shots with custom dogstatsd implementation

7bb3998

implement flushing strategy

9dccc5d

remove hot-shots

e1a206b

nhulston force-pushed the nicholas.hulston/implement-dogstatsd branch from ac7357b to e1a206b Compare May 7, 2025 16:57

nhulston added 8 commits May 7, 2025 12:59

lint

ed3f52f

fix metric timestamp

20da437

set max flush timeout to prevent infinite blocking

7d484ba

fix metric log using fractional timestamps

85c2c18

fix unit tests; test that we use dogstatsd with timestamp+extension

6258126

update license

79bdb43

add debug log on error

1a2a38a

unit tests

f1f3615

nhulston changed the title ~~[draft] Nicholas.hulston/implement dogstatsd~~ Implement dogstatsd, support dogstatsd with timestamps, fix flushing May 7, 2025

nhulston changed the title ~~Implement dogstatsd, support dogstatsd with timestamps, fix flushing~~ Implement dogstatsd, add timestamp support, fix flushing May 7, 2025

nhulston added 2 commits May 7, 2025 14:33

fix debug logs

e0fa27c

log when socket times out

48eb47d

nhulston force-pushed the nicholas.hulston/implement-dogstatsd branch from 9708108 to 48eb47d Compare May 7, 2025 19:00

nhulston marked this pull request as ready for review May 7, 2025 19:17

nhulston requested a review from a team as a code owner May 7, 2025 19:17

duncanista reviewed May 7, 2025

View reviewed changes

src/metrics/listener.ts Outdated Show resolved Hide resolved

duncanista reviewed May 7, 2025

View reviewed changes

nits

5ea40f6

nhulston mentioned this pull request May 8, 2025

Fix parsing metrics with decimal timestamp DataDog/serverless-components#17

Merged

round in dogstatsd; unit test

11fe06e

nhulston requested a review from duncanista May 8, 2025 15:21

duncanista reviewed May 8, 2025

View reviewed changes

duncanista approved these changes May 8, 2025

View reviewed changes

bind to local port before setting min send buffer size; add debug

b77211a

nhulston commented May 8, 2025

View reviewed changes

nhulston added 3 commits May 8, 2025 11:47

lint

fc08ed8

fix

fdb0bee

remove setSendBufferSize, since it sets MAX send size, not MIN send…

b7967e6

… size... no way to set min send size in Node

nhulston force-pushed the nicholas.hulston/implement-dogstatsd branch from 537d86d to b7967e6 Compare May 8, 2025 19:36

nhulston merged commit 45d33ad into main May 8, 2025
25 checks passed

nhulston deleted the nicholas.hulston/implement-dogstatsd branch May 8, 2025 19:46

nhulston mentioned this pull request May 9, 2025

[breaking] FIPS compliant metrics + secrets management #649

Merged

11 tasks

nhulston mentioned this pull request Jun 9, 2025

(fix): Fix Typescript Timeouts when Lambda handler returns undefined #661

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement dogstatsd, add timestamp support, fix flushing #648

Implement dogstatsd, add timestamp support, fix flushing #648

Uh oh!

nhulston commented May 7, 2025 •

edited

Loading

Uh oh!

duncanista May 7, 2025

Uh oh!

duncanista May 7, 2025

Uh oh!

Uh oh!

duncanista May 7, 2025

Uh oh!

duncanista left a comment

Uh oh!

nhulston commented May 8, 2025

Uh oh!

duncanista May 8, 2025

Uh oh!

nhulston May 8, 2025

Uh oh!

nhulston May 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	const timeStampPart = timestamp != null ? `\|T${timestamp}` : "";
	const timestampPart = timestamp != null ? `\|T${timestamp}` : "";

	this.statsDClient?.distribution(name, value, secondsSinceEpoch, tags);
	this.statsDClient.distribution(name, value, secondsSinceEpoch, tags);

Implement dogstatsd, add timestamp support, fix flushing #648

Implement dogstatsd, add timestamp support, fix flushing #648

Uh oh!

Conversation

nhulston commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Testing Guidelines

Additional Notes

Types of Changes

Check all that apply

Uh oh!

duncanista May 7, 2025

Choose a reason for hiding this comment

Uh oh!

duncanista May 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

duncanista May 7, 2025

Choose a reason for hiding this comment

Uh oh!

duncanista left a comment

Choose a reason for hiding this comment

Uh oh!

nhulston commented May 8, 2025

Uh oh!

duncanista May 8, 2025

Choose a reason for hiding this comment

Uh oh!

nhulston May 8, 2025

Choose a reason for hiding this comment

Uh oh!

nhulston May 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nhulston commented May 7, 2025 •

edited

Loading