Skip to content

Conversation

@ZStriker19
Copy link
Collaborator

@ZStriker19 ZStriker19 commented Apr 21, 2025

What does this PR do?

This PR adds the DatadogSampler, which has the following cap

Add DatadogSampler for intelligent trace sampling in dd-trace-rs

Summary

This PR adds a comprehensive sampling implementation for the Datadog tracer in Rust. The DatadogSampler provides probabilistic and rule-based sampling capabilities to selectively record traces based on configurable criteria.

Features

  • Rule-based sampling: Define sampling rules based on service name, operation name, resource name, and custom tags
  • Probabilistic sampling: Apply different sampling rates to different services and operations
  • Service rate sampling: Support for dynamic sampling rates based on service/environment combinations
  • Rate limiting: Prevent excessive trace generation with configurable rate limits
  • Priority sampling: Full support for Datadog's sampling priority mechanism
  • Glob pattern matching: Use wildcards in rule definitions for flexible matching
  • Proper context propagation: Consistent sampling decisions throughout trace hierarchies
  • JSON configuration: Configure the sampler via structured JSON input
  • Initialize the sampler with configuration and pass it to the tracer
  • Checks span attributes according to Datadog semantic conversions (will refactor these constants out in next PR)

Implementation details

The implementation consists of several cooperating components:

  • DatadogSampler: The main sampler implementing OpenTelemetry's ShouldSample trait
  • SamplingRule: Configurable rules for matching spans and applying sampling decisions
  • RateSampler: Deterministic sampling based on trace ID hash values
  • RateLimiter: Thread-safe limiting of sampling throughput
  • GlobMatcher: Fast pattern matching with wildcard support

The sampler is fully integrated with the Datadog OpenTelemetry tracer, which can be initialized with:

let sampler = DatadogSampler::new(None, None);
let tracer_provider = tracer_provider_builder
    .with_sampler(sampler)
    .build();

Testing

The implementation includes a comprehensive test suite covering all aspects of the sampler's behavior, including rule matching, sampling decisions, tag generation, and configuration loading.

@ZStriker19 ZStriker19 changed the title feat(sampling): Datadog sampler feat(sampling): datadog sampler Apr 22, 2025
@ZStriker19 ZStriker19 force-pushed the zachg/datadog_sampler branch from 2680189 to 7b61b31 Compare April 22, 2025 00:23
@ZStriker19 ZStriker19 marked this pull request as ready for review April 22, 2025 17:46
@ZStriker19 ZStriker19 requested a review from a team as a code owner April 22, 2025 17:46
Copy link
Collaborator

@bantonsson bantonsson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm only halfway through this, but here are some comments.

Copy link
Collaborator

@bantonsson bantonsson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good AFAICT. Only nitpicks and performance questions.

@ZStriker19 ZStriker19 force-pushed the zachg/datadog_sampler branch from 6c296a9 to 361bb78 Compare May 8, 2025 21:43

/// Where this rule comes from (customer, dynamic, default)
#[serde(default = "default_provenance")]
pub provenance: String,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be an enumeration if the values this can take is bounded?
Also where does this come from, can customer provide this?

This PR refactors the sampler a bit in order to store thre sampling decision in the TraceRegistry.
The `ShouldSample` implementation is know in the datadog-opentelemetry crate, and the DatadogSampler uses simpler code.

To test that this works, this PR adds integration tests with the test agent that create a new span, and send it. The snapshots should contain the expected sampling tags, and we check that the tracecontext contains expected keys.
@iunanua
Copy link
Collaborator

iunanua commented Jun 9, 2025

Handling agent response is missing and Sampler configuration can be modified by agent (see TraceExporterWorker::handle_agent_reponse)

Is it planned for another PR?

@paullegranddc
Copy link
Collaborator

Is it planned for another PR?

Yes, the changes on this one are already big enough 😄

@paullegranddc paullegranddc merged commit 27d794f into main Jun 10, 2025
20 checks passed
@paullegranddc paullegranddc deleted the zachg/datadog_sampler branch June 10, 2025 13:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants