Skip to content

Conversation

elohmeier
Copy link
Contributor

Summary

Fix HTTP 400 errors when Prometheus sends conflicting metric metadata by aligning Vector behavior with Prometheus/Thanos (silently ignore conflicts, continue processing timeseries data).

Vector configuration

[sources.prometheus_remote_write]
type = "prometheus_remote_write"
address = "0.0.0.0:8001"

How did you test this PR?

I've collected HTTP payloads from a Prometheus server configured to remote write to the Vector source via mitmproxy and replayed them to test.

Prometheus sent metadata payloads with e.g. this (protobuf/snappy encoded) v1 remote write Payload:

metadata:
- type: GAUGE
  metricFamilyName: go_memstats_alloc_bytes
  help: Number of bytes allocated and still in use.
- type: COUNTER
  metricFamilyName: go_memstats_alloc_bytes
  help: Total number of bytes allocated, even if freed.
- type: GAUGE
  metricFamilyName: go_memstats_alloc_bytes
  help: Number of bytes allocated in heap and currently in use. Equals to /memory/classes/heap/objects:bytes.

Which would lead to this error in Vector without our fix:

2025-09-13T08:33:22.096665Z  WARN source{component_kind="source" component_id=remote_write component_type=prometheus_remote_write}:http-request{method=POST path=/}: vector::internal_events::http: Received bad request. error=Could not decode write request: multiple metric kinds given for metric name `go_memstats_alloc_bytes` error_code=http_response_400 error_type="request_failed" error_stage="receiving" http_code=400 internal_log_rate_limit=true
{"error":"Could not decode write request: multiple metric kinds given for metric name `go_memstats_alloc_bytes`","error_code":"http_response_400","error_stage":"receiving","error_type":"request_failed","host":"mac","http_code":"400","internal_log_rate_limit":true,"message":"Received bad request.","metadata":{"kind":"event","level":"WARN","module_path":"vector::internal_events::http","target":"vector::internal_events::http"},"pid":87279,"source_type":"internal_logs","timestamp":"2025-09-13T08:33:22.096638Z","vector":{"component_id":"remote_write","component_kind":"source","component_type":"prometheus_remote_write"}}

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

Notes

  • Please read our Vector contributor resources.
  • Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
  • Some CI checks run only after we manually approve them.
    • We recommend adding a pre-push hook, please see this template.
    • Alternatively, we recommend running the following locally before pushing to the remote branch:
      • make fmt
      • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
      • make test
  • After a review is requested, please avoid force pushes to help us review incrementally.
    • Feel free to push as many commits as you want. They will be squashed into one before merging.
    • For example, you can run git merge origin master and git push.
  • If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
    run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

@elohmeier elohmeier requested a review from a team as a code owner September 13, 2025 08:37
@github-actions github-actions bot added the domain: sources Anything related to the Vector's sources label Sep 13, 2025
@elohmeier elohmeier force-pushed the ignore-prom-meta-conflicts branch from 23f098e to 7834aa8 Compare September 14, 2025 14:02
@elohmeier
Copy link
Contributor Author

Added missing authors to changelog file.

…ead of returning HTTP 400

Previously, Vector would return HTTP 400 errors when receiving
Prometheus remote write requests with conflicting metric metadata (e.g.,
same metric family declared as both COUNTER and GAUGE). This behavior
differs from Prometheus and Thanos, which silently ignore metadata
conflicts.

This change modifies the prometheus-parser to use a "first-wins"
strategy for conflicting metadata.

cargo fmt

add metadata_conflics option
@elohmeier elohmeier force-pushed the ignore-prom-meta-conflicts branch from 61f0d2b to 243ad9e Compare September 17, 2025 02:34
@elohmeier
Copy link
Contributor Author

Rebased onto latest master and resolved the conflicts.

@elohmeier elohmeier requested a review from pront September 17, 2025 14:57
@elohmeier
Copy link
Contributor Author

@pront - is there anything I can do to get this over the finish line? This is currently affecting a larger client unfortunately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: sources Anything related to the Vector's sources
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants