Skip to content

Conversation

@VihasMakwana
Copy link
Contributor

@VihasMakwana VihasMakwana commented Oct 23, 2025

Proposed commit message

We're injecting default processors while running in standalone beatreceivers. This is not the case in standalone beats.
We should inject only if we're running under agent (Fixed via #46539).

Benchmarks shows improvement and are close to normal mode. There is still some difference and I'll work on that as a follow-up.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool.

How to test this PR locally

You can test the build from this PR with benchbuilder and compare the results without the changes of this PR.

Related issues

Results

  • Benchmark for standalone filebeat - normal mode.
### Benchmark summary table
| Environment ID | Race Id | Suite Name | Benchmark Name | Benchmark Type | Version | Duration (sec) | EPS | Store Bytes | Bytes/Event | Error |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| xyz | 8dd8bf0e-ef1b-4767-8c29-ad986afea6dd | logs-datastream | filestream-json-1024b-balanced | filebeat | `9.3.0` | `1m1.196885s` | `3268.000000` | `56872890` | `284.3644` |  |
| xyz | 43af615c-4b18-49b9-8ca2-4925c44644a3 | logs-datastream | filestream-json-1024b-throughput | filebeat | `9.3.0` | `45.795869s` | `4367.000000` | `56104542` | `280.5227` |  |
| xyz | d76fedb6-5352-4d62-969e-a7bc1aa14d3e | logs-datastream | filestream-json-1024b-scale | filebeat | `9.3.0` | `1m5.14271s` | `3070.000000` | `56834671` | `284.1734` |  |
| xyz | ae0a33c9-5032-47e3-a1e2-ddf0a8f3696d | logs-datastream | filestream-json-1024b-latency | filebeat | `9.3.0` | `19m12.550807s` | `174.000000` | `58669647` | `293.3482` |  |

_[Open in Kibana](https://ingest-metricstore.kb.us-central1.gcp.cloud.es.io:9243/app/dashboards#/view/ce699545-7f3f-42ce-b0f9-16c51e8b01ef?_g=%28time%3A%28from%3A%272025-10-25T14%3A27%3A07.052Z%27%2Cto%3A%272025-10-25T14%3A50%3A30.908Z%27%29%29&_a=%28filters%3A%21%28%28%27%24state%27%3A%28store%3AappState%29%2Cmeta%3A%28alias%3A%21n%2Cdisabled%3A%21f%2Cfield%3Aenvironment_id%2Cindex%3A%27e80ebe49-6c56-452d-a932-2e9026f9ff65%27%2Ckey%3Aenvironment_id%2Cnegate%3A%21f%2Cparams%3A%28query%3A%27xyz%27%29%2Ctype%3Aphrase%29%2Cquery%3A%28match_phrase%3A%28environment_id%3A%27xyz%27%29%29%29%29%2Cquery%3A%28language%3Akuery%2Cquery%3A%27%27%29%29)_
  • Benchmark for filebeat otel mode (before my changes)
### Benchmark summary table
| Environment ID | Race Id | Suite Name | Benchmark Name | Benchmark Type | Version | Duration (sec) | EPS | Store Bytes | Bytes/Event | Error |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| xyz | 755ead19-ab37-4de9-a54c-9395ec17c43a | logs-datastream | filestream-json-1024b-balanced | filebeat | `9.3.0` | `1m56.138989s` | `1722.000000` | `105533417` | `527.6671` |  |
| xyz | 7a7e1caf-7a92-4396-b0c8-6050d6d40ca0 | logs-datastream | filestream-json-1024b-throughput | filebeat | `9.3.0` | `37.276624s` | `5365.000000` | `107132772` | `535.6639` |  |
| xyz | 467e93ba-5d28-4c1c-99f8-d7e83b84f100 | logs-datastream | filestream-json-1024b-scale | filebeat | `9.3.0` | `1m57.625477s` | `1700.000000` | `104303606` | `521.5180` |  |
| xyz | 16d50a55-4e64-4a30-b84c-64561fa7fb1e | logs-datastream | filestream-json-1024b-latency | filebeat | `9.3.0` | `20m8.961231s` | `165.000000` | `108167271` | `540.8364` |  |

_[Open in Kibana](https://ingest-metricstore.kb.us-central1.gcp.cloud.es.io:9243/app/dashboards#/view/ce699545-7f3f-42ce-b0f9-16c51e8b01ef?_g=%28time%3A%28from%3A%272025-10-25T20%3A12%3A12.283Z%27%2Cto%3A%272025-10-25T20%3A38%3A14.524Z%27%29%29&_a=%28filters%3A%21%28%28%27%24state%27%3A%28store%3AappState%29%2Cmeta%3A%28alias%3A%21n%2Cdisabled%3A%21f%2Cfield%3Aenvironment_id%2Cindex%3A%27e80ebe49-6c56-452d-a932-2e9026f9ff65%27%2Ckey%3Aenvironment_id%2Cnegate%3A%21f%2Cparams%3A%28query%3A%27xyz%27%29%2Ctype%3Aphrase%29%2Cquery%3A%28match_phrase%3A%28environment_id%3A%27xyz%27%29%29%29%29%2Cquery%3A%28language%3Akuery%2Cquery%3A%27%27%29%29)_
  • Benchmark for filebeat otel mode (after my changes)
### Benchmark summary table
| Environment ID | Race Id | Suite Name | Benchmark Name | Benchmark Type | Version | Duration (sec) | EPS | Store Bytes | Bytes/Event | Error |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| xyz | 4eefb649-3ae4-498a-b46d-ad6b77f411b0 | logs-datastream | filestream-json-1024b-balanced | filebeat | `9.3.0` | `1m12.23267s` | `2769.000000` | `66011174` | `330.0559` |  |
| xyz | bfd54408-1075-405b-83f5-d20f06181a0e | logs-datastream | filestream-json-1024b-throughput | filebeat | `9.3.0` | `21.711077s` | `9212.000000` | `68151536` | `340.7577` |  |
| xyz | d9bab3f9-0628-480a-aee2-8e6107219e45 | logs-datastream | filestream-json-1024b-scale | filebeat | `9.3.0` | `1m12.787666s` | `2748.000000` | `65854534` | `329.2727` |  |
| xyz | 97cf4796-c3c6-400b-a5f5-e34d4a1f639e | logs-datastream | filestream-json-1024b-latency | filebeat | `9.3.0` | `19m17.321353s` | `173.000000` | `66767287` | `333.8364` |  |

_[Open in Kibana](https://ingest-metricstore.kb.us-central1.gcp.cloud.es.io:9243/app/dashboards#/view/ce699545-7f3f-42ce-b0f9-16c51e8b01ef?_g=%28time%3A%28from%3A%272025-10-25T19%3A46%3A19.037Z%27%2Cto%3A%272025-10-25T20%3A09%3A43.914Z%27%29%29&_a=%28filters%3A%21%28%28%27%24state%27%3A%28store%3AappState%29%2Cmeta%3A%28alias%3A%21n%2Cdisabled%3A%21f%2Cfield%3Aenvironment_id%2Cindex%3A%27e80ebe49-6c56-452d-a932-2e9026f9ff65%27%2Ckey%3Aenvironment_id%2Cnegate%3A%21f%2Cparams%3A%28query%3A%27xyz%27%29%2Ctype%3Aphrase%29%2Cquery%3A%28match_phrase%3A%28environment_id%3A%27xyz%27%29%29%29%29%2Cquery%3A%28language%3Akuery%2Cquery%3A%27%27%29%29)_

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 23, 2025
@github-actions
Copy link
Contributor

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@mergify
Copy link
Contributor

mergify bot commented Oct 23, 2025

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @VihasMakwana? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
  • backport-active-all is the label that automatically backports to all active branches.
  • backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
  • backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

@VihasMakwana VihasMakwana added backport-8.19 Automated backport to the 8.19 branch backport-9.1 Automated backport to the 9.1 branch backport-9.2 Automated backport to the 9.2 branch labels Oct 23, 2025
@khushijain21
Copy link
Contributor

khushijain21 commented Oct 23, 2025

This is going to break a lot of tests as we expect docuemnts indexed by standalone beats and beatreceivers to be same. And this PR changes the default processors to be available under agent only.

Also for managemt.Enabled() to return true, the beats process send "management.enabled=true" flag. But beatreceivers do not set this arg and hence will always result in false.

Have you tested this change with elastic-agent otel mode?

@VihasMakwana VihasMakwana force-pushed the inject-processor-underagent branch from 5d903cd to 9d73c1c Compare October 25, 2025 14:56
@VihasMakwana VihasMakwana marked this pull request as ready for review October 27, 2025 07:04
@VihasMakwana VihasMakwana requested a review from a team as a code owner October 27, 2025 07:04
@VihasMakwana
Copy link
Contributor Author

This is going to break a lot of tests as we expect docuemnts indexed by standalone beats and beatreceivers to be same. And this PR changes the default processors to be available under agent only.

Also for managemt.Enabled() to return true, the beats process send "management.enabled=true" flag. But beatreceivers do not set this arg and hence will always result in false.

For otel, we specify "management.otel.enabled=true" and that turns managemt.Enabled() flag. I have updated the tests. PTAL.

@VihasMakwana VihasMakwana requested review from khushijain21, leehinman and mauri870 and removed request for andrzej-stencel October 27, 2025 07:05
@VihasMakwana VihasMakwana added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Oct 27, 2025
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 27, 2025
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@cmacknz
Copy link
Member

cmacknz commented Oct 27, 2025

Before merging this, open a PR in https://github.com/elastic/elastic-agent with the beats dependency updated to this branch and make sure there are no test failures introduced there.

@VihasMakwana VihasMakwana force-pushed the inject-processor-underagent branch from 93eee08 to a589164 Compare October 30, 2025 09:10
@VihasMakwana VihasMakwana force-pushed the inject-processor-underagent branch from a85b0d5 to f337a5a Compare October 30, 2025 10:57
@VihasMakwana VihasMakwana merged commit 78f115e into elastic:main Oct 31, 2025
209 of 210 checks passed
mergify bot pushed a commit that referenced this pull request Oct 31, 2025
…7299)

* chore: only add default processors if running unde ragent

* fix test

* fix test

* tab

* fix race condition

* iterate over keys instead of map

* fix test case

(cherry picked from commit 78f115e)

# Conflicts:
#	x-pack/metricbeat/tests/integration/otel_test.go
mergify bot pushed a commit that referenced this pull request Oct 31, 2025
…7299)

* chore: only add default processors if running unde ragent

* fix test

* fix test

* tab

* fix race condition

* iterate over keys instead of map

* fix test case

(cherry picked from commit 78f115e)
mergify bot pushed a commit that referenced this pull request Oct 31, 2025
…7299)

* chore: only add default processors if running unde ragent

* fix test

* fix test

* tab

* fix race condition

* iterate over keys instead of map

* fix test case

(cherry picked from commit 78f115e)

# Conflicts:
#	x-pack/filebeat/tests/integration/otel_test.go
#	x-pack/metricbeat/tests/integration/otel_test.go
VihasMakwana added a commit that referenced this pull request Oct 31, 2025
…7299) (#47414)

* chore: only add default processors if running unde ragent

* fix test

* fix test

* tab

* fix race condition

* iterate over keys instead of map

* fix test case

(cherry picked from commit 78f115e)

Co-authored-by: Vihas Makwana <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-8.19 Automated backport to the 8.19 branch backport-9.1 Automated backport to the 9.1 branch backport-9.2 Automated backport to the 9.2 branch skip-changelog Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants