Use a random port for otel collector monitoring endpoint #10240

swiatekm · 2025-10-01T17:08:23Z

What does this PR do?

Uses a random port for the otel collector's Prometheus endpoint. Right now this is hardcoded at the default.

Normally, this port needs to be known during config generation (so we can add the right self-monitoring configuration), so it needs to be determined before the coordinator starts. In this PR, we instead put it into an environment variable and use the otel collector's ability to load configuration values from the variable. As a result, we can defer determining the actual port as late as possible, and even use a different port on each configuration reload, allowing us to recover from port binding conflicts.

This becomes a bit awkward with the embedded collector, where we need to call SetEnv, always an anti-pattern. But we eventually expect to retire that mode of execution, and it's not the default anymore, so it should be fine.

In the process, I'm also allowing both the metrics port and the healthcheck port to be passed into the otel manager as parameters. This is preparation for making them configurable in a follow-up.

Why is it important?

The port shouldn't be hardcoded. In the event of a conflict, the otel collector can't start.

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
~~[ ] I have added an entry in ./changelog/fragments using the changelog tool~~
~~[ ] I have added an integration test or an E2E test~~

How to test this PR locally

Build the agent locally, run it with otel self-monitoring, generate diagnostics, then check the configuration.

Related issues

Closes https://github.com/elastic/ingest-dev/issues/6175

mergify · 2025-10-01T17:09:02Z

This pull request does not have a backport label. Could you fix it @swiatekm? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

internal/pkg/agent/application/monitoring/component/v1_monitor.go

elasticmachine · 2025-10-02T08:35:25Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

internal/pkg/agent/application/application.go

leehinman · 2025-10-02T13:00:27Z

I'm probably paranoid from the npipe permission issue, but do we have a test that starts and stops the collector multiple times in rapid succession? Since we are re-using the same port number it might be worth double checking.

swiatekm · 2025-10-02T15:56:05Z

I'm going to try an approach using an environment variable, so I can inject this much later in the pipeline. Will undraft once I know if that's feasible.

internal/pkg/otel/manager/execution_embedded.go

internal/pkg/otel/manager/execution_subprocess.go

ycombinator

LGTM pending resolution of #10240 (comment)

blakerouse

Looks good, glad the environment variable substitution worked out.

internal/pkg/otel/manager/execution_subprocess.go

Co-authored-by: Blake Rouse <[email protected]>

elasticmachine · 2025-10-07T18:41:45Z

💚 Build Succeeded

Buildkite Build
Commit: d7c48eb

History

💛 Build #28154 was flaky ddc00a8
💛 Build #28085 was flaky cf6f6bd
💔 Build #28061 failed cf6f6bd
💔 Build #27893 failed 30c23f4

cc @swiatekm

* Use a random port for otel collector monitoring endpoint * mage notice * Use an env variable * Fix linter warnings * Fix random port determination for the embedded otel collector * Drop the ports functions from the utils package * fixup! Fix random port determination for the embedded otel collector * Ensure no port conflicts * Clean up port assignment * Verify that returned ports are unique * Add port conflict test * Fix docstring typo * More comments * Add comments explaining the port conflict test * Update internal/pkg/otel/manager/execution_subprocess.go Co-authored-by: Blake Rouse <[email protected]> --------- Co-authored-by: Blake Rouse <[email protected]> (cherry picked from commit 5cb8c31)

…0390) * Use a random port for otel collector monitoring endpoint * mage notice * Use an env variable * Fix linter warnings * Fix random port determination for the embedded otel collector * Drop the ports functions from the utils package * fixup! Fix random port determination for the embedded otel collector * Ensure no port conflicts * Clean up port assignment * Verify that returned ports are unique * Add port conflict test * Fix docstring typo * More comments * Add comments explaining the port conflict test * Update internal/pkg/otel/manager/execution_subprocess.go --------- (cherry picked from commit 5cb8c31) Co-authored-by: Mikołaj Świątek <[email protected]> Co-authored-by: Blake Rouse <[email protected]>

ebeahan · 2025-10-13T16:56:20Z

@swiatekm there's a question about backporting this to 8.19: #10449 (comment). Please double check what branches this change should land in.

* Use a random port for otel collector monitoring endpoint * mage notice * Use an env variable * Fix linter warnings * Fix random port determination for the embedded otel collector * Drop the ports functions from the utils package * fixup! Fix random port determination for the embedded otel collector * Ensure no port conflicts * Clean up port assignment * Verify that returned ports are unique * Add port conflict test * Fix docstring typo * More comments * Add comments explaining the port conflict test * Update internal/pkg/otel/manager/execution_subprocess.go Co-authored-by: Blake Rouse <[email protected]> --------- Co-authored-by: Blake Rouse <[email protected]> (cherry picked from commit 5cb8c31) # Conflicts: # NOTICE-fips.txt # NOTICE.txt # go.mod # internal/pkg/otel/manager/execution_embedded.go # internal/pkg/otel/manager/manager.go # internal/pkg/otel/manager/manager_test.go

…ing endpoint (#10520) * Use a random port for otel collector monitoring endpoint (#10240) * Use a random port for otel collector monitoring endpoint * mage notice * Use an env variable * Fix linter warnings * Fix random port determination for the embedded otel collector * Drop the ports functions from the utils package * fixup! Fix random port determination for the embedded otel collector * Ensure no port conflicts * Clean up port assignment * Verify that returned ports are unique * Add port conflict test * Fix docstring typo * More comments * Add comments explaining the port conflict test * Update internal/pkg/otel/manager/execution_subprocess.go Co-authored-by: Blake Rouse <[email protected]> --------- Co-authored-by: Blake Rouse <[email protected]> (cherry picked from commit 5cb8c31) # Conflicts: # NOTICE-fips.txt # NOTICE.txt # go.mod # internal/pkg/otel/manager/execution_embedded.go # internal/pkg/otel/manager/manager.go # internal/pkg/otel/manager/manager_test.go * Fix conflicts --------- Co-authored-by: Mikołaj Świątek <[email protected]>

Use a random port for otel collector monitoring endpoint

30c23f4

mergify bot assigned swiatekm Oct 1, 2025

swiatekm added skip-changelog bug Something isn't working labels Oct 1, 2025

mage notice

b1a7991

cmacknz reviewed Oct 1, 2025

View reviewed changes

internal/pkg/agent/application/monitoring/component/v1_monitor.go Outdated Show resolved Hide resolved

cmacknz mentioned this pull request Oct 1, 2025

[beats receivers] Add ability to specify fixed port for the healthcheck extension and Prometheus metrics #10206

Closed

cmacknz added the backport-9.2 Automated backport to the 9.2 branch label Oct 1, 2025

swiatekm added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Oct 2, 2025

swiatekm marked this pull request as ready for review October 2, 2025 08:35

swiatekm requested a review from a team as a code owner October 2, 2025 08:35

swiatekm requested review from kaanyalti and ycombinator October 2, 2025 08:35

swiatekm requested review from faec and pkoutsovasilis October 2, 2025 08:38

pkoutsovasilis reviewed Oct 2, 2025

View reviewed changes

internal/pkg/agent/application/application.go Outdated Show resolved Hide resolved

swiatekm requested a review from cmacknz October 2, 2025 12:31

swiatekm marked this pull request as draft October 2, 2025 14:56

swiatekm added 5 commits October 3, 2025 19:26

Use an env variable

55084df

Fix linter warnings

a942e44

Fix random port determination for the embedded otel collector

1a39861

Drop the ports functions from the utils package

50b7e37

fixup! Fix random port determination for the embedded otel collector

cf6f6bd

swiatekm marked this pull request as ready for review October 6, 2025 10:15

swiatekm requested a review from pkoutsovasilis October 6, 2025 10:16

Ensure no port conflicts

04edae0

ycombinator reviewed Oct 6, 2025

View reviewed changes

internal/pkg/otel/manager/execution_embedded.go Show resolved Hide resolved

ycombinator reviewed Oct 6, 2025

View reviewed changes

internal/pkg/otel/manager/execution_subprocess.go Show resolved Hide resolved

swiatekm added 3 commits October 6, 2025 20:19

Verify that returned ports are unique

ac0c435

Add port conflict test

c541af7

Fix docstring typo

00477ac

swiatekm requested a review from cmacknz October 6, 2025 18:23

swiatekm added 2 commits October 6, 2025 20:35

More comments

f53b8e8

Add comments explaining the port conflict test

ddc00a8

swiatekm requested a review from ycombinator October 7, 2025 11:37

ycombinator previously approved these changes Oct 7, 2025

View reviewed changes

blakerouse previously approved these changes Oct 7, 2025

View reviewed changes

internal/pkg/otel/manager/execution_subprocess.go Outdated Show resolved Hide resolved

pkoutsovasilis previously approved these changes Oct 7, 2025

View reviewed changes

Update internal/pkg/otel/manager/execution_subprocess.go

dab13dd

Co-authored-by: Blake Rouse <[email protected]>

swiatekm dismissed stale reviews from pkoutsovasilis, blakerouse, and ycombinator via dab13dd October 7, 2025 16:33

Merge branch 'main' into fix/collector-random-tcp-port

d7c48eb

cmacknz approved these changes Oct 7, 2025

View reviewed changes

swiatekm merged commit 5cb8c31 into main Oct 7, 2025
21 checks passed

swiatekm deleted the fix/collector-random-tcp-port branch October 7, 2025 19:10

mergify bot mentioned this pull request Oct 7, 2025

[9.2] (backport #10240) Use a random port for otel collector monitoring endpoint #10390

Merged

4 tasks

pkoutsovasilis mentioned this pull request Oct 10, 2025

[8.19] (backport #10412) fix: do not report agent managed otel extensions statuses #10449

Merged

8 tasks

swiatekm added the backport-8.19 Automated backport to the 8.19 branch label Oct 13, 2025

mergify bot mentioned this pull request Oct 13, 2025

[8.19] (backport #10240) Use a random port for otel collector monitoring endpoint #10520

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use a random port for otel collector monitoring endpoint #10240

Use a random port for otel collector monitoring endpoint #10240

Uh oh!

swiatekm commented Oct 1, 2025 •

edited

Loading

Uh oh!

mergify bot commented Oct 1, 2025

Uh oh!

Uh oh!

elasticmachine commented Oct 2, 2025

Uh oh!

Uh oh!

leehinman commented Oct 2, 2025

Uh oh!

swiatekm commented Oct 2, 2025

Uh oh!

Uh oh!

Uh oh!

ycombinator left a comment

Uh oh!

blakerouse left a comment

Uh oh!

Uh oh!

elasticmachine commented Oct 7, 2025

Uh oh!

Uh oh!

ebeahan commented Oct 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Use a random port for otel collector monitoring endpoint #10240

Use a random port for otel collector monitoring endpoint #10240

Uh oh!

Conversation

swiatekm commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

Uh oh!

mergify bot commented Oct 1, 2025

Uh oh!

Uh oh!

elasticmachine commented Oct 2, 2025

Uh oh!

Uh oh!

leehinman commented Oct 2, 2025

Uh oh!

swiatekm commented Oct 2, 2025

Uh oh!

Uh oh!

Uh oh!

ycombinator left a comment

Choose a reason for hiding this comment

Uh oh!

blakerouse left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticmachine commented Oct 7, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

ebeahan commented Oct 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

swiatekm commented Oct 1, 2025 •

edited

Loading