[8.19] (backport #10240) Use a random port for otel collector monitoring endpoint

[mergify]

What does this PR do?

Uses a random port for the otel collector's Prometheus endpoint. Right now this is hardcoded at the default.

Normally, this port needs to be known during config generation (so we can add the right self-monitoring configuration), so it needs to be determined before the coordinator starts. In this PR, we instead put it into an environment variable and use the otel collector's ability to load configuration values from the variable. As a result, we can defer determining the actual port as late as possible, and even use a different port on each configuration reload, allowing us to recover from port binding conflicts.

This becomes a bit awkward with the embedded collector, where we need to call SetEnv, always an anti-pattern. But we eventually expect to retire that mode of execution, and it's not the default anymore, so it should be fine.

In the process, I'm also allowing both the metrics port and the healthcheck port to be passed into the otel manager as parameters. This is preparation for making them configurable in a follow-up.

Why is it important?

The port shouldn't be hardcoded. In the event of a conflict, the otel collector can't start.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

How to test this PR locally

Build the agent locally, run it with otel self-monitoring, generate diagnostics, then check the configuration.

Related issues

• Closes https://github.com/elastic/ingest-dev/issues/6175

---

This is an automatic backport of pull request #10240 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 5cb8c31 has failed:

On branch mergify/bp/8.19/pr-10240
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 5cb8c31da.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   internal/pkg/agent/application/application.go
	modified:   internal/pkg/agent/application/monitoring/component/v1_monitor.go
	modified:   internal/pkg/agent/application/monitoring/component/v1_monitor_test.go
	modified:   internal/pkg/agent/cmd/inspect.go
	modified:   internal/pkg/otel/manager/common.go
	modified:   internal/pkg/otel/manager/common_test.go
	modified:   internal/pkg/otel/manager/execution_subprocess.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   NOTICE-fips.txt
	both modified:   NOTICE.txt
	both modified:   go.mod
	both modified:   internal/pkg/otel/manager/execution_embedded.go
	both modified:   internal/pkg/otel/manager/manager.go
	both modified:   internal/pkg/otel/manager/manager_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

⏳ Build in-progress, with failures

• Buildkite Build
• Commit: 775a9ad

Failed CI Steps

• Unit tests - fips140=only Ubuntu 22.04
• Unit tests - fips140=only Ubuntu 22.04

History

• 💔 Build #28573 failed ef1e82b

cc @swiatekm

[pierrehilbert]

Unrelated failure, I'm forcing the merge.