[beats receivers] Add ability to specify fixed port for the healthcheck extension and Prometheus metrics

[cmacknz]

elastic-agent/internal/pkg/otel/manager/execution_subprocess.go

Lines 90 to 93 in cb9c826

	httpHealthCheckPort, err := findRandomTCPPort()
	if err != nil {
	return nil, fmt.Errorf("could not find port for http health check: %w", err)
	}

We find a random TCP port for the health check extension, this is and should continue to be the default behavior.

We should allow specifying a fixed port however for cases where users may need to whitelist the new port in local iptables or firewall rules.

Probably this belongs in the agent.monitoring section alongside the same port specification used for scrape prometheus metrics from collector internal telemetry that is being introduced in. #9928

[blakerouse]

We should strive for this to really be a unix socket or a named pipe, removing the whole class of issues related to it being a local port.

Default really should be the socket or named pipe with the option with it being a fixed local port.

[cmacknz]

Agree, I remember discussing this but can't remember why it was localhost, perhaps another case of the collector not supporting it yet.

[pkoutsovasilis]

@blakerouse @cmacknz this exact point came up again in #8248 (discussion). Given the current limitation, I believe the following is a good-for-now plan:

• Keep TCP on 127.0.0.1 with random port as the default.
• [captured by this issue] Add a fixed-port override for environments that need firewall/iptables whitelisting.
  • @cmacknz what the path under agent.monitoring? I don't see any new sections there in Ingest internal telemetry from the OTel Collector when it is running #9928

Of course, if upstream adds UDS/named pipe support in the future, we can revisit and flip the implementation.

[cmacknz]

@cmacknz what the path under agent.monitoring? I don't see any new sections there

Right we haven't specified this and #10240 will have the same need. CC @swiatekm

1. We introduce a new agent.collector settings section and put the healthcheck port and the internal telemetry port. This is maybe leaking implementation details out with the naming but it's the most straight forward and gives us another place to stick collector settings if we ever need them.

agent.collector:
  healthcheck_port: 1234 # Defaults to a random free port
  telemetry_port: 4567 # Defaults to a random free port

2. We stick the ports in the existing agent.monitoring.http section but this doesn't feel like it integrates well with the existing host and port and enabled flags in there which would always be ignored.

agent.monitoring
  http:
#     # enables http endpoint
#     enabled: false
#     # The HTTP endpoint will bind to this hostname, IP address, unix socket or named pipe.
#     # When using IP addresses, it is recommended to only use localhost.
#     host: localhost
#     # Port on which the HTTP endpoint will bind. Default is 0 meaning feature is disabled.
#     port: 6791
    healthcheck_port: 1234 # Defaults to a random free port
    telemetry_port: 4567 # Defaults to a random free port

3. Some combination of the two like an agent.monitoring.collector section but if we are fine with a collector section probably better to just have one.

[cmacknz]

Option 1 feels like the choice we'd regret the least but I'm open to suggestion.

[ycombinator]

I'm definitely a no for option 2 (agent.monitoring.http) for the reasons mentioned related to the non-effect of the other settings in that section.

I'm torn between options 1 (agent.collector) and 3 (agent.monitoring.collector) but I'm leaning towards 1 as well. The agent.monitoring section feels like it's about monitoring the Agent itself, not necessarily it's components. And, as you point out, we might need other settings for the collector component in the future. So I'll vote for option 1.

[blakerouse]

I like option 1 with agent.collector. I believe we will have other things we will want to adjust and this is the best place to put those. I don't think it should be healthcheck_port or telemetry_port. I think it would be better to give that more structure to ensure that we have more configuration in the future with those options.

I would think something like this overall would be a good option and future proof the configuration:

agent.collector:
  healthcheck:
    endpoint: http://localhost:1234 <- this allows http, https, unit, npipe in the future (default is empty, which at the moment is `http://localhost:{random_port}`
  telemetry:
    endpoint: http://localhost:4567

There might be more options we want for healthcheck and telemetry, and this will open that up.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

Assigning to @swiatekm as discussed in the project meeting today.

[cmacknz]

For the config structure let's go with @blakerouse's proposal in #10206 (comment)