feat: utilise continue_on_err in beatsauthextension

[pkoutsovasilis]

What does this PR do?

This PR improves error handling for Elasticsearch output configurations in the Hybrid Elastic Agent by:

1. Moving partially configuration translation ownership: Relocates some of the Elasticsearch output translation logic from the beats library (libbeat/otelbeat/oteltranslate/outputs/elasticsearch) into the elastic-agent package (internal/pkg/otel/translate/output_elasticsearch.go). In the future we should do a full transition to elastic-agent repo as this gives elastic-agent full control over the translation.

2. Enabling graceful error handling: Adds continue_on_error: true to the beatsauth extension configuration in getBeatsAuthExtensionConfig(). This prevents the OpenTelemetry collector from exiting on startup when encountering invalid SSL configurations (e.g., missing certificate files) respective PR.

Why is it important?

When an Elasticsearch output has invalid configuration (like a missing SSL certificate), the collector exits with a vague error message that doesn't identify which output caused the failure:

error found during service initialization: failed to build extensions: failed to create extension "beatsauth": failed unpacking config: open /etc/client/cert.pem: no such file or directory

Benefits of this PR:

• Collector stays running instead of exiting at startup, thus allowing other pipelines that utilise different exporters with valid config to continue push data
• Errors are surfaced at the exporter level when requests are made, making it clear which output failed

Screenshot shows the intended behavior: collector continues running and errors are properly surfaced at the exporter level.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

No disruptive user impact expected.

How to test this PR locally

build and install elastic-agent from this branch with the following configuration

  id: agent-pernode-debug
  outputs:
    default:
      hosts:
      - ${ES_HOST}
      password: ${ES_PASSWORD}
      type: elasticsearch
      username: ${ES_USERNAME}
      # invalid ssl settings
      ssl.certificate: /etc/client/cert.pem
      ssl.enabled: true
      ssl.key: /etc/client/cert.key
      ssl.key_passphrase: null
      ssl.key_passphrase_path: null
      ssl.verification_mode: none
    system:
      hosts:
      - ${ES_HOST}
      password: ${ES_PASSWORD}
      type: elasticsearch
      username: ${ES_USERNAME}
  secret_references: []
  agent:
    monitoring:
      # enable otel collector for self-monitoring
      _runtime_experimental: otel
      enabled: true
      logs: true
      metrics: true
      namespace: default
      use_output: default
  inputs:
    - data_stream:
        namespace: default
      id: system-logs
      streams:
      - data_stream:
          dataset: system.auth
          type: logs
        exclude_files:
        - \.gz$
        ignore_older: 72h
        multiline:
          match: after
          pattern: ^\s
        paths:
        - /var/log/auth.log*
        - /var/log/secure*
        processors:
        - add_locale: null
        tags:
        - system-auth
      - data_stream:
          dataset: system.syslog
          type: logs
        exclude_files:
        - \.gz$
        ignore_older: 72h
        multiline:
          match: after
          pattern: ^\s
        paths:
        - /var/log/messages*
        - /var/log/syslog*
        - /var/log/system*
        processors:
        - add_locale: null
        tags: null
      type: logfile
      use_output: system
    - data_stream:
        namespace: default
      id: system-metrics
      streams:
      - cpu.metrics:
        - percentages
        - normalized_percentages
        data_stream:
          dataset: system.cpu
          type: metrics
        metricsets:
        - cpu
        period: 10s
      - data_stream:
          dataset: system.diskio
          type: metrics
        diskio.include_devices: null
        metricsets:
        - diskio
        period: 10s
      - data_stream:
          dataset: system.filesystem
          type: metrics
        metricsets:
        - filesystem
        period: 1m
        processors:
        - drop_event.when.regexp:
            system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
      - data_stream:
          dataset: system.fsstat
          type: metrics
        metricsets:
        - fsstat
        period: 1m
        processors:
        - drop_event.when.regexp:
            system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
      - condition: ${host.platform} != 'windows'
        data_stream:
          dataset: system.load
          type: metrics
        metricsets:
        - load
        period: 10s
      - data_stream:
          dataset: system.memory
          type: metrics
        metricsets:
        - memory
        period: 10s
      - data_stream:
          dataset: system.network
          type: metrics
        metricsets:
        - network
        network.interfaces: null
        period: 10s
      - data_stream:
          dataset: system.process
          type: metrics
        metricsets:
        - process
        period: 10s
        process.cgroups.enabled: false
        process.cmdline.cache.enabled: true
        process.include_cpu_ticks: false
        process.include_top_n.by_cpu: 5
        process.include_top_n.by_memory: 5
        processes:
        - .*
      - data_stream:
          dataset: system.process_summary
          type: metrics
        metricsets:
        - process_summary
        period: 10s
      - data_stream:
          dataset: system.socket_summary
          type: metrics
        metricsets:
        - socket_summary
        period: 10s
      - data_stream:
          dataset: system.uptime
          type: metrics
        metricsets:
        - uptime
        period: 10s
      type: system/metrics
      use_output: system

Related issues

N/A

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[blakerouse]

This change looks good to me. In call it was decided to just merge this with the otel bump. CI is green!

[swiatekm]

This change looks good to me. In call it was decided to just merge this with the otel bump. CI is green!

We should also bump the version in beats, though.

[pkoutsovasilis]

We should also bump the version in beats, though.

@swiatekm does this need to happen before this PR, or after? How these get bumped today in beats?

[swiatekm]

We should also bump the version in beats, though.

@swiatekm does this need to happen before this PR, or after? How these get bumped today in beats?

You bump in opentelemetry-collector-components, then in beats, and then in agent. I think it's fine to merge this PR, but if you want to be thorough, you should also bump component versions to ones which match the otel core version. Does that make sense?

[swiatekm]

The changes look good to me, but I'd really like to have a test verifying that the basic function of this PR works - that is, if we provide invalid TLS configuration to an elasticsearch output used by a beats receiver, the collector still starts and we get the right status. I think an integration test is the right way to do this, but I'd also accept a unit test for the otel manager.

[pkoutsovasilis]

You bump in opentelemetry-collector-components, then in beats, and then in agent. I think it's fine to merge this PR, but if you want to be thorough, you should also bump component versions to ones which match the otel core version. Does that make sense?

I can’t confidently own that bump chain right now. I’ll move this PR to Draft and keep it scoped to the minimal change, letting the existing version-bump mechanisms land first. Once those are in, I’ll rebase and flip back to Ready for Review.

The changes look good to me, but I'd really like to have a test verifying that the basic function of this PR works - that is, if we provide invalid TLS configuration to an elasticsearch output used by a beats receiver, the collector still starts and we get the right status. I think an integration

Agreed. I’ll add a test for that

cc @ebeahan @cmacknz

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feat/rework_otel_es_output_translation upstream/feat/rework_otel_es_output_translation
git merge upstream/main
git push upstream feat/rework_otel_es_output_translation

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 5255921

Failed CI Steps

• Unit tests - Ubuntu 22.04 ARM64
• Unit tests - Windows 11
• v1.29.14:amd64:wolfi,slim-wolfi,complete-wolfi,elastic-otel-collector-wolfi

History

• 💔 Build #28339 failed 3616b35
• 💛 Build #28201 was flaky 2f276ab
• 💔 Build #28111 failed edde990
• 💔 Build #28091 failed dab0b93

cc @pkoutsovasilis