xpack management TLS CA file no longer works 8.8.0

[Aqualie]

Logstash information:

Please include the following information:

1. Logstash version (e.g. bin/logstash --version)
   8.8.0

2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)

Docker
4. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
K8S

Plugins installed: (bin/logstash-plugin list --verbose)
Default

JVM (e.g. java -version):
Starting Logstash {"logstash.version"=>"8.8.0", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.7+7 on 17.0.7+7 +indy +jit [x86_64-linux]"}

OS version (uname -a if on a Unix-like system):

Description of the problem including expected versus actual behavior:
After upgrading to 8.8.0 from 8.7.1 Logstash fails to start due to xpack management TLS certificate error message

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1. Upgrade from 8.7.1 to 8.8.0 with xpack.management.elasticsearch.ssl.certificate_authority configured

Provide logs (if relevant):

logstash-1 logstash [2023-05-25T14:13:27,199][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}
logstash-1 logstash [2023-05-25T14:13:27,201][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://REPLACED:9200/_xpack][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target {:url=>https://REPLACED:xxxxxx@REPLACED:9200/, :error_message=>"Elasticsearch Unreachable: [https://REPLACED:9200/_xpack][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
logstash-1 logstash [2023-05-25T14:13:27,202][WARN ][logstash.licensechecker.licensereader] Attempt to validate Elasticsearch license failed. Sleeping for 0.04 {:fail_count=>2, :exception=>"Elasticsearch Unreachable: [https://REPLACED:9200/_xpack][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
logstash-1 logstash [2023-05-25T14:13:27,243][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
logstash-1 logstash [2023-05-25T14:13:27,254][ERROR][logstash.configmanagement.elasticsearchsource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
logstash-1 logstash [2023-05-25T14:13:27,255][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::LicenseChecker::LicenseError: Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.>, :backtrace=>["/usr/share/logstash/x-pack/lib/license_checker/licensed.rb:67:in `with_license_check'", "/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:43:in `initialize'", "org/jruby/RubyClass.java:890:in `new'", "/usr/share/logstash/x-pack/lib/config_management/hooks.rb:41:in `after_bootstrap_checks'", "org/logstash/execution/EventDispatcherExt.java:94:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:363:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/clamp-1.0.1/lib/clamp/command.rb:68:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:287:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/clamp-1.0.1/lib/clamp/command.rb:133:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:90:in `<main>'"]}
logstash-1 logstash [2023-05-25T14:13:27,259][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit

config:

xpack.management.elasticsearch.ssl.certificate_authority: "/usr/share/logstash/conf.d/ssl/elasticsearch-ca.pem"
xpack.management.elasticsearch.ssl.verification_mode: "certificate"

mounted secret in container:

logstash@logstash-0:~$ ls -lh  /usr/share/logstash/conf.d/ssl/elasticsearch-ca.pem
lrwxrwxrwx 1 root logstash 27 May 13 22:23 /usr/share/logstash/conf.d/ssl/elasticsearch-ca.pem -> ..data/elasticsearch-ca.pem

[edmocosta]

Hi @Aqualie, thank you for taking the time to file this issue.
Could you please try to upgrade the logstash-output-elasticsearch plugin to the latest version 11.15.7? there was a regression on that plugin that might be causing this issue. This PR probably fixed the problem.

[roaksoax]

Hi @Aqualie ,

As per @edmocosta , we believe this issue is now fixed and to be able to verify, you need to upgrade the ES output to the specified version above. I'm going to go ahead and close this issue.

If you, however, believe this issue is still present after upgrading & restarting Logstash, feel free to re-open this issue or file a new one.

[Aqualie]

Upgraded to Logstash 8.8.1 same error

[edmocosta]

Hi @Aqualie! Considering that the certificates are valid, this issue might be related to the recently discovered bug #1141. Updating the logstash-output-elasticsearch plugin to 11.15.8 should fix the problem (sorry for asking you to update it again). Please feel free to re-open this issue or file a new one if that doesn't work for you.

[BBQigniter]

so do i understand this correctly - this change didn't make it into the logstash v8.8.1 docker images?

Edit: Just verfied myself - inside the v8.8.1 logstash image there is logstash-output-elasticsearch plugin 11.15.7 installed. I now tried to upgrade the plugin via an initContainer in one of our logstash-workloads, but it didn't help.

[BBQigniter]

thanks to elastic's support - @Aqualie as a workaround try to set

xpack.monitoring.elasticsearch.ssl.ca_trusted_fingerprint and xpack.management.elasticsearch.ssl.ca_trusted_fingerprint too in the config.

the fingerprint has to be created from the CA-cert - if you use ECK, you can quickly get the fingerprint via:

# you may have to install "jq" via your OS package manager   
# fingerprint must be updated if elastic-operator is rolling over the certs! 
kubectl get secrets -n <your-elk-namespace> <your-elk>-es-http-certs-public -o json | jq -r '.data."ca.crt"' | base64 -d | openssl x509 -outform der | sha256sum | awk '{print $1}'

edit: at least this is how it worked for me :)

[Aqualie]

Thanks worked for me aswell

[c4m4]

Updating logstash solved for me