Skip to content

Conversation

@yaauie
Copy link
Member

@yaauie yaauie commented Feb 20, 2025

Release notes

[rn:skip]

What does this PR do?

routes output from setup-related bits in the entrypoint scripts to stderr, so that stdout can include only the output of the actual program.

Why is it important/What is the impact to the user?

This change makes it possible to use the stdout of entrypoints like bin/logstash-plugin without the setup-related pollution:

Using system java: /Users/rye/.jenv/shims/java

For example, with #17124 addition of --no-expand, and given an allowlist containing the names of plugins to keep,

comm -1 -2 <(cat allowlist | sort) <(bin/logstash-plugin list --no-expand | sort) | xargs bin/logstash-plugin remove 

The above is not possible if the stdout of bin/logstash-plugin invocation is littered with info about the JVM.

Checklist

  • My code follows the style guidelines of this project
  • [ ] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files (and/or docker env variables)
  • [ ] I have added tests that prove my fix is effective or that my feature works

Author's Checklist

  • manual windows validation (are acceptance tests enough?)

How to test this PR locally

observe that the side-channel bits are on STDERR and can be redirected away

bin/logstash-plugin list --verbose 2>/dev/null

Logs

With patch, the info about which java is being used is on STDERR:

╭─{ rye@perhaps:~/src/elastic/logstash@main (pluginmanager-list-simple ✘) }
╰─● bin/logstash-plugin list | awk '{print "[PRGOUT]:" $0}' 
Using system java: /Users/rye/.jenv/shims/java
[PRGOUT]:logstash-codec-avro
[PRGOUT]:logstash-codec-cef
[PRGOUT]:logstash-codec-collectd
[PRGOUT]:logstash-codec-dots
[PRGOUT]:logstash-codec-edn
[PRGOUT]:logstash-codec-edn_lines
[PRGOUT]:logstash-codec-es_bulk
[PRGOUT]:logstash-codec-fluent
[PRGOUT]:logstash-codec-graphite
[PRGOUT]:logstash-codec-json
[PRGOUT]:logstash-codec-json_lines
[PRGOUT]:logstash-codec-line
[PRGOUT]:logstash-codec-msgpack
[PRGOUT]:logstash-codec-multiline
[PRGOUT]:logstash-codec-netflow
[PRGOUT]:logstash-codec-plain
[PRGOUT]:logstash-codec-rubydebug
[PRGOUT]:logstash-filter-aggregate
[PRGOUT]:logstash-filter-anonymize
[PRGOUT]:logstash-filter-cidr
[PRGOUT]:logstash-filter-clone
[PRGOUT]:logstash-filter-csv
[PRGOUT]:logstash-filter-date
[PRGOUT]:logstash-filter-de_dot
[PRGOUT]:logstash-filter-dissect
[PRGOUT]:logstash-filter-dns
[PRGOUT]:logstash-filter-drop
[PRGOUT]:logstash-filter-elastic_integration
[PRGOUT]:logstash-filter-elasticsearch
[PRGOUT]:logstash-filter-fingerprint
[PRGOUT]:logstash-filter-geoip
[PRGOUT]:logstash-filter-grok
[PRGOUT]:logstash-filter-http
[PRGOUT]:logstash-filter-json
[PRGOUT]:logstash-filter-kv
[PRGOUT]:logstash-filter-memcached
[PRGOUT]:logstash-filter-metrics
[PRGOUT]:logstash-filter-mutate
[PRGOUT]:logstash-filter-prune
[PRGOUT]:logstash-filter-ruby
[PRGOUT]:logstash-filter-sleep
[PRGOUT]:logstash-filter-split
[PRGOUT]:logstash-filter-syslog_pri
[PRGOUT]:logstash-filter-throttle
[PRGOUT]:logstash-filter-translate
[PRGOUT]:logstash-filter-truncate
[PRGOUT]:logstash-filter-urldecode
[PRGOUT]:logstash-filter-useragent
[PRGOUT]:logstash-filter-uuid
[PRGOUT]:logstash-filter-xml
[PRGOUT]:logstash-input-azure_event_hubs
[PRGOUT]:logstash-input-beats
[PRGOUT]: └── logstash-input-elastic_agent (alias)
[PRGOUT]:logstash-input-couchdb_changes
[PRGOUT]:logstash-input-dead_letter_queue
[PRGOUT]:logstash-input-elastic_serverless_forwarder
[PRGOUT]:logstash-input-elasticsearch
[PRGOUT]:logstash-input-exec
[PRGOUT]:logstash-input-file
[PRGOUT]:logstash-input-ganglia
[PRGOUT]:logstash-input-gelf
[PRGOUT]:logstash-input-generator
[PRGOUT]:logstash-input-graphite
[PRGOUT]:logstash-input-heartbeat
[PRGOUT]:logstash-input-http
[PRGOUT]:logstash-input-http_poller
[PRGOUT]:logstash-input-jms
[PRGOUT]:logstash-input-pipe
[PRGOUT]:logstash-input-redis
[PRGOUT]:logstash-input-stdin
[PRGOUT]:logstash-input-syslog
[PRGOUT]:logstash-input-tcp
[PRGOUT]:logstash-input-twitter
[PRGOUT]:logstash-input-udp
[PRGOUT]:logstash-input-unix
[PRGOUT]:logstash-integration-aws
[PRGOUT]: ├── logstash-codec-cloudfront
[PRGOUT]: ├── logstash-codec-cloudtrail
[PRGOUT]: ├── logstash-input-cloudwatch
[PRGOUT]: ├── logstash-input-s3
[PRGOUT]: ├── logstash-input-sqs
[PRGOUT]: ├── logstash-output-cloudwatch
[PRGOUT]: ├── logstash-output-s3
[PRGOUT]: ├── logstash-output-sns
[PRGOUT]: └── logstash-output-sqs
[PRGOUT]:logstash-integration-jdbc
[PRGOUT]: ├── logstash-input-jdbc
[PRGOUT]: ├── logstash-filter-jdbc_streaming
[PRGOUT]: └── logstash-filter-jdbc_static
[PRGOUT]:logstash-integration-kafka
[PRGOUT]: ├── logstash-input-kafka
[PRGOUT]: └── logstash-output-kafka
[PRGOUT]:logstash-integration-logstash
[PRGOUT]: ├── logstash-input-logstash
[PRGOUT]: └── logstash-output-logstash
[PRGOUT]:logstash-integration-rabbitmq
[PRGOUT]: ├── logstash-input-rabbitmq
[PRGOUT]: └── logstash-output-rabbitmq
[PRGOUT]:logstash-integration-snmp
[PRGOUT]: ├── logstash-input-snmp
[PRGOUT]: └── logstash-input-snmptrap
[PRGOUT]:logstash-output-csv
[PRGOUT]:logstash-output-elasticsearch
[PRGOUT]:logstash-output-email
[PRGOUT]:logstash-output-file
[PRGOUT]:logstash-output-graphite
[PRGOUT]:logstash-output-http
[PRGOUT]:logstash-output-lumberjack
[PRGOUT]:logstash-output-nagios
[PRGOUT]:logstash-output-null
[PRGOUT]:logstash-output-pipe
[PRGOUT]:logstash-output-redis
[PRGOUT]:logstash-output-stdout
[PRGOUT]:logstash-output-tcp
[PRGOUT]:logstash-output-udp
[PRGOUT]:logstash-output-webhdfs
[PRGOUT]:logstash-patterns-core
[success]                                                                                                                                                                                                                                                                                                                                                                                                                        

Without patch, the info about which java is being used is on STDOUT:

╭─{ rye@perhaps:~/src/elastic/logstash@main (pluginmanager-list-simple ✘) }
╰─● bin/logstash-plugin list | awk '{print "[PRGOUT]:" $0}'
[PRGOUT]:Using system java: /Users/rye/.jenv/shims/java
[PRGOUT]:logstash-codec-avro
[PRGOUT]:logstash-codec-cef
[PRGOUT]:logstash-codec-collectd
[PRGOUT]:logstash-codec-dots
[PRGOUT]:logstash-codec-edn
[PRGOUT]:logstash-codec-edn_lines
[PRGOUT]:logstash-codec-es_bulk
[PRGOUT]:logstash-codec-fluent
[PRGOUT]:logstash-codec-graphite
[PRGOUT]:logstash-codec-json
[PRGOUT]:logstash-codec-json_lines
[PRGOUT]:logstash-codec-line
[PRGOUT]:logstash-codec-msgpack
[PRGOUT]:logstash-codec-multiline
[PRGOUT]:logstash-codec-netflow
[PRGOUT]:logstash-codec-plain
[PRGOUT]:logstash-codec-rubydebug
[PRGOUT]:logstash-filter-aggregate
[PRGOUT]:logstash-filter-anonymize
[PRGOUT]:logstash-filter-cidr
[PRGOUT]:logstash-filter-clone
[PRGOUT]:logstash-filter-csv
[PRGOUT]:logstash-filter-date
[PRGOUT]:logstash-filter-de_dot
[PRGOUT]:logstash-filter-dissect
[PRGOUT]:logstash-filter-dns
[PRGOUT]:logstash-filter-drop
[PRGOUT]:logstash-filter-elastic_integration
[PRGOUT]:logstash-filter-elasticsearch
[PRGOUT]:logstash-filter-fingerprint
[PRGOUT]:logstash-filter-geoip
[PRGOUT]:logstash-filter-grok
[PRGOUT]:logstash-filter-http
[PRGOUT]:logstash-filter-json
[PRGOUT]:logstash-filter-kv
[PRGOUT]:logstash-filter-memcached
[PRGOUT]:logstash-filter-metrics
[PRGOUT]:logstash-filter-mutate
[PRGOUT]:logstash-filter-prune
[PRGOUT]:logstash-filter-ruby
[PRGOUT]:logstash-filter-sleep
[PRGOUT]:logstash-filter-split
[PRGOUT]:logstash-filter-syslog_pri
[PRGOUT]:logstash-filter-throttle
[PRGOUT]:logstash-filter-translate
[PRGOUT]:logstash-filter-truncate
[PRGOUT]:logstash-filter-urldecode
[PRGOUT]:logstash-filter-useragent
[PRGOUT]:logstash-filter-uuid
[PRGOUT]:logstash-filter-xml
[PRGOUT]:logstash-input-azure_event_hubs
[PRGOUT]:logstash-input-beats
[PRGOUT]: └── logstash-input-elastic_agent (alias)
[PRGOUT]:logstash-input-couchdb_changes
[PRGOUT]:logstash-input-dead_letter_queue
[PRGOUT]:logstash-input-elastic_serverless_forwarder
[PRGOUT]:logstash-input-elasticsearch
[PRGOUT]:logstash-input-exec
[PRGOUT]:logstash-input-file
[PRGOUT]:logstash-input-ganglia
[PRGOUT]:logstash-input-gelf
[PRGOUT]:logstash-input-generator
[PRGOUT]:logstash-input-graphite
[PRGOUT]:logstash-input-heartbeat
[PRGOUT]:logstash-input-http
[PRGOUT]:logstash-input-http_poller
[PRGOUT]:logstash-input-jms
[PRGOUT]:logstash-input-pipe
[PRGOUT]:logstash-input-redis
[PRGOUT]:logstash-input-stdin
[PRGOUT]:logstash-input-syslog
[PRGOUT]:logstash-input-tcp
[PRGOUT]:logstash-input-twitter
[PRGOUT]:logstash-input-udp
[PRGOUT]:logstash-input-unix
[PRGOUT]:logstash-integration-aws
[PRGOUT]: ├── logstash-codec-cloudfront
[PRGOUT]: ├── logstash-codec-cloudtrail
[PRGOUT]: ├── logstash-input-cloudwatch
[PRGOUT]: ├── logstash-input-s3
[PRGOUT]: ├── logstash-input-sqs
[PRGOUT]: ├── logstash-output-cloudwatch
[PRGOUT]: ├── logstash-output-s3
[PRGOUT]: ├── logstash-output-sns
[PRGOUT]: └── logstash-output-sqs
[PRGOUT]:logstash-integration-jdbc
[PRGOUT]: ├── logstash-input-jdbc
[PRGOUT]: ├── logstash-filter-jdbc_streaming
[PRGOUT]: └── logstash-filter-jdbc_static
[PRGOUT]:logstash-integration-kafka
[PRGOUT]: ├── logstash-input-kafka
[PRGOUT]: └── logstash-output-kafka
[PRGOUT]:logstash-integration-logstash
[PRGOUT]: ├── logstash-input-logstash
[PRGOUT]: └── logstash-output-logstash
[PRGOUT]:logstash-integration-rabbitmq
[PRGOUT]: ├── logstash-input-rabbitmq
[PRGOUT]: └── logstash-output-rabbitmq
[PRGOUT]:logstash-integration-snmp
[PRGOUT]: ├── logstash-input-snmp
[PRGOUT]: └── logstash-input-snmptrap
[PRGOUT]:logstash-output-csv
[PRGOUT]:logstash-output-elasticsearch
[PRGOUT]:logstash-output-email
[PRGOUT]:logstash-output-file
[PRGOUT]:logstash-output-graphite
[PRGOUT]:logstash-output-http
[PRGOUT]:logstash-output-lumberjack
[PRGOUT]:logstash-output-nagios
[PRGOUT]:logstash-output-null
[PRGOUT]:logstash-output-pipe
[PRGOUT]:logstash-output-redis
[PRGOUT]:logstash-output-stdout
[PRGOUT]:logstash-output-tcp
[PRGOUT]:logstash-output-udp
[PRGOUT]:logstash-output-webhdfs
[PRGOUT]:logstash-patterns-core
[success]

routes output from setup-related functions to stderr, so that stdout can
include only the output of the actual program.
@elastic-sonarqube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@elasticmachine
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Copy link
Member

@donoghuc donoghuc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This certainly looks correct and idiomatic for shell and batch wrappers. I cant think of any compatability issues here in terms of surprising new behavior. 👍

@yaauie yaauie merged commit 91258c3 into elastic:main Feb 20, 2025
7 checks passed
@yaauie
Copy link
Member Author

yaauie commented Feb 20, 2025

@logstashmachine backport 9.0

@yaauie
Copy link
Member Author

yaauie commented Feb 20, 2025

@logstashmachine backport 8.x

github-actions bot pushed a commit that referenced this pull request Feb 20, 2025
routes output from setup-related functions to stderr, so that stdout can
include only the output of the actual program.

(cherry picked from commit 91258c3)
github-actions bot pushed a commit that referenced this pull request Feb 20, 2025
routes output from setup-related functions to stderr, so that stdout can
include only the output of the actual program.

(cherry picked from commit 91258c3)
donoghuc added a commit to donoghuc/logstash that referenced this pull request Feb 21, 2025
In elastic#17125 jvm setup was redirected to
stderr to avoid polluting stdout. This test was actually having to do some
additional processing to parse that information. Now that we have split the
destinations the tests can be simplified to look for the data they are trying to
validate on the appropriate stream.
donoghuc added a commit that referenced this pull request Feb 21, 2025
In #17125 jvm setup was redirected to
stderr to avoid polluting stdout. This test was actually having to do some
additional processing to parse that information. Now that we have split the
destinations the tests can be simplified to look for the data they are trying to
validate on the appropriate stream.
github-actions bot pushed a commit that referenced this pull request Feb 21, 2025
In #17125 jvm setup was redirected to
stderr to avoid polluting stdout. This test was actually having to do some
additional processing to parse that information. Now that we have split the
destinations the tests can be simplified to look for the data they are trying to
validate on the appropriate stream.

(cherry picked from commit 227c0d8)
github-actions bot pushed a commit that referenced this pull request Feb 21, 2025
In #17125 jvm setup was redirected to
stderr to avoid polluting stdout. This test was actually having to do some
additional processing to parse that information. Now that we have split the
destinations the tests can be simplified to look for the data they are trying to
validate on the appropriate stream.

(cherry picked from commit 227c0d8)
donoghuc pushed a commit that referenced this pull request Feb 21, 2025
routes output from setup-related functions to stderr, so that stdout can
include only the output of the actual program.

(cherry picked from commit 91258c3)

Co-authored-by: Ry Biesemeyer <[email protected]>
donoghuc pushed a commit that referenced this pull request Feb 21, 2025
routes output from setup-related functions to stderr, so that stdout can
include only the output of the actual program.

(cherry picked from commit 91258c3)

Co-authored-by: Ry Biesemeyer <[email protected]>
donoghuc added a commit that referenced this pull request Feb 21, 2025
…#17139)

In #17125 jvm setup was redirected to
stderr to avoid polluting stdout. This test was actually having to do some
additional processing to parse that information. Now that we have split the
destinations the tests can be simplified to look for the data they are trying to
validate on the appropriate stream.

(cherry picked from commit 227c0d8)

Co-authored-by: Cas Donoghue <[email protected]>
donoghuc added a commit that referenced this pull request Feb 21, 2025
…#17140)

In #17125 jvm setup was redirected to
stderr to avoid polluting stdout. This test was actually having to do some
additional processing to parse that information. Now that we have split the
destinations the tests can be simplified to look for the data they are trying to
validate on the appropriate stream.

(cherry picked from commit 227c0d8)

Co-authored-by: Cas Donoghue <[email protected]>
v1v pushed a commit to v1v/logstash that referenced this pull request Feb 25, 2025
routes output from setup-related functions to stderr, so that stdout can
include only the output of the actual program.
v1v pushed a commit to v1v/logstash that referenced this pull request Feb 25, 2025
…#17138)

In elastic#17125 jvm setup was redirected to
stderr to avoid polluting stdout. This test was actually having to do some
additional processing to parse that information. Now that we have split the
destinations the tests can be simplified to look for the data they are trying to
validate on the appropriate stream.
v1v added a commit to v1v/logstash that referenced this pull request Feb 25, 2025
…gify

* upstream/main: (27 commits)
  Add Windows 2025 to CI (elastic#17133)
  Update container acceptance tests with stdout/stderr changes (elastic#17138)
  entrypoint: avoid polluting stdout (elastic#17125)
  Fix acceptance test assertions for updated plugin remove (elastic#17126)
  Fix acceptance test assertions for updated plugin `remove` (elastic#17122)
  plugins: improve `remove` command to support multiple plugins (elastic#17030)
  spec: improve ls2ls spec (elastic#17114)
  allow concurrent Batch deserialization (elastic#17050)
  CPM handle 404 response gracefully with user-friendly log (elastic#17052)
  qa: use clean expansion of LS tarball per fixture instance (elastic#17082)
  Allow capturing heap dumps in DRA BK jobs (elastic#17081)
  Use centralized source of truth for active branches (elastic#17063)
  Update logstash_releases.json (elastic#17055)
  fix logstash-keystore to accept spaces in values when added via stdin (elastic#17039)
  Don't honor VERSION_QUALIFIER if set but empty (elastic#17032)
  Release note placeholder might be empty, making parsing lines nil tolerant. (elastic#17026)
  Fix BufferedTokenizer to properly resume after a buffer full condition respecting the encoding of the input string (elastic#16968)
  Add short living 9.0 next and update main in CI release version definition. (elastic#17008)
  Core version bump to 9.1.0 (elastic#16991)
  Add 9.0 branch to the CI branches definition (elastic#16997)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants