Skip to content

Commit 6f89de5

Browse files
donoghucdonoghuc
authored
Improve smoke tests for observability SRE image (#17486)
* Improve smoke tests for observability SRE image This commit adds a new rspec test to run the observability SRE container in a docker compose network with filebeat and elasticsearch. It uses some simple test data through a pipeline with plugins we expect to be used in production. The rspec tests will ensure the test data is flowing from filebeat to logstash to elasticsearch by querying elasticsearch for expected transformed data. * REVERT ME: debug whats goig on in CI :( * Run filebeat container as root * Work around strict file ownership perms for filebeat We add the filebeat config in a volume, the permissions checks fail due test runner not being a root user. This commit disables that check in filebeat as seems to be the consensus solution online for example: https://event-driven.io/en/tricks_on_how_to_set_up_related_docker_images/ * Dynaimcally generate PKI instead of checking it in Instead of checking in PKI, dynamically generate it with gradle task for starting containers and running the tests. This improvement avoids github warning of checked in keys and avoid expiration headaches. Generation is very fast and does not add any significant overhead to test setup. * Remove use of "should" in rspec docstrings see https://github.com/rubocop/rspec-style-guide?tab=readme-ov-file#should-in-example-docstrings * Ensure permissions readable for volume Now that certs are dynamically generated, ensure they are able to be read in container * Use elasticsearch-fips image for smoke testing * Add git ignore for temp certs
1 parent 96bf497 commit 6f89de5

File tree

11 files changed

+478
-0
lines changed

11 files changed

+478
-0
lines changed

‎.buildkite/pull_request_pipeline.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -157,9 +157,15 @@ steps:
157157
set -euo pipefail
158158
source .buildkite/scripts/common/vm-agent.sh
159159
QUALIFIED_VERSION="$(.buildkite/scripts/common/qualified-version.sh)"
160+
# Build the image locally with the gradle task
160161
./gradlew --stacktrace artifactDockerObservabilitySRE -PfedrampHighMode=true
162+
# Ensure it can at least start logstash
161163
docker run docker.elastic.co/logstash/logstash-observability-sre:$${QUALIFIED_VERSION} \
162164
logstash -e 'input { generator { count => 3 } } output { stdout { codec => rubydebug } }'
165+
# Run the smoke tests on the PR code
166+
docker tag docker.elastic.co/logstash/logstash-observability-sre:$${QUALIFIED_VERSION} \
167+
pr-built-observability-sre-image
168+
./gradlew observabilitySREsmokeTests --stacktrace
163169
164170
- label: ":lab_coat: Integration Tests - FIPS mode / part 1-of-3"
165171
key: "integration-tests-fips-part-1-of-3"

‎x-pack/build.gradle‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,3 +75,42 @@ tasks.register("buildFipsValidationGem") {
7575
rake(rootProject.projectDir, rootProject.buildDir, 'plugin:build-fips-validation-plugin')
7676
}
7777
}
78+
tasks.register("observabilitySREsmokeTests", Test) {
79+
description = "Run ObservabilitySRE smoke tests using docker-compose and RSpec"
80+
// Need to have set up the ruby environment for rspec even through we are running in container
81+
dependsOn(":bootstrap", ":logstash-core:assemble", ":installDevelopmentGems")
82+
inputs.files fileTree("${projectDir}/distributions/internal/observabilitySRE/qa/smoke")
83+
doFirst {
84+
// Generate the certificates first
85+
exec {
86+
workingDir file("distributions/internal/observabilitySRE/qa/smoke/docker/certs")
87+
commandLine 'bash', './generate.sh'
88+
ignoreExitValue = false
89+
}
90+
def result = exec {
91+
workingDir file("distributions/internal/observabilitySRE/qa/smoke/docker")
92+
commandLine 'docker-compose', 'up', '-d'
93+
ignoreExitValue = true
94+
}
95+
if (result.exitValue != 0) {
96+
throw new GradleException("Docker compose failed to start")
97+
}
98+
// Give containers time to start and show logs
99+
sleep(30000)
100+
exec {
101+
workingDir file("distributions/internal/observabilitySRE/qa/smoke/docker")
102+
commandLine 'docker-compose', 'logs'
103+
}
104+
}
105+
systemProperty 'logstash.root.dir', projectDir.parent
106+
include '**/org/logstash/xpack/test/RSpecObservabilitySRETests.class'
107+
doLast {
108+
exec {
109+
workingDir file("distributions/internal/observabilitySRE/qa/smoke/docker")
110+
commandLine 'docker-compose', 'down', '-v'
111+
ignoreExitValue = true
112+
}
113+
// Clean up the generated certificates
114+
delete fileTree("distributions/internal/observabilitySRE/qa/smoke/docker/certs").include("*.key", "*.crt", "*.csr", "*.srl")
115+
}
116+
}

‎x-pack/distributions/internal/observabilitySRE/qa/smoke/docker/certs/.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
*.crt
2+
*.csr
3+
*.key

‎x-pack/distributions/internal/observabilitySRE/qa/smoke/docker/certs/generate.sh‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
#!/bin/bash
2+
3+
echo "Generating CA certificate"
4+
openssl req -x509 -newkey rsa:3072 -days 365 -nodes -keyout ca.key -out ca.crt -subj "/CN=Elastic-CA" -sha256
5+
6+
echo "Generating Elasticsearch certificate"
7+
openssl req -newkey rsa:3072 -nodes -keyout elasticsearch.key -out elasticsearch.csr -subj "/CN=elasticsearch" -sha256
8+
openssl x509 -req -in elasticsearch.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out elasticsearch.crt -days 365 -sha256
9+
10+
echo "Generating Logstash certificate"
11+
openssl req -newkey rsa:3072 -nodes -keyout logstash.key -out logstash.csr -subj "/CN=logstash" -sha256
12+
openssl x509 -req -in logstash.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out logstash.crt -days 365 -sha256
13+
14+
echo "Generating Filebeat certificate"
15+
openssl req -newkey rsa:3072 -nodes -keyout filebeat.key -out filebeat.csr -subj "/CN=filebeat" -sha256
16+
openssl x509 -req -in filebeat.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out filebeat.crt -days 365 -sha256
17+
18+
chmod 644 *.crt *.key

‎x-pack/distributions/internal/observabilitySRE/qa/smoke/docker/docker-compose.yml‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
version: '3'
2+
3+
services:
4+
elasticsearch:
5+
image: docker.elastic.co/elasticsearch/elasticsearch-fips:8.19.0-SNAPSHOT
6+
environment:
7+
- discovery.type=single-node
8+
- xpack.security.enabled=true
9+
- ELASTIC_PASSWORD=changeme
10+
- ES_JAVA_OPTS=-Xms512m -Xmx512m
11+
- xpack.security.http.ssl.enabled=true
12+
- xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certs/elasticsearch.key
13+
- xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certs/elasticsearch.crt
14+
- xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca.crt
15+
- xpack.security.transport.ssl.enabled=true
16+
- xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/elasticsearch.key
17+
- xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/elasticsearch.crt
18+
- xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca.crt
19+
ports:
20+
- "9200:9200"
21+
volumes:
22+
- ./certs:/usr/share/elasticsearch/config/certs
23+
networks:
24+
- smoketest
25+
26+
logstash:
27+
# We build the observability SRE image with the gradle task, but then tag it
28+
# as this in CI to ensure we are getting the local one built from the PR and not from
29+
# the container registry
30+
image: pr-built-observability-sre-image
31+
volumes:
32+
- ./logstash/pipeline:/usr/share/logstash/pipeline
33+
- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
34+
- ./certs:/usr/share/logstash/config/certs
35+
ports:
36+
- "5044:5044"
37+
depends_on:
38+
- elasticsearch
39+
networks:
40+
- smoketest
41+
42+
filebeat:
43+
image: docker.elastic.co/beats/filebeat:8.19.0-SNAPSHOT
44+
# Test runner mounts volume with non root user, do not require this file be root
45+
entrypoint: "filebeat -e --strict.perms=false"
46+
volumes:
47+
- ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
48+
- ./certs:/usr/share/filebeat/certs
49+
- ./test-logs:/test-logs:ro
50+
depends_on:
51+
- logstash
52+
networks:
53+
- smoketest
54+
55+
networks:
56+
smoketest:
57+
driver: bridge

‎x-pack/distributions/internal/observabilitySRE/qa/smoke/docker/filebeat/filebeat.yml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
filebeat.inputs:
2+
- type: log
3+
enabled: true
4+
paths:
5+
- /test-logs/*.log
6+
7+
output.logstash:
8+
hosts: ["logstash:5044"]
9+
ssl:
10+
enabled: true
11+
certificate_authorities: ["/usr/share/filebeat/certs/ca.crt"]
12+
certificate: "/usr/share/filebeat/certs/filebeat.crt"
13+
key: "/usr/share/filebeat/certs/filebeat.key"
14+
verification_mode: "full"
15+
supported_protocols: ["TLSv1.2"]

‎x-pack/distributions/internal/observabilitySRE/qa/smoke/docker/logstash/config/logstash.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
api.http.host: "0.0.0.0"
2+
xpack.monitoring.enabled: false
3+
4+
pipeline.ordered: false
5+
pipeline.workers: 2
6+
pipeline.buffer.type: heap

‎x-pack/distributions/internal/observabilitySRE/qa/smoke/docker/logstash/pipeline/logstash.conf‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
input {
2+
beats {
3+
port => 5044
4+
ssl_enabled => true
5+
ssl_certificate_authorities => ["/usr/share/logstash/config/certs/ca.crt"]
6+
ssl_certificate => "/usr/share/logstash/config/certs/logstash.crt"
7+
ssl_key => "/usr/share/logstash/config/certs/logstash.key"
8+
ssl_client_authentication => "required"
9+
ssl_supported_protocols => ["TLSv1.2"]
10+
}
11+
}
12+
13+
filter {
14+
grok {
15+
match => { "message" => "TEST-LOG: %{GREEDYDATA:log_content}" }
16+
}
17+
18+
if [log_content] =~ /timestamp=/ {
19+
grok {
20+
match => { "log_content" => ".*timestamp=%{TIMESTAMP_ISO8601:timestamp}.*" }
21+
}
22+
date {
23+
match => [ "timestamp", "ISO8601" ]
24+
target => "@timestamp"
25+
}
26+
}
27+
28+
age {}
29+
if [@metadata][age] > 86400 {
30+
mutate {
31+
add_tag => ["old_event"]
32+
}
33+
}
34+
35+
if [log_content] =~ /DEBUG/ {
36+
drop { }
37+
}
38+
39+
if [log_content] =~ /json=/ {
40+
grok {
41+
match => { "log_content" => ".*json=%{GREEDYDATA:json_string}.*" }
42+
}
43+
json {
44+
source => "json_string"
45+
target => "parsed_json"
46+
}
47+
}
48+
49+
fingerprint {
50+
source => ["message"]
51+
target => "fingerprint"
52+
method => "MD5"
53+
}
54+
55+
mutate {
56+
add_field => { "environment" => "test" }
57+
}
58+
}
59+
60+
output {
61+
elasticsearch {
62+
hosts => ["https://elasticsearch:9200"]
63+
user => "elastic"
64+
password => "changeme"
65+
ssl_enabled => true
66+
ssl_verification_mode => "full"
67+
ssl_certificate_authorities => ["/usr/share/logstash/config/certs/ca.crt"]
68+
index => "logs-%{+YYYY.MM.dd}"
69+
ssl_supported_protocols => ["TLSv1.2"]
70+
}
71+
}

‎x-pack/distributions/internal/observabilitySRE/qa/smoke/docker/test-logs/test.log‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
TEST-LOG: Simple example log entry
2+
TEST-LOG: Error message with status=500
3+
TEST-LOG: Log with json={"user":"testuser","action":"login","status":"success"}
4+
TEST-LOG: Log with json={"user":"admin","action":"update","items":5,"details":{"category":"config","changed":true}}
5+
TEST-LOG: Log with timestamp=2025-04-01T12:00:00Z for testing date filter
6+
TEST-LOG: Debug log message DEBUG should be dropped
7+
TEST-LOG: Log with timestamp=2024-04-01T12:00:00Z should be tagged as old_event

0 commit comments

Comments
 (0)