Skip to content
Open
3 changes: 2 additions & 1 deletion .github/ISSUE_TEMPLATE/bug_report.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
name: Bug Report
description: File a bug report
title: "[Bug]: "
labels: ["bug", "needs-triage"]
labels: ["needs-triage"]
type: ["bug"]
body:
- type: checkboxes
id: terms
Expand Down
3 changes: 2 additions & 1 deletion .github/ISSUE_TEMPLATE/feature_request.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
name: Feature Request
description: Request a feature or enhancement
title: "[Feature]: "
labels: ["feature", "needs-triage"]
labels: ["needs-triage"]
type: ["feature"]
body:
- type: markdown
attributes:
Expand Down
21 changes: 12 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

This collection installs and manages the Elastic Stack. It provides roles for every component of the Stack. Furthermore, it is possible to differentiate between Enterprise or OSS releases.

Every role is documented with all variables, please refer to the documentation found in **[Getting-Started](./docs/getting-started.md)**
Every role is documented with all variables, please refer to the documentation found in **[Getting-Started](./docs/02-getting-started.md)**

**Please note**: If you are already using this collection before version `1.0.0`, please note that we had to rename a significant amount of variables due to naming schema changes made by Ansible. Please review the variables you have set in your playbooks and variable files.

Expand All @@ -16,19 +16,22 @@ Every role is documented with all variables, please refer to the documentation f
> [!NOTE]
> Some roles have fixed requirements that you must observe. Please have a look at the [requirements](docs/01-requirements.md) before using the collection. (There is a high probability that some of them will be refactored soon)

* [Beats](docs/role-beats.md)
* [Elasticsearch](docs/role-elasticsearch.md)
* [Kibana](docs/role-kibana.md)
* [Logstash](docs/role-logstash.md)
* [Repos](docs/role-repos.md)
* [Beats](docs/14-role-beats.md)
* [Elasticsearch](docs/11-role-elasticsearch.md)
* [Kibana](docs/15-role-kibana.md)
* [Logstash](docs/12-role-logstash.md)
* [Repos](docs/10-role-repos.md)

## Modules documentation

* [elasticsearch_role](docs/module-elasticsearch_role.md)
* [elasticsearch_user](docs/module-elasticsearch_user.md)
* [elasticsearch_role](docs/20-module-elasticsearch_role.md)
* [elasticsearch_user](docs/21-module-elasticsearch_user.md)

## Global variables

* `elasticstack_collection_managed`: When set to true, roles inside this collection can fully rely on each other’s default configuration, naming conventions, and managed resources. This means the collection handles repositories, package names, and other components internally, leaving no room for external tools or custom configurations to override these defaults.
When set to false, you can provide your own repositories, package names, or other configurations, but roles cannot assume defaults from other roles anymore. (Default: `true`)

* `elasticstack_force_pip`: Will force installation of required Python modules via `pip`. This is useful if your package manager doesn't provide current versions of modules. (Default: `false`) See [PEP668](https://peps.python.org/pep-0668/) for more details.
* `elasticstack_manage_pip`: Will install `pip` on your system. (Default: `false`)

Expand Down Expand Up @@ -83,7 +86,7 @@ We know from personal experience, that the collections work in following combina

## Usage

Every role is documented with all variables, please refer to the documentation found in **[Getting-Started](./docs/getting-started.md)**
Every role is documented with all variables, please refer to the documentation found in **[Getting-Started](./docs/02-getting-started.md)**

Make sure all hosts that should be configured are part of your playbook. (See below for details on groups etc.). The collection is built to first collect all facts from all hosts (including those only running beats) and then use facts like hostnames or ip addresses to connect the tools to each other.

Expand Down
10 changes: 5 additions & 5 deletions docs/getting-started.md → docs/02-getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,11 @@ Or refer to the role with the FQCN of the role.
Roles
-------

* [Beats](role-beats.md)
* [Elasticsearch](role-elasticsearch.md)
* [Kibana](role-kibana.md)
* [Logstash](role-logstash.md)
* [Repos](role-repos.md)
* [Beats](14-role-beats.md)
* [Elasticsearch](11-role-elasticsearch.md)
* [Kibana](15-role-kibana.md)
* [Logstash](12-role-logstash.md)
* [Repos](10-role-repos.md)


Variables
Expand Down
45 changes: 45 additions & 0 deletions docs/10-role-repos.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Role `repos`

This role will add the Elastic repositories to the package manager (dnf/yum. apt or zypper). It will not install packages (these can be done with the role `elasticsearch`).

## Requirements

GPG needs to be installed on the systems to verify the package signature. This will be installed as part of the role. Below you can find a list of packages that will be installed.
* Debian family: `apt-transport-https`, `gpg` and `gpg-agent`
* RedHat family: `gnupg`
* Suse family: `gpg2`

For SuSE hosts you will need the Ansible collection `community.general` on your Ansible controller.

## Variables

There are variables that are needed in more than one role of the collection. These are defined inside the "meta" role `elasticstack`. The role `elasticstack` is imported in this role.

**List of variables from `elasticsearch` used inside `repos`:**
* `elasticstack_release`: Major release version of Elastic stack to configure. (default: `7`). `7` and `8` are supported.
* `elasticstack_variant`: Variant of the stack to install. Valid values: `elastic` or `oss`. (default: `elastic`).

Please note that no `oss` versions are available for Elastic Stack later than `7`. This role will fail if you try to install them.

**Variables that are explicity defined inside `repos`:**
* `repos_enable_repos`: Enable repositories after creating them. Only works on RPM based distributions (default: `true`)
* `repos_deb_remove_legacy_key`: Remove repository key from legacy trusted.gpg keyring (Only needed on older distributions with apt as package manager. Details can be found in [this](https://askubuntu.com/questions/1286545/what-commands-exactly-should-replace-the-deprecated-apt-key) post) (Default: `false`)

## Usage

### Upgrades

If you want to be able to update your operating system without worrying about accidentally upgrading Elastic Stack, set `repos_enable_repos` to `false`. The roles in this collection will enable the repository in case they need it. Keep in mind that this will only work on rpm based distributions.

### Example playbook

The following playbook will add the Elastic repository to the package manager.

```
- hosts: all
become: yes
collections:
- netways.elasticstack
roles:
- repos
```
File renamed without changes.
8 changes: 4 additions & 4 deletions docs/role-logstash.md → docs/12-role-logstash.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ It can optionally configure two types of Logstash pipelines:
* Pipeline configuration managed in an external git repository
* A default pipeline which will read from different Redis keys and write into Elasticsearch

For details on how to configure pipelines please refer to our [docs about pipelines](./logstash-pipelines.md).
For details on how to configure pipelines please refer to our [docs about pipelines](./13-logstash-pipelines.md).

Details about configured pipelines will be written into `pipelines.yml` as comments. Same goes for logging configuration in `log4j.options`.

Expand All @@ -24,7 +24,7 @@ You will need these packages / libraries installed. Some very basic packages lik

* `passlib` Python library if you do not disable password hashing for logstash user. It should be installed with pip on the Ansible controller.

You need to have the Elastic Repos configured on your system. You can use our [role](./role-repos.md)
You need to have the Elastic Repos configured on your system. You can use our [role](./14-role-repos.md)

If you want to use the default pipeline configuration you need to have `git` available.

Expand Down Expand Up @@ -86,7 +86,7 @@ Aside from `logstash.yml` we can manage Logstashs pipelines.
* *logstash_sniffing*: Enable sniffing (Default: `false`).
* *logstash_sniffing_delay*: How long to wait, in seconds, between sniffing attempts (Default: `not set`).
* *logstash_sniffing_path*: HTTP Path to be used for the sniffing requests (Default: `not set`).
* *logstash_legacy_monitoring*: Enables legacy monitoring - ignored when `elasticstack_full_stack` is not set. (default: `true`)
* *logstash_legacy_monitoring*: Enables legacy monitoring - ignored when `elasticstack_collection_managed` is not set. (default: `true`)
* *logstash_redis_password*: If set this will use this password when connecting our simple inputs and outputs to Redis. (default: not set)

* *logstash_mermaid*: Print overview over Logstash pipelines in Mermaid syntax. (default: `true`)
Expand Down Expand Up @@ -114,7 +114,7 @@ The following variables are identical over all our elastic related roles, hence

The following variables only apply if you use this role together with our Elasticsearch and Kibana roles.

* *elasticstack_full_stack*: Use `ansible-role-elasticsearch` as well (default: `false`)
* *elasticstack_collection_managed*: Use `ansible-role-elasticsearch` as well (default: `false`)
* *elasticstack_ca_dir*: Directory where the CA and certificates lie on the main Elasticsearch host (default: `/opt/es-ca`)
* *elasticstack_elasticsearch_http_port*: Port of Elasticsearch to send events to (Default: `9200`)
* *elasticstack_initial_passwords*: File where initial passwords are stored on the main Elasticsearch host (default: `/usr/share/elasticsearch/initial_passwords`)
Expand Down
File renamed without changes.
4 changes: 2 additions & 2 deletions docs/role-beats.md → docs/14-role-beats.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ beats_filebeat_journald_inputs:
* *beats_metricbeat_modules*: List of modules to enable. (Default: `- system`)
* *beats_metricbeat_loadbalance*: Enable loadbalancing for Metricbeats Logstash output (default: `true`)

* *beats_security*: Activate TLS for connections to targets. Can either be use with our other roles and `elasticstack_full_stack` to automatically create certificates or `beats_tls*` variables for custom certificates. (default: `false`)
* *beats_security*: Activate TLS for connections to targets. Can either be use with our other roles and `elasticstack_collection_managed` to automatically create certificates or `beats_tls*` variables for custom certificates. (default: `false`)
* *beats_target_hosts*: Only use when this role is used standalone. When used in combination with our other roles, the target hosts will be determined automatically. Use a YAML list. (default: `- localhost`)
* *elasticstack_elasticsearch_http_port*: Port of Elasticsearch to send events to (Default: `9200`)
* *elasticstack_beats_port*: Port of Logstash to send events to (Default: `5044`)
Expand All @@ -90,7 +90,7 @@ beats_filebeat_journald_inputs:

The following variables only apply if you use this role together with our other Elastic Stack roles.

* *elasticstack_full_stack*: Use `elasticsearch` as well (default: `false`)
* *elasticstack_collection_managed*: Use `elasticsearch` as well (default: `false`)
* *elasticstack_variant*: Define which variant of elastic stack to use. (default: `elastic`)
* *elasticstack_ca_dir*: Directory where on the Elasticsearch CA host certificates are stored. This is only useful in connection with out other Elastic Stack related roles. (default: `/opt/es-ca`)
* *elasticstack_ca_pass*: Password for Elasticsearch CA (default: `PleaseChangeMe`)
Expand Down
6 changes: 3 additions & 3 deletions docs/role-kibana.md → docs/15-role-kibana.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@ Role Variables
* *kibana_tls_key*: Path to the key Kibana should use when communicating with clients (default: `/etc/kibana/certs/key.pem`)
* *kibana_extra_config*: You can add arbitraty configuration options with this option. Just start it with `|-` and indent the following lines. So you can add as many lines and options to `kibana.yml` as you like. (default: none)

* *kibana_security*: Activate TLS and authentication when connecting to Elasticsearch. **Note**: Only works when `elasticstack_full_stack` is enabled. (default: `true`)
* *kibana_security*: Activate TLS and authentication when connecting to Elasticsearch. **Note**: Only works when `elasticstack_collection_managed` is enabled. (default: `true`)

These variables are identical over all our elastic related roles, hence the different naming scheme.

* *elasticstack_full_stack*: Use `ansible-role-elasticsearch` as well (default: `false`). If you set to `true`, you will be able to use the following variables:
* *elasticstack_collection_managed*: Use `ansible-role-elasticsearch` as well (default: `false`). If you set to `true`, you will be able to use the following variables:
* *kibana_elasticsearch_hosts*: A list of DNS resolvable hostnames of Elasticsearch hosts to connect your Kibana instance to. (default: `- localhost`)
* *elasticstack_elasticsearch_http_port*: Port of Elasticsearch http (Default: `9200`)
* *kibana_sniff_on_start*: Attempt to find other Elasticsearch nodes on startup (default: `false`)
Expand Down Expand Up @@ -50,7 +50,7 @@ If you use `localhost` in `kibana_elasticsearch_hosts` , certificate verificatio
- netways.elasticstack
hosts: kibana-host
vars:
elasticstack_full_stack: true
elasticstack_collection_managed: true
elasticstack_variant: oss
roles:
- repos
Expand Down
File renamed without changes.
File renamed without changes.
43 changes: 0 additions & 43 deletions docs/role-repos.md

This file was deleted.

2 changes: 1 addition & 1 deletion molecule/beats_default/converge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
collections:
- netways.elasticstack
vars:
elasticstack_full_stack: false
elasticstack_collection_managed: false
elasticsearch_jna_workaround: true
elasticsearch_disable_systemcallfilterchecks: true
elasticstack_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"
Expand Down
2 changes: 1 addition & 1 deletion molecule/beats_peculiar/converge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
testfield2: thisisanothertest
elasticsearch_jna_workaround: true
elasticsearch_disable_systemcallfilterchecks: true
elasticstack_full_stack: false
elasticstack_collection_managed: false
elasticstack_no_log: false
beats_filebeat_mysql_slowlog_input: true
beats_auditbeat: true
Expand Down
2 changes: 1 addition & 1 deletion molecule/elasticsearch_default/converge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
hosts: all
vars:
#elasticsearch_security: true # needed for tests of > 7 releases
elasticstack_full_stack: false
elasticstack_collection_managed: false
elasticsearch_jna_workaround: true
elasticsearch_disable_systemcallfilterchecks: true
elasticstack_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"
Expand Down
2 changes: 1 addition & 1 deletion molecule/elasticsearch_test_modules/converge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
- netways.elasticstack
hosts: all
vars:
elasticstack_full_stack: false
elasticstack_collection_managed: false
elasticsearch_jna_workaround: true
elasticsearch_disable_systemcallfilterchecks: true
elasticstack_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"
Expand Down
6 changes: 1 addition & 5 deletions molecule/elasticstack_default/converge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
elasticsearch_monitoring_enabled: false
elasticstack_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"
elasticsearch_heap: "2"
elasticstack_full_stack: true
elasticstack_collection_managed: true
elasticstack_no_log: false
logstash_pipeline_unsafe_shutdown: true
logstash_redis_password: "{{ lookup('ansible.builtin.password', '/tmp/redispassword', chars=['ascii_letters'], length=15) }}"
Expand All @@ -28,10 +28,6 @@
kibana_extra_config: |-
ops.interval: 5000
tasks:
- name: Enable Elastic installation on RHEL 9
ansible.builtin.set_fact:
elasticstack_rpm_workaround: true
when: ansible_os_family == 'RedHat' and ansible_distribution_major_version >= "9"
- name: Include Redis
ansible.builtin.include_role:
name: geerlingguy.redis
Expand Down
2 changes: 1 addition & 1 deletion molecule/kibana_default/converge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
- name: Converge
hosts: all
vars:
elasticstack_full_stack: false
elasticstack_collection_managed: false
elasticstack_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"
elasticstack_no_log: false
collections:
Expand Down
2 changes: 1 addition & 1 deletion molecule/logstash_full_stack-oss/converge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
collections:
- netways.elasticstack
vars:
elasticstack_full_stack: true
elasticstack_collection_managed: true
elasticstack_variant: oss
logstash_security: false
logstash_pipeline_unsafe_shutdown: true
Expand Down
2 changes: 1 addition & 1 deletion molecule/logstash_pipelines/converge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
logstash_pipeline_unsafe_shutdown: true
redis_requirepass: "{{ logstash_redis_password }}"
elasticstack_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"
elasticstack_full_stack: false
elasticstack_collection_managed: false
elasticstack_no_log: false
tasks:
- name: "Include Elastics repos role"
Expand Down
2 changes: 1 addition & 1 deletion molecule/logstash_specific_version/converge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
logstash_pipeline_identifier: false
logstash_pipeline_unsafe_shutdown: true
elasticstack_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"
elasticstack_full_stack: false
elasticstack_collection_managed: false
elasticstack_no_log: false
tasks:

Expand Down
3 changes: 1 addition & 2 deletions molecule/repos_default/converge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,7 @@
collections:
- netways.elasticstack
vars:
elasticstack_rpm_workaround: true
elasticstack_full_stack: false
elasticstack_collection_managed: false
elasticstack_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"
elasticstack_no_log: false
tasks:
Expand Down
1 change: 0 additions & 1 deletion molecule/repos_oss/converge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
- netways.elasticstack
vars:
elasticstack_variant: oss
elasticstack_rpm_workaround: true
elasticstack_release: 7
elasticstack_no_log: false
tasks:
Expand Down
1 change: 0 additions & 1 deletion roles/beats/defaults/main.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
---
# defaults file for beats
beats_security: false
beats_filebeat: true
beats_auditbeat: false
Expand Down
2 changes: 0 additions & 2 deletions roles/beats/handlers/main.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
---
# handlers file for beats
#
- name: Restart Filebeat
ansible.builtin.service:
name: filebeat
Expand Down
Loading
Loading