Skip to content

extending otel config #4702

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
+149 −67
Open

extending otel config #4702

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5de91b9
extending otel config
dhtclk Nov 4, 2025
ad92b7b
Merge branch 'main' of https://github.com/ClickHouse/clickhouse-docs …
dhtclk Nov 5, 2025
2e39ec2
updating local data guide in example datasets to reflect otel config …
dhtclk Nov 5, 2025
009ccce
removing sections that don't make sense
dhtclk Nov 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 58 additions & 63 deletions docs/use-cases/observability/clickstack/example-datasets/local-data.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,57 +10,36 @@
---

import Image from '@theme/IdealImage';
import hyperdx from '@site/static/images/use-cases/observability/hyperdx-1.png';
import hyperdx_20 from '@site/static/images/use-cases/observability/hyperdx-20.png';
import hyperdx_3 from '@site/static/images/use-cases/observability/hyperdx-3.png';
import hyperdx_4 from '@site/static/images/use-cases/observability/hyperdx-4.png';
import hyperdx_21 from '@site/static/images/use-cases/observability/hyperdx-21.png';
import hyperdx_22 from '@site/static/images/use-cases/observability/hyperdx-22.png';
import hyperdx_23 from '@site/static/images/use-cases/observability/hyperdx-23.png';
import copy_api_key from '@site/static/images/use-cases/observability/copy_api_key.png';

This getting started guide allows you collect local logs and metrics from your system, sending them to ClickStack for visualization and analysis.
This getting started guide allows you to collect local logs and metrics from your system, sending them to ClickStack for visualization and analysis.

**This example works on OSX and Linux systems only**

The following example assumes you have started ClickStack using the [instructions for the all-in-one image](/use-cases/observability/clickstack/getting-started) and connected to the [local ClickHouse instance](/use-cases/observability/clickstack/getting-started#complete-connection-credentials) or a [ClickHouse Cloud instance](/use-cases/observability/clickstack/getting-started#create-a-cloud-connection).

:::note HyperDX in ClickHouse Cloud
This sample dataset can also be used with HyperDX in ClickHouse Cloud, with only minor adjustments to the flow as noted. If using HyperDX in ClickHouse Cloud, users will require an Open Telemetry collector to be running locally as described in the [getting started guide for this deployment model](/use-cases/observability/clickstack/deployment/hyperdx-clickhouse-cloud).
This sample dataset can also be used with HyperDX in ClickHouse Cloud, with only minor adjustments to the flow as noted. If using HyperDX in ClickHouse Cloud, users will require an OpenTelemetry collector to be running locally as described in the [getting started guide for this deployment model](/use-cases/observability/clickstack/deployment/hyperdx-clickhouse-cloud).

Check warning on line 23 in docs/use-cases/observability/clickstack/example-datasets/local-data.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.FutureTense 

Instead of future tense 'will require', use present tense.
:::

<VerticalStepper>

## Navigate to the HyperDX UI {#navigate-to-the-hyperdx-ui}

Visit [http://localhost:8080](http://localhost:8080) to access the HyperDX UI if deploying locally. If using HyperDX in ClickHouse Cloud, select your service and `HyperDX` from the left menu.
## Create a custom OpenTelemetry configuration {#create-otel-configuration}

## Copy ingestion API key {#copy-ingestion-api-key}

:::note HyperDX in ClickHouse Cloud
This step is not required if using HyperDX in ClickHouse Cloud.
:::

Navigate to [`Team Settings`](http://localhost:8080/team) and copy the `Ingestion API Key` from the `API Keys` section. This API key ensures data ingestion through the OpenTelemetry collector is secure.

<Image img={copy_api_key} alt="Copy API key" size="lg"/>

## Create a local OpenTelemetry configuration {#create-otel-configuration}

Create a `otel-local-file-collector.yaml` file with the following content.

**Important**: Populate the value `<YOUR_INGESTION_API_KEY>` with your ingestion API key copied above (not required for HyperDX in ClickHouse Cloud).
Create a `custom-local-config.yaml` file with the following content:

```yaml
receivers:
filelog:
include:
- /var/log/**/*.log # Linux
- /var/log/syslog
- /var/log/messages
- /private/var/log/*.log # macOS
- /tmp/all_events.log # macos - see below
start_at: beginning # modify to collect new files only
- /host/var/log/**/*.log # Linux logs from host
- /host/var/log/syslog
- /host/var/log/messages
- /host/private/var/log/*.log # macOS logs from host
start_at: beginning
resource:
service.name: "system-logs"

hostmetrics:
collection_interval: 1s
Expand Down Expand Up @@ -96,29 +75,25 @@
network:
processes:

exporters:
otlp:
endpoint: localhost:4317
headers:
authorization: <YOUR_INGESTION_API_KEY>
tls:
insecure: true
sending_queue:
enabled: true
num_consumers: 10
queue_size: 262144 # 262,144 items × ~8 KB per item ≈ 2 GB

service:
pipelines:
logs:
logs/local:
receivers: [filelog]
exporters: [otlp]
metrics:
processors:
- memory_limiter
- batch
exporters:
- clickhouse
metrics/hostmetrics:
receivers: [hostmetrics]
exporters: [otlp]
processors:
- memory_limiter
- batch
exporters:
- clickhouse
```

This configuration collects system logs and metric for OSX and Linux systems, sending the results to ClickStack via the OTLP endpoint on port 4317.
This configuration collects system logs and metrics for OSX and Linux systems, sending the results to ClickStack. The configuration extends the ClickStack collector by adding new receivers and pipelines—you reference the existing `clickhouse` exporter and processors (`memory_limiter`, `batch`) that are already configured in the base ClickStack collector.

:::note Ingestion timestamps
This configuration adjusts timestamps at ingest, assigning an updated time value to each event. Users should ideally [preprocess or parse timestamps](/use-cases/observability/clickstack/ingesting-data/otel-collector#processing-filtering-transforming-enriching) using OTel processors or operators in their log files to ensure accurate event time is retained.
Expand All @@ -130,33 +105,53 @@

For more details on the OpenTelemetry (OTel) configuration structure, we recommend [the official guide](https://opentelemetry.io/docs/collector/configuration/).

:::note Detailed logs for OSX
Users wanting more detailed logs on OSX can run the command `log stream --debug --style ndjson >> /tmp/all_events.log` before starting the collector below. This will capture detailed operating system logs to the file `/tmp/all_events.log`, already included in the above configuration.
:::

## Start the collector {#start-the-collector}
## Start ClickStack with custom configuration {#start-clickstack}

Run the following docker command to start an instance of the OTel collector.
Run the following docker command to start the all-in-one container with your custom configuration:

```shell
docker run --network=host --rm -it \
docker run -d --name clickstack \
-p 8080:8080 -p 4317:4317 -p 4318:4318 \
--user 0:0 \
-v "$(pwd)/otel-local-file-collector.yaml":/etc/otel/config.yaml \
-v /var/log:/var/log:ro \
-v /private/var/log:/private/var/log:ro \
otel/opentelemetry-collector-contrib:latest \
--config /etc/otel/config.yaml
-e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
-v "$(pwd)/custom-local-config.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
-v /var/log:/host/var/log:ro \
-v /private/var/log:/host/private/var/log:ro \
docker.hyperdx.io/hyperdx/hyperdx-all-in-one:latest
```

:::note Root user
We run the collector as the root user to access all system logs—this is necessary to capture logs from protected paths on Linux-based systems. However, this approach is not recommended for production. In production environments, the OpenTelemetry Collector should be deployed as a local agent with only the minimal permissions required to access the intended log sources.

Note that we mount the host's `/var/log` to `/host/var/log` inside the container to avoid conflicts with the container's own log files.
:::

If using HyperDX in ClickHouse Cloud with a standalone collector, use this command instead:

```shell
docker run -d \
-p 4317:4317 -p 4318:4318 \
--user 0:0 \
-e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
-e OPAMP_SERVER_URL=${OPAMP_SERVER_URL} \
-e CLICKHOUSE_ENDPOINT=${CLICKHOUSE_ENDPOINT} \
-e CLICKHOUSE_USER=${CLICKHOUSE_USER} \
-e CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD} \
-v "$(pwd)/custom-local-config.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
-v /var/log:/host/var/log:ro \
-v /private/var/log:/host/private/var/log:ro \
docker.hyperdx.io/hyperdx/hyperdx-otel-collector
```

The collector will immediately begin collecting local system logs and metrics.

## Navigate to the HyperDX UI {#navigate-to-the-hyperdx-ui}

Visit [http://localhost:8080](http://localhost:8080) to access the HyperDX UI if deploying locally. If using HyperDX in ClickHouse Cloud, select your service and `HyperDX` from the left menu.

## Explore system logs {#explore-system-logs}

Navigate to the HyperDX UI. The search UI should be populated with local system logs. Expand the filters to select the `system.log`:
The search UI should be populated with local system logs. Expand the filters to select the `system.log`:

<Image img={hyperdx_20} alt="HyperDX Local logs" size="lg"/>

Expand All @@ -180,4 +175,4 @@

<Image img={hyperdx_23} alt="Memory % of time" size="lg"/>

</VerticalStepper>
</VerticalStepper>
95 changes: 91 additions & 4 deletions docs/use-cases/observability/clickstack/ingesting-data/collector.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,9 +100,96 @@ With Docker Compose, modify the collector configuration using the same environme

### Advanced configuration {#advanced-configuration}

Currently, the ClickStack distribution of the OTel collector does not support modification of its configuration file. If you need a more complex configuration e.g. [configuring TLS](#securing-the-collector), or modifying the batch size, we recommend copying and modifying the [default configuration](https://github.com/hyperdxio/hyperdx/blob/main/docker/otel-collector/config.yaml) and deploying your own version of the OTel collector using the ClickHouse exporter documented [here](/observability/integrating-opentelemetry#exporting-to-clickhouse) and [here](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/clickhouseexporter/README.md#configuration-options).
The ClickStack distribution of the OTel collector supports extending the base configuration by mounting a custom configuration file and setting an environment variable. The custom configuration is merged with the base configuration managed by HyperDX via OpAMP.

The default ClickStack configuration for the OpenTelemetry (OTel) collector can be found [here](https://github.com/hyperdxio/hyperdx/blob/main/docker/otel-collector/config.yaml).
#### Extending the collector configuration {#extending-collector-config}

To add custom receivers, processors, or pipelines:

1. Create a custom configuration file with your additional configuration
2. Mount the file at `/etc/otelcol-contrib/custom.config.yaml`
3. Set the environment variable `CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml`

**Example custom configuration:**

```yaml
receivers:
# Collect logs from local files
filelog:
include:
- /var/log/**/*.log
- /var/log/syslog
- /var/log/messages
start_at: beginning

# Collect host system metrics
hostmetrics:
collection_interval: 30s
scrapers:
cpu:
metrics:
system.cpu.utilization:
enabled: true
memory:
metrics:
system.memory.utilization:
enabled: true
disk:
network:
filesystem:
metrics:
system.filesystem.utilization:
enabled: true

service:
pipelines:
# Logs pipeline
logs/host:
receivers: [filelog]
processors:
- memory_limiter
- transform
- batch
exporters:
- clickhouse

# Metrics pipeline
metrics/hostmetrics:
receivers: [hostmetrics]
processors:
- memory_limiter
- batch
exporters:
- clickhouse
```

**Deploy with the all-in-one image:**
```bash
docker run -d --name clickstack \
-p 8080:8080 -p 4317:4317 -p 4318:4318 \
-e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
-v "$(pwd)/custom-config.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
docker.hyperdx.io/hyperdx/hyperdx-all-in-one:latest
```

**Deploy with the standalone collector:**
```bash
docker run -d \
-e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
-e OPAMP_SERVER_URL=${OPAMP_SERVER_URL} \
-e CLICKHOUSE_ENDPOINT=${CLICKHOUSE_ENDPOINT} \
-e CLICKHOUSE_USER=default \
-e CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD} \
-v "$(pwd)/custom-config.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
-p 4317:4317 -p 4318:4318 \
docker.hyperdx.io/hyperdx/hyperdx-otel-collector
```

:::note
You only define new receivers, processors, and pipelines in the custom config. The base processors (`memory_limiter`, `batch`) and exporters (`clickhouse`) are already defined—reference them by name. The custom configuration is merged with the base configuration and cannot override existing components.
:::

For more complex configurations, refer to the [default ClickStack collector configuration](https://github.com/hyperdxio/hyperdx/blob/main/docker/otel-collector/config.yaml) and the [ClickHouse exporter documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/clickhouseexporter/README.md#configuration-options).
Copy link
Collaborator

@gingerwizard gingerwizard Nov 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all looks good.


#### Configuration structure {#configuration-structure}

Expand All @@ -120,7 +207,7 @@ To further secure your deployment, we recommend:

- Configuring the collector to communicate with ClickHouse over HTTPS.
- Create a dedicated user for ingestion with limited permissions - see below.
- Enabling TLS for the OTLP endpoint, ensuring encrypted communication between SDKs/agents and the collector. **Currently, this requires users to deploy a default distribution of the collector and manage the configuration themselves**.
- Enabling TLS for the OTLP endpoint, ensuring encrypted communication between SDKs/agents and the collector. This can be configured via [custom collector configuration](#extending-collector-config).

### Creating an ingestion user {#creating-an-ingestion-user}

Expand Down Expand Up @@ -276,7 +363,7 @@ However, if you require high delivery guarantees or the ability to replay data (
In this case, OTel agents can be configured to send data to Kafka via the [Kafka exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/kafkaexporter/README.md). Gateway instances, in turn, consume messages using the [Kafka receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kafkareceiver/README.md). We recommend the Confluent and OTel documentation for further details.

:::note OTel collector configuration
The ClickStack OpenTelemetry collector distribution cannot be used with Kafka as it requires a configuration modification. Users will need to deploy a default OTel collector using the ClickHouse exporter.
The ClickStack OpenTelemetry collector distribution can be configured with Kafka using [custom collector configuration](#extending-collector-config).
:::

## Estimating resources {#estimating-resources}
Expand Down