Zabbix History Syncer Monitor

A Go application that monitors Zabbix history syncer processes using either polling or eBPF uprobes, reporting real-time performance metrics.

Features

• Two monitoring modes:
  • Polling mode: Scans /proc/[pid]/cmdline files at regular intervals
  • eBPF mode: Uses uprobes on zbx_setproctitle function for event-driven monitoring (requires root)
• Parses and reports:
  • Processed values
  • Triggers processed
  • Processing time
  • Values per second (calculated rate)
  • Triggers per second (calculated rate)
• Supports both JSON and human-readable output formats

Building

Polling Mode (simple)

# Using Makefile (creates static binary)
make

# Or manually
go build -o zabbix-monitor

eBPF Mode (requires additional dependencies)

# Install dependencies (Ubuntu/Debian)
sudo apt-get install clang llvm libbpf-dev

# Generate eBPF bytecode and build
make ebpf

# Or manually
go generate ./...
go build -o zabbix-monitor

Usage

Polling Mode

# Basic usage with default settings (2 second polling interval)
./zabbix-monitor

# Custom polling interval
./zabbix-monitor -interval 5s

# JSON output format
./zabbix-monitor -json -interval 1s

eBPF Mode (recommended, requires root)

# Basic eBPF monitoring
sudo ./zabbix-monitor -ebpf

# With custom binary path
sudo ./zabbix-monitor -ebpf -binary /usr/local/sbin/zabbix_server

# JSON output
sudo ./zabbix-monitor -ebpf -json

# Verbose logging
sudo ./zabbix-monitor -ebpf -verbose

Command-Line Flags

• -ebpf: Use eBPF uprobes instead of polling (requires root privileges)
• -binary string: Path to zabbix_server binary (default: /usr/sbin/zabbix_server, eBPF mode only)
• -interval duration: Polling interval (default: 2s, polling mode only). Examples: 1s, 500ms, 5s
• -json: Output in JSON format instead of human-readable format
• -verbose: Enable verbose logging to stderr

Output Examples

Human-readable format (default)

The application displays individual syncer metrics as they update, followed by aggregate totals:

[14:32:15] Syncer #49 (PID 20720): 10181 values, 3451 triggers in 1.328s | 7667.92 values/sec, 2598.94 triggers/sec
[14:32:15] Syncer #50 (PID 20721): 9534 values, 3102 triggers in 1.201s | 7939.22 values/sec, 2583.68 triggers/sec
[14:32:15] TOTAL (2 syncers): 19715 values, 6553 triggers | 15607.14 values/sec, 5182.62 triggers/sec

[14:32:17] Syncer #49 (PID 20720): 9823 values, 3302 triggers in 1.251s | 7852.52 values/sec, 2639.49 triggers/sec
[14:32:17] TOTAL (2 syncers): 19357 values, 6404 triggers | 15791.74 values/sec, 5223.17 triggers/sec

Aggregate Statistics:

• After each polling interval (or every 2 seconds in eBPF mode), aggregate totals are displayed
• Aggregates use the latest metrics from each syncer
• Shows total values, total triggers, and summed rates across all active syncers

JSON format

{"pid":20720,"syncer_number":49,"processed_values":10181,"triggers":3451,"time_seconds":1.327717,"values_per_second":7667.92,"triggers_per_second":2598.94}
{"pid":20721,"syncer_number":50,"processed_values":9534,"triggers":3102,"time_seconds":1.201,"values_per_second":7939.22,"triggers_per_second":2583.68}

Note: JSON output shows only individual syncer metrics (no aggregate totals)

How It Works

Polling Mode

1. Initial Scan: Scans /proc/[0-9]*/cmdline for processes matching zabbix_server: history syncer
2. Monitoring: Polls the cmdline file of each found process at the specified interval
3. Change Detection: Compares current cmdline with last known value
4. Parsing: Extracts metrics using regex from format: history syncer #N [processed X values, Y triggers in Z.ZZZ sec
5. Rate Calculation: Computes values/sec = X / Z and triggers/sec = Y / Z
6. Output: Reports changes to stdout in selected format

eBPF Mode (recommended)

1. eBPF Uprobe Attachment: Attaches a uprobe to the zbx_setproctitle function in the Zabbix binary
2. Event Capture: Captures every call to zbx_setproctitle in real-time (no polling overhead)
3. Parsing: Extracts the format string argument and parses metrics
4. Rate Calculation: Computes values/sec = X / Z and triggers/sec = Y / Z
5. Output: Reports changes to stdout in selected format

Advantages of eBPF mode:

• Real-time event-driven monitoring (no polling delay)
• Lower CPU overhead (no periodic scanning)
• Captures all process title changes immediately
• Works for all Zabbix processes, not just history syncers

Requirements

Common Requirements

• Linux kernel 4.18+ (for eBPF mode)
• Go 1.21 or later
• Running Zabbix server with history syncer processes

eBPF Mode Additional Requirements

• Root privileges (CAP_BPF or CAP_SYS_ADMIN)
• clang and llvm for compiling eBPF programs
• Linux headers matching your kernel version
• Zabbix binary with debug symbols (or known function symbols)

Notes

Polling Mode

• The application monitors only processes found during the initial scan
• If a monitored process dies, errors will be logged (with -verbose flag)
• Rates are calculated based on the time reported in the cmdline

eBPF Mode

• Requires root privileges to load eBPF programs
• Automatically monitors all Zabbix processes (existing and new ones)
• The uprobe attaches to the function symbol zbx_setproctitle
• If the binary is stripped, you may need to rebuild Zabbix with symbols