Libraries for Linux Tracepoints and user_events

This repository contains C and C++ libraries and tools for collecting and decoding Linux Tracepoint events and for generating Tracepoint events from user mode using the user_events facility. This includes support for eventheader events and perf.data files.

Related repositories:

• LinuxTracepoints-Net - .NET libraries and tools for decoding Linux Tracepoint events and for generating Tracepoint events from user mode using the user_events facility. This includes support for eventheader events and perf.data files.
• LinuxTracepoints-Rust - Rust libraries for generating Tracepoint events from user mode using the user_eventsfacility. This includes support for eventheader events.

Overview

• libtracepoint - low-level C/C++ tracing interface. Designed to support replacement at link-time if a different implementation is needed (e.g. for testing).

  • tracepoint-provider.h - a developer-friendly C/C++ API for writing tracepoint events to any implementation of the tracepoint.h interface.
  • tracepoint.h - low-level interface for writing tracepoint events.
  • libtracepoint.a - default implementation that writes directly to the Linux user_events facility.

• libtracepoint-control-cpp - C++ library for controlling a tracepoint event collection session.

  • perf-collect is a tool that collects tracepoint events into a perf.data file using the libtracepoint-control-cpp library. This tool is similar to the perf record command, but it includes special "pre-register" support to simplify collection of user_events tracepoints that are not yet registered when trace collection begins.
  • TracepointSession.h implements an event collection session that can collect tracepoint events and enumerate the events that the session has collected.
  • TracepointPath.h has functions for finding the /sys/kernel/tracing mount point and reading format files.
  • TracepointName.h represents a tracepoint name (system name + event name); for instance, user_events:eventName.
  • TracepointSpec.h represents a tracepoint specification (system name + event name + field definitions); for instance, user_events:eventName int field1.
  • TracepointCache.h implements a cache for tracking parsed format files and locating cached data by TracepointName or by common_type id.

• libtracepoint-decode-cpp - C++ library for decoding tracepoints and reading/writing perf.data files. Works on both Linux and Windows.

  • PerfDataFile.h defines the PerfDataFile class that decodes perf.data files.
  • PerfEventInfo.h defines the PerfSampleEventInfo and PerfNonSampleEventInfo structures for raw event information.
  • PerfEventMetadata.h defines classes for parsing ftrace event metadata information.

• libeventheader-tracepoint - eventheader envelope that supports extended attributes including severity level and optional field information (field types and field names).

  • TraceLoggingProvider.h - a developer-friendly C/C++ API for writing eventheader-encapsulated events to any implementation of the tracepoint interface.
  • EventHeaderDynamic.h - C++ API for writing runtime-defined eventheader-encapsulated events, intended for use as an implementation layer for a higher-level API like OpenTelemetry.

• libeventheader-decode-cpp - C++ library for decoding events that use the eventheader envelope. Works on both Linux and Windows.

  • EventEnumerator class parses an event into fields.
  • EventFormatter class converts event data into a string.
  • perf-decode tool that decodes perf.data files to JSON with support for eventheader events.

General Usage

• Configure a Linux system with the user_events feature enabled.

  • Supported on Linux kernel 6.4 and later.
  • Kernel must be built with user_events support (CONFIG_USER_EVENTS=y).
  • Must have either tracefs or debugfs mounted. For example, you might add the following line to your /etc/fstab file: tracefs /sys/kernel/tracing tracefs defaults 0 0
  • The user that will generate events must have x access to the tracing directory and w access to the tracing/user_events_data file. One possible implementation is to create a tracers group, then:
    • chgrp tracers /sys/kernel/tracing
    • chgrp tracers /sys/kernel/tracing/user_events_data
    • chmod g+x /sys/kernel/tracing
    • chmod g+w /sys/kernel/tracing/user_events_data

• Use one of the event generation APIs to write a program that generates events.

  • C/C++ programs can use tracepoint-provider.h to generate regular Linux Tracepoint events that are defined at compile-time. (Link with libtracepoint.)
  • C/C++ programs can use TraceLoggingProvider.h to generate eventheader-enabled Tracepoint events that are defined at compile-time. (Link with libtracepoint and libeventheader-tracepoint.)
  • C++ middle-layer APIs (e.g. an OpenTelemetry exporter) can use EventHeaderDynamic.h to generate eventheader-enabled Tracepoint events that are runtime-dynamic. (Link with libtracepoint and libeventheader-tracepoint.)
  • Rust or .NET programs can use LinuxTracepoints-Rust or LinuxTracepoints-Net to generate Tracepoint events.

• To collect events in a C++ program, use libtracepoint-control-cpp. Note that your program must run as a privileged user (CAP_PERFMON capability plus read access to /sys/kernel/tracing/events) because access to the event collection system is restricted by default.

• To collect events without writing C++ code, use the included perf-collect tool or the Linux perf tool to collect events to a perf.data file, e.g. perf-collect -o File.perf user_events:MyEvent1 user_events:MyEvent2 or perf record -o File.perf -k monotonic -e user_events:MyEvent1,user_events:MyEvent2. Note that you must run the tool as a privileged user to collect events (CAP_PERFMON capability plus read access to /sys/kernel/tracing/events).

  • The perf tool binary is typically available as part of the linux-perf package (e.g. can be installed by apt install linux-perf). However, this package installs a perf_VERSION binary rather than a perf binary, so you will need to add an appropriate VERSION suffix to your perf commands or use a wrapper script.
  • To capture tracepoints using perf, you'll also need to install libtraceevent, e.g. apt install libtraceevent1.
  • The linux-base package installs a perf wrapper script that redirects to the version of perf that matches your current kernel (if present) so that you can run the appropriate version of perf without the VERSION suffix. This frequently doesn't work because the latest perf binary from apt doesn't always match the running kernel, so you may want to make your own wrapper script instead.
  • Note that for purposes of collecting events, it is usually not important for the version of the perf tool to match the kernel version, so it's ok to use e.g. perf_5.10 even if you are running a newer kernel.

• Note that tracepoints must be registered before you can start collecting them. The perf-collect tool has facilities to pre-register a user_events tracepoint. The perf command will report an error if the tracepoint is not yet registered.

  • You can usually register tracepoints by starting the program that generates them. Most programs will register all of their tracepoints when they start running. (They will usually unregister when they stop running.)
  • You can also use the tracepoint-register tool to pre-register an event so you can start collecting it before starting the program that generates it.
  • If writing your own event collection tool, you might do something similar in your tool to pre-register the events that you need to collect. For example, you might use the PreregisterTracepoint or PreregisterEventHeaderTracepoint methods of the TracepointCache class in libtracepoint=control.

• Use the perf-decode tool to decode the perf.data file to JSON text, or write your own decoding tool using libtracepoint-decode-cpp and libeventheader-decode-cpp.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.