Linux Tracing Technologies Guide

Tracing in the Linux kernel is a powerful mechanism that allows developers and system administrators to analyze and debug system behavior. This guide provides documentation on various tracing frameworks and tools available in the Linux kernel.

Introduction to Tracing

This section provides an overview of Linux tracing mechanisms and debugging approaches.

• Using the tracer for debugging
• Using the Linux Kernel Tracepoints
• Notes on Analysing Behaviour Using Events and Tracepoints
• Tracefs ring-buffer memory mapping

Core Tracing Frameworks

The following are the primary tracing frameworks integrated into the Linux kernel.

• ftrace - Function Tracer
• Function Tracer Design
• Using ftrace to hook to functions
• Kernel Probes (Kprobes)
• Kprobe-based Event Tracing
• Fprobe-based Event Tracing
• Eprobe - Event-based Probe Tracing
• Fprobe - Function entry/exit probe
• Lockless Ring Buffer Design

Event Tracing and Analysis

A detailed explanation of event tracing mechanisms and their applications.

• Event Tracing
• Subsystem Trace Points: kmem
• Subsystem Trace Points: power
• NMI Trace Events
• MSR Trace Events
• Landlock Trace Events
• Subsystem Trace Points: PCI
• Subsystem Trace Points: PCI Controller
• Boot-time tracing
• Event Histograms
• Histogram Design Notes

Hardware and Performance Tracing

This section covers tracing features that monitor hardware interactions and system performance.

• Intel(R) Trace Hub (TH)
• System Trace Module
• MIPI SyS-T over STP
• CoreSight - ARM Hardware Trace
• Runtime Verification
• HiSilicon PCIe Tune and Trace device
• In-kernel memory-mapped I/O tracing
• Hardware Latency Detector
• OSNOISE Tracer
• Timerlat tracer

User-Space Tracing

These tools allow tracing user-space applications and interactions.

• user_events: User-based Event Tracing
• Uprobe-tracer: Uprobe-based Event Tracing

Remote Tracing

This section covers the framework to read compatible ring-buffers, written by entities outside of the kernel (most likely firmware or hypervisor)

• Tracing Remotes

Additional Resources

For more details, refer to the respective documentation of each tracing tool and framework.