Technotes for future me

Kernel Tracing


  • Kernel Trace Systems Overview

  • Core Dumps with Ubuntu: Ubuntu per-default redirects cores to its bug reporting tool “apport” so you won’t get core dumps. If you still need to overrule the core dump pipe in /proc/sys/kernel/core_pattern:

    echo core >/proc/sys/kernel/core_pattern 
  • Dtrace for Linux: the Solaris framework ported by Sun/Oracle

  • FTrace: kernel trace point injection

  • Task Stats: per-task accounting in Linux

  • ktap: Runs with just a kernel module without debugging symbols, requires debugfs

  • FlameGraph: generate call stack volume graphs for SystemTap, DTrace, Perf…


Event Tracing


mount -t debugfs none /sys/kernel/debug

# Variant 1
echo "sched_wakeup"  >> /sys/kernel/debug/tracing/set_event
echo "!sched_wakeup" >> /sys/kernel/debug/tracing/set_event
echo "sched:*"        > /sys/kernel/debug/tracing/set_event

# Variant 2
echo 1 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable
echo 0 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable
echo 1 > /sys/kernel/debug/tracing/events/sched/enable

# Available events
find /sys/kernel/debug/tracing/events/sched

# To add trace points from boot loader, add
trace_event=<event name>

# Setting triggers: stacktrace, snapshot, traceon/off
# -> Always stracktrace
echo 'stacktrace' > /sys/kernel/debug/tracing/events/<event>/trigger

# -> Snapshot on next event
echo 'snapshot if nr_rq > 1' > /sys/kernel/debug/tracing/events/<event>/trigger

Perf Probe

apt-get install linux-tools-common linux-tools-generic

perf probe --line vfs_read
perf probe --add 'vfs_read file file->f_mode'
perf probe --list

# Trace Event
perf trace                         # Trace to terminal (useless because terminal feedback loop)
perf trace <command>               # Trace a command execution

perf record -e "syscalls:*" ls     # Record system calls on execution of "ls"
perf record -e "syscalls:*" -aR       # Record all processes until Ctrl-C is pressed

# Note: when "syscalls" does not exist try "raw_syscalls"

perf list                          # For list of possible -e filters

perf report -i

# Delete Event
perf probe --del '*'

Misc perf stuff


  • System Tap: Tracing framework by Red Hat, IBM, Hitachi, and Oracle.

  • System Tap on Debian: Installation HowTo

  • System Tap on Ubuntu: Installation HowTo, short version below

    apt-get install systemtap systemtap-doc
    # Install kernel debug symbols
    codename=$(lsb_release -c | awk  '{print $2}')
    sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
    deb ${codename}      main restricted universe multiverse
    deb ${codename}-security main restricted universe multiverse
    deb ${codename}-updates  main restricted universe multiverse
    deb ${codename}-proposed main restricted universe multiverse
    sudo apt-key adv --keyserver --recv-keys ECDCAD72428D7C01
    sudo apt-get update
    sudo apt-get install linux-image-$(uname -r)-dbgsym
    # List existing SystemTap scripts
    dpkg -L systemtap-doc | grep "\.stp$"
    # Run a script


statistical profiler for Linux (<20% overhead)

# After boot run "init" and "setup" with or without kernel statistics
opcontrol --init
opcontrol --setup --no-vmlinux  
opcontrol --setup --vmlinux=/usr/lib/debug/lib/modules/`uname -r`/vmlinux

opcontrol --reset
opcontrol --start
opcontrol --dump
opcontrol --shutdown

# Per library/image summaries
opreport /usr/sbin/sshd # Limit to this binary

# Call graph for one application
opreport -cl --demangle=smart /usr/sbin/apache2

# To resolve kernel symbols
opreport --symbols --image-path=/lib/modules/<version>/kernel
Last updated on 31 Jan 2021
Published on 25 Dec 2019
Edit on GitHub