Debugging

See Also:

Debugging / Performance Tools

• List last reboots

    last reboot
  

• Reboot when /sbin is unusable

    echo b >/proc/sysrq-trigger
  

• List Context Switches per Process

    pidstat -w
  

• Drop Filesystem Cache

    echo 1 > /proc/sys/vm/drop_caches
  

• Disk Performance Test

    hdparm -tT /dev/...
  

• dmesg - block IO debugging:

    echo 1 > /proc/sys/vm/block_dump
  
    # wait some time...
    echo 0 > /proc/sys/vm/block_dump
  
    # Now check syslog for block dump lines
  

• Check for changed sysctl() settings:

    sysctl -p
  

• Tuning sysctl for busy large server

• TCP/IP tuning for scaling

• Increase max connection settings

    sysctl -w net.core.somaxconn=1024
    sysctl -w net.core.netdev_max_backlog=16000
  

• SYN flood protection: Count connections

    netstat -n -p TCP | grep SYN_RECV | grep $port | wc -l
    netstat -s -P tcp | grep tcpHalfOpenDrop
  

  and change settings

    sysctl -w net.ipv4.tcp_syncookies=1     # Set enabled
    sysctl -w net.ipv4.tcp_max_syn_backlog=8192 # Increase as needed
    sysctl -w net.ipv4.tcp_fin_timeout=30       # Lower timeout as needed
  

• HTP Support Docs:

    sysctl -a | grep hugepage
    cat /proc/meminfo | grep HugePage
  

• Enable OOM killer:

    sysctl vm.overcommit_memory=0
    sysctl vm.oom_kill_allocating_task=1
  

• Disable OOM killer:

    sysctl vm.overcommit_memory=2
  

• dmesg - Filtering Output:

    dmesg -T      # Enable human readable timestamps
    dmesg -x      # Show facility and log level
    dmesg -f daemon     # Filter for facility daemon
    dmesg -l err,crit,alert,emerg # Filter for errors
  

• lslk - Find file locks: Use lslk to find which PID is blocking an flock() to a file.

• lsof - Find owners of open file handles:

    lsof      # Complete list
    lsof -i :22    # Filter single TCP port
    lsof [email protected]:22 # Filter single connection endpoint
    lsof -u <user>   # Filter per user
    lsof -c <name>   # Filter per process name
    lsof -p 12345    # Filter by PID
    lsof /etc/hosts   # Filter single file
  

• Perf Tutorial: 2.6+ generic kernel performance statistics tool.

    perf stat -B some_command
  

• dstat: Replaces vmstat, iostat, netstat and ifstat and allows to determine PID that is most CPU and most I/O expensive

    dstat -a --top-bio --top-cpu
  

• iotop: Python script to monitor I/O like top

• Sysdig: Some of the project examples

    sysdig fd.name contains /etc
    sysdig -c topscalls_time    # Top system calls
    sysdig -c topfiles_time proc.name=httpd    # Top files by process
    sysdig -c topfiles_bytes     # Top I/O per file
    sysdig -c fdcount_by fd.cip "evt.type=accept"   # Top connections by IP
    sysdig -c fdbytes_by fd.cip  # Top bytes per IP
  
    # Monitor directory
    sysdig evt.type=open and fd.name contains /var/
  
    # Monitor latency >1ms
    sysdig -c fileslower 1
  
    # Track I/O of a process
    sysdig -A -c echo_fds proc.name=httpd
  
    # Sick MySQL check via Apache
    sysdig -A -c echo_fds fd.sip=192.168.30.5 and proc.name=apache2 and evt.buffer contains SELECT
  
    sysdig -cl # List plugins
    sysdig -c bottlenecks  # Run bottlenecks plugin
  
    # To install sysdig
    curl -s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash