Technotes for future me


See Also:

Debugging / Performance Tools

  • List last reboots

      last reboot
  • Reboot when /sbin is unusable

      echo b >/proc/sysrq-trigger
  • List Context Switches per Process

      pidstat -w
  • Drop Filesystem Cache

      echo 1 > /proc/sys/vm/drop_caches
  • Disk Performance Test

      hdparm -tT /dev/...
  • dmesg - block IO debugging:

      echo 1 > /proc/sys/vm/block_dump
      # wait some time...
      echo 0 > /proc/sys/vm/block_dump
      # Now check syslog for block dump lines
  • Check for changed sysctl() settings:

      sysctl -p
  • Tuning sysctl for busy large server

  • TCP/IP tuning for scaling

  • Increase max connection settings

      sysctl -w net.core.somaxconn=1024
      sysctl -w net.core.netdev_max_backlog=16000
  • SYN flood protection: Count connections

      netstat -n -p TCP | grep SYN_RECV | grep $port | wc -l
      netstat -s -P tcp | grep tcpHalfOpenDrop

    and change settings

      sysctl -w net.ipv4.tcp_syncookies=1     # Set enabled
      sysctl -w net.ipv4.tcp_max_syn_backlog=8192 # Increase as needed
      sysctl -w net.ipv4.tcp_fin_timeout=30       # Lower timeout as needed
  • HTP Support Docs:

      sysctl -a | grep hugepage
      cat /proc/meminfo | grep HugePage
  • Enable OOM killer:

      sysctl vm.overcommit_memory=0
      sysctl vm.oom_kill_allocating_task=1
  • Disable OOM killer:

      sysctl vm.overcommit_memory=2
  • dmesg - Filtering Output:

      dmesg -T      # Enable human readable timestamps
      dmesg -x      # Show facility and log level
      dmesg -f daemon     # Filter for facility daemon
      dmesg -l err,crit,alert,emerg # Filter for errors
  • lslk - Find file locks: Use lslk to find which PID is blocking an flock() to a file.

  • lsof - Find owners of open file handles:

      lsof      # Complete list
      lsof -i :22    # Filter single TCP port
      lsof [email protected]:22 # Filter single connection endpoint
      lsof -u <user>   # Filter per user
      lsof -c <name>   # Filter per process name
      lsof -p 12345    # Filter by PID
      lsof /etc/hosts   # Filter single file
  • Perf Tutorial: 2.6+ generic kernel performance statistics tool.

      perf stat -B some_command
  • dstat: Replaces vmstat, iostat, netstat and ifstat and allows to determine PID that is most CPU and most I/O expensive

      dstat -a --top-bio --top-cpu
  • iotop: Python script to monitor I/O like top

  • Sysdig: Some of the project examples

      sysdig contains /etc
      sysdig -c topscalls_time    # Top system calls
      sysdig -c topfiles_time    # Top files by process
      sysdig -c topfiles_bytes     # Top I/O per file
      sysdig -c fdcount_by fd.cip "evt.type=accept"   # Top connections by IP
      sysdig -c fdbytes_by fd.cip  # Top bytes per IP
      # Monitor directory
      sysdig evt.type=open and contains /var/
      # Monitor latency >1ms
      sysdig -c fileslower 1
      # Track I/O of a process
      sysdig -A -c echo_fds
      # Sick MySQL check via Apache
      sysdig -A -c echo_fds fd.sip= and and evt.buffer contains SELECT
      sysdig -cl # List plugins
      sysdig -c bottlenecks  # Run bottlenecks plugin
      # To install sysdig
      curl -s | sudo bash
Last updated on 31 Jan 2021
Published on 25 Dec 2019
Edit on GitHub