Linux Performance Tuning
Linux performance tuning is essential for running efficient production workloads. Understanding how the kernel manages CPU, memory, disk, and network resources allows you to identify bottlenecks and optimize accordingly.
The USE Method
Brendan Gregg's USE (Utilization, Saturation, Errors) method provides a systematic approach to performance analysis:
* **Utilization**: What percentage of the resource is busy?
* **Saturation**: How much extra work is queued?
* **Errors**: How many error events are there?
Apply this to CPU, memory, storage, and network resources to quickly identify the bottleneck.
CPU Performance Tuning
Monitoring Tools
# Real-time CPU monitoring
htop
# Per-process CPU usage
top -o %CPU
# CPU statistics and context switches
vmstat 1 5
# Detailed per-CPU utilization
mpstat -P ALL 1
High context switch rates (above 50,000 per second per core) may indicate inefficient application architecture. Use `pidstat -w` to identify processes causing excessive context switches.
Kernel Parameters
# cat /etc/sysctl.d/99-performance.conf
kernel.sched_min_granularity_ns = 3000000
kernel.sched_wakeup_granularity_ns = 4000000
kernel.sched_migration_cost_ns = 500000
kernel.sched_nr_migrate = 32
These scheduler parameters reduce latency for interactive workloads. Adjust carefully -- aggressive settings can hurt throughput.
Memory Tuning
Monitoring Memory
# Memory usage overview
free -h
# Detailed memory breakdown
cat /proc/meminfo
# Page fault statistics
sar -B 1
# Top memory consumers
ps aux --sort=-%mem | head
Check `sar -B` for page fault rates. High `pgmajfault` values indicate the system is swapping -- add more RAM or reduce memory pressure.
Swappiness
# Reduce swapping tendency (default is 60)
vm.swappiness = 10
# Set temporarily
sysctl vm.swappiness=10
For database servers, set swappiness to 1 to avoid swapping. For desktops and general-purpose servers, 10-20 balances responsiveness with memory efficiency.
Transparent Huge Pages
Disable THP for database workloads:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
THP can cause latency spikes in database systems due to memory defragmentation pauses.
Disk I/O Tuning
I/O Scheduler
Choose the right I/O scheduler for your workload:
# Check current scheduler
cat /sys/block/nvme0n1/queue/scheduler
# Set to none for NVMe, mq-deadline for spinning disks
echo none > /sys/block/nvme0n1/queue/scheduler
Modern NVMe drives perform best with the `none` (or `nvme`) scheduler. Spinning disks benefit from `mq-deadline` which minimizes seek times.
Monitoring Disk Performance
# I/O statistics per device
iostat -x 1
# Process-level I/O
iotop
# File system latency
bcc-tools/biolatency
High `await` times (above 20ms) indicate disk saturation. Check `iowait` in `top` and `svctm` in `iostat` for confirmation.
Network Tuning
Kernel Network Parameters
# /etc/sysctl.d/99-network.conf
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 50000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 5
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
Increase socket buffer sizes for high-throughput applications. `tcp_tw_reuse` allows reuse of connections in TIME_WAIT state, important for high-connection-rate servers.
Network Monitoring
# Per-interface statistics
sar -n DEV 1
# Socket statistics
ss -s
# Connection tracking
netstat -s | grep -i "connections established"
Monitor for dropped packets in `/proc/net/softnet_stat` and TCP retransmits with `netstat -s | grep retransmit`.
File System Tuning
Mount Options
Optimize filesystem mount options for performance:
# /etc/fstab
/dev/sda1 / ext4 noatime,nodiratime,data=ordered 0 0
The `noatime` option eliminates update timestamps on every file read, significantly reducing disk writes. Use `nodiratime` for directories.
I/O Limits with cgroups
Control per-process I/O with cgroups v2:
# Limit read/write to 50 MB/s
echo "8:0 rbps=52428800 wbps=52428800" > /sys/fs/cgroup/
Application-Level Tuning
Profile before tuning. Use `perf` for CPU profiling, `flamegraphs` for visualization, and `strace` for system call analysis:
# Sample CPU stacks at 99Hz
perf record -F 99 -a -g -- sleep 30
perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > profile.svg
Always measure before and after changes. A single benchmark run is unreliable -- run multiple iterations and report the median.
Summary
Linux performance tuning is a systematic process of identifying bottlenecks, applying targeted optimizations, and measuring the impact. Start with the USE method to find the bottleneck, use the right monitoring tools, and tune one parameter at a time. Common wins include setting the correct I/O scheduler, reducing swappiness for databases, disabling THP, and increasing network buffer sizes. Profile first, tune second, and always validate with benchmarks.