Debugging Tools: lldb, gdb, strace, ltrace, rr Reverse Debugging

Introduction

Debugging is a core developer skill. Beyond print statements and basic breakpoints, advanced debugging tools provide deep insight into program behavior. This article covers lldb and gdb for native code debugging, strace for system call tracing, ltrace for library call analysis, and rr for the revolutionary capability of reverse execution.

lldb (LLVM Debugger)

The modern debugger from the LLVM project, preferred for Clang-compiled code:

# Start debugging

lldb ./myapp

lldb -p 12345 # Attach to process

lldb -c core.dump ./myapp # Analyze core dump

# Breakpoints

(lldb) breakpoint set --name main

(lldb) breakpoint set --file app.c --line 42

(lldb) breakpoint set --selector viewDidLoad

(lldb) breakpoint set --func-regex ".*alloc.*"

(lldb) breakpoint command add 1

> frame variable

> continue

> DONE

# Execution control

(lldb) run

(lldb) continue

(lldb) next # Step over

(lldb) step # Step into

(lldb) finish # Step out

# Variable inspection

(lldb) frame variable

(lldb) frame variable --regex ".*error.*"

(lldb) expression myVar

(lldb) expression self->name

(lldb) expression --objc -- print [NSString stringWithFormat:@"%@", myString]

# Thread and backtrace

(lldb) thread backtrace

(lldb) thread backtrace all

(lldb) thread select 2

# Watchpoints (break on data access)

(lldb) watchpoint set variable myVar

(lldb) watchpoint set expression -- myPointer[5]

gdb (GNU Debugger)

The traditional Unix debugger:

# Compile with debug symbols

gcc -g -O0 myapp.c -o myapp

# Start debugging

gdb ./myapp

gdb -p 12345

gdb ./myapp core

# Common commands

(gdb) break main

(gdb) break app.c:42 if x > 5

(gdb) run arg1 arg2

(gdb) bt # Backtrace

(gdb) info locals # Show local variables

(gdb) print x

(gdb) display x # Auto-display every stop

(gdb) watch y # Break when y changes

(gdb) x/20x ptr # Examine memory as hex

(gdb) disassemble # Show assembly

# Scripted debugging

(gdb) set pagination off

(gdb) set logging on

(gdb) commands

> print x

> continue

> end

# TUI mode (split view)

gdb -tui ./myapp

strace (System Call Trace)

Trace all system calls a program makes:

# Trace a command

strace -o trace.log ./myapp

strace -f ./myapp # Follow forks

# Common filters

strace -e open,read,write ./myapp # Specific syscalls

strace -e network ./myapp # Network-related calls

strace -e trace=file ./myapp # File operations

strace -e trace=process ./myapp # Process management

strace -e trace=signal ./myapp # Signal handling

# Performance analysis

strace -c ./myapp # Syscall count and time summary

strace -T ./myapp # Time spent in each syscall

strace -r ./myapp # Relative timestamps

# Useful patterns

strace -p 12345 -e write -s 1000 # See what a process writes

strace -f -e openat -p $(pgrep nginx) # File opens by nginx workers

strace -e read -s 10000 python3 app.py # Show read content

# PID tracking and timing

strace -fy -t -T -o trace.log -p 12345

**What to look for**: `ENOENT` (file not found), `EACCES` (permission denied), `ECONNREFUSED` (connection refused), slow syscalls (high `-T` values), unexpected file opens, excessive context switching.

ltrace (Library Call Trace)

Trace library function calls:

# Basic usage

ltrace ./myapp

ltrace -o libcalls.log ./myapp

# Filter specific libraries

ltrace -e malloc+free ./myapp # Memory allocation calls

ltrace -e str* ./myapp # String functions

ltrace -e 'libc.*' ./myapp # All libc functions

# Count library calls

ltrace -c ./myapp

# With timing

ltrace -T ./myapp # Time per call

ltrace -S ./myapp # Show syscalls too

# Suppress repetitive calls

ltrace -n 2 ./myapp # Indent by call depth

rr (Reverse Debugger)

Record and replay debugging with reverse execution:

# Record execution

rr record ./myapp

rr record -- ./myapp arg1 arg2

# Replay (deterministic)

rr replay

rr replay -p 12345

# Reverse debugging commands

(gdb) reverse-continue # Go back to most recent event

(gdb) reverse-step # Step back

(gdb) reverse-next # Step over back

(gdb) reverse-finish # Go back to function entry

# Find when a variable changed

(gdb) watch -l myVar

(gdb) reverse-continue # Stops when myVar last changed

# Go to specific event

rr replay -M 12345 # Replay to event number 12345

# Chaos mode (test concurrency)

rr record --chaos ./myapp # Randomize thread scheduling

**Key strength**: Debug intermittent bugs by recording once, then replaying infinitely. When you miss the bug, just restart replay and be more prepared. Chaos mode helps find concurrency bugs.

Debugging Workflow

# 1. Program crashes — get a core dump

ulimit -c unlimited

./myapp

gdb ./myapp core

# 2. Use strace to see what the program is doing

strace -f -o trace.log ./myapp

less trace.log # Look for error syscalls

# 3. For tricky bugs, use rr

rr record ./myapp

rr replay

# Now you can step forward AND backward

# 4. Library call issues

ltrace -c ./myapp # Which library calls are most frequent?

Comparison

| Tool | Best For | Overhead | Learning Curve |

|------|----------|----------|----------------|

| lldb | Native debugging (LLVM) | Low | Medium |

| gdb | Native debugging (GCC) | Low | High (many commands) |

| strace | System call tracing | Medium | Low |

| ltrace | Library call tracing | Medium | Low |

| rr | Reverse debugging | High (record) | Medium |

Recommendations

* **General debugging**: lldb for new projects, gdb for legacy code. Both support the same core debugging workflow.

* **File/permission issues**: strace is the fastest way to find "file not found" or "permission denied" problems.

* **Performance debugging**: strace -c shows where your program spends time in syscalls.

* **Intermittent bugs**: rr is revolutionary — record once, replay infinitely with reverse execution.

* **Library comprehension**: ltrace shows which library functions are called and how often.

Start with strace for quick diagnostics. Use lldb/gdb for step-through debugging. Deploy rr for your hardest bugs — the reverse execution capability makes "oops, I missed that" a thing of the past.