Observability Tooling for the Machine Under Your Desk

The Cost of Blindly Tracing a Silent Build

A workstation fan surges. A supposedly trivial local build stops producing output. The terminal cursor blinks steadily, offering absolutely no useful explanation for the hang. Developers facing this scenario frequently reach for the most complex debugger available to inspect the compiler's internal state. Attaching a heavy tracer to a heavily threaded build process can increase execution time by a factor in the range of 10x to 50x depending on syscall frequency. The diagnostic tool itself alters the environment so severely that the original problem becomes impossible to reproduce.

Start with the cheapest probe that can answer the question. Polling with basic utilities introduces negligible overhead, consuming approximately 5 milliseconds of CPU time or less per invocation. This minimal interference preserves the state of the stalled application while providing immediate clues about its behavior.

Here is the pragmatic decision matrix for a stalled process. Use ps or top to identify the process and its current state. Reach for strace to catch blocking system calls, perf to analyze CPU behavior and call stacks, and iotop to measure actual storage activity. For repeatable command timing, use hyperfine. Save bpftrace for questions the simpler tools cannot answer. Memorizing this progression prevents wasted hours chasing phantom bottlenecks caused by the profiling tools themselves.

Establishing Identity and Resource Baselines

Before you can diagnose a process, you must establish exactly what it is doing at a macro level. Beginners often jump straight into profiling without verifying the target's basic identity, leading to confusion when the profiled application turns out to be a background updater rather than the intended build script.

Run ps -o pid, ppid, stat,%cpu,%mem, etime, cmd -p PID. This specific invocation establishes identity, parentage, state, runtime, and resource use before you attach another tool. You need to know if the process is a zombie, if it has been running for three seconds or three days, and who spawned it. Pay particular attention to the state column. A process in uninterruptible sleep requires a completely different diagnostic approach than one actively consuming CPU cycles.

When a multithreaded process looks busy and you need to see whether one specific thread is responsible for the load, switch to top -H -p PID. Modern applications written in Go or Rust spawn dozens of threads by default. This command breaks out individual threads, exposing the single runaway loop hiding inside a massive application.

A single snapshot can be misleading. To watch CPU, context-switch, memory, or I/O behavior over time, introduce pidstat -p PID 1. This tool captures metrics at 1-second intervals. That granularity reveals micro-bursts of CPU saturation that a standard 3-second top refresh completely masks.

Exposing Blocked System Calls with strace

Initially, we attempted to pipe the entire strace -f output of a failing local web server directly to the terminal to spot the error in real-time. The sheer volume of epoll_wait and clock_gettime calls scrolled past too quickly to read. Worse, the syscall-heavy terminal rendering distortion artificially slowed the process down by an additional 400 to 600 milliseconds per request. We dropped the live-tail approach entirely.

Capture first, then measure. Lead with strace -f -tt -T -p PID. The -f flag follows descendants, -tt timestamps calls, and -T reports the exact time spent in each call. Reviewing this output exposes the actual bottlenecks. Repeated openat failures expose missing files or bad search paths. Long connect calls reveal network waits. Excessive futex activity suggests lock contention or poor thread coordination. Stalled read calls identify an idle pipe or socket waiting for data.

For launch-time diagnostics, narrow the scope. Run strace -f -e trace=file, network -o trace.log command. Filtering by these trace classes can reduce the trace log size from gigabytes to megabytes during a short capture of a busy local service.

Sanitize Trace Logs Before Bug Submission

Writing raw system calls to disk will capture sensitive paths, environment variables, and network endpoints. Always scrub these logs before sharing them in public issue trackers.

Isolating CPU Bottlenecks from Execution Noise

When a process is burning CPU, you need to separate hot code from noisy timings. Start with perf stat -p PID for a coarse hardware-counter view of an existing process. This reveals instruction counts, branch misses, and page faults. High branch misses indicate unpredictable logic, while excessive page faults point to memory allocation thrashing.

When you need actual call stacks, run perf record -g -p PID -- sleep 10 followed by perf report. This targets specific failures. You might find a genuinely CPU-bound loop, excessive context switching, cache-unfriendly work, or time concentrated in an unexpected call path. While these profiling techniques are highly effective for compiled binaries, they offer limited visibility into interpreted languages without specialized runtime hooks.

To measure the impact of your fixes, use hyperfine. Running hyperfine --warmup 3 executes the command three times before measurement begins. This can help populate the filesystem cache for larger binary payloads so subsequent runs are less dominated by disk I/O.

You must qualify what perf can actually prove. Relying on perf record to generate accurate call graphs requires the target binary to be compiled with frame pointers (-fno-omit-frame-pointer) or debug symbols. Without them, the profiler will only report shallow, unresolvable hex addresses that provide no actionable insight into the application's source code. Missing debug symbols, aggressive compiler inlining, sampling frequency limits, virtualization layers, and strict kernel security restrictions can all reduce stack quality or block access to hardware counters entirely.

Verifying Storage Thrash Against Idle File Handles

High disk utilization requires proof before you blame the CPU or the application logic. Use sudo iotop -oPa to show processes accumulating actual storage activity and filter out idle entries. This command distinguishes sustained reads or writes from a process that merely has files open.

Local diagnostic output can show a clear divergence. A process might show 0 bytes/sec in iotop while lsof reveals it holds open 15 to 20 file descriptors. This distinction separates a process actively thrashing the disk from one that is merely idling with open handles. An application holding a database connection open does not necessarily generate disk traffic.

Several factors limit what appears in this output. Permissions, kernel accounting support, buffered I/O, and short-lived bursts can obscure the true source of disk pressure. Buffered writes, for example, appear as memory operations until the kernel flushes them to physical storage, at which point the kernel threads themselves appear responsible for the I/O.

Recognize when this tool is overkill. A known file operation with an obvious progress indicator rarely needs system-wide I/O accounting. If you are copying a massive database dump, the bottleneck is already identified, and running an I/O monitor simply wastes resources.

Deploying eBPF for Intermittent Kernel Events

Reach for bpftrace only when simpler tools run out. When you need to track specific kernel interactions without the overhead of a full trace, give a focused invocation. Run sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat /pid == $1/ { @[comm] = count(); }' 1234. You must explicitly replace 1234 with your target PID.

This answers a very specific question. It determines whether a particular process is repeatedly entering a chosen kernel operation, without collecting a massive general-purpose trace. A targeted script counting sys_enter_openat calls usually compiles and attaches quickly through the eBPF verifier. Once running, its per-syscall overhead is low enough for production-like local load testing.

This tool earns its complexity when diagnosing intermittent behavior, cross-process interactions, latency distributions, or kernel events that disappear before conventional polling catches them. Writing custom scripts allows you to aggregate data in the kernel, passing only the final histograms or counts back to user space. For advanced syntax and tracepoint discovery, consult the official bpftrace documentation.

When your next local service hangs silently and the workstation fan spins up, will you reach for a heavy profiler out of habit, or will you start by asking the operating system exactly what the process is waiting for?