Mpstat: Profiling Per-Core CPU Saturation, Auditing Hardware Interrupt Balance, and Triaging Multiprocessor Imbalance in Production

This disconnect is one of the most deceptively dangerous traps in modern operations: the tyranny of aggregate averages. On a modern server equipped with dozens of processor cores, sixty-three cores can sit in low-power idle states while a single overwhelmed core is completely pinned at 100 percent capacity, bottlenecking your entire customer journey. Because standard monitoring systems bundle every processor into a single arithmetic mean, that localized firestorm gets smoothed out into a harmless-looking blip.

To uncover what is genuinely happening beneath the surface, an engineer must look past the global averages and inspect the exact heartbeat of each individual core. This is where the mpstat utility—a core diagnostic tool within the Linux sysstat framework—proves indispensable. Rather than reducing multi-core activity to a single aggregate number, mpstat (multiprocessor statistics) opens a real-time window into how the operating system distributes user applications, kernel tasks, hardware interrupts, and virtualisation delays across every processor in the machine.

When an outage hits and you need to determine within seconds whether a rogue process or unbalanced queue is suffocating a specific processor core, there is one foundational command you should run before doing anything else:

mpstat -P ALL 1 1

Linux 6.8.0-48-generic (db-primary-node-01)     08/18/2026  _x86_64_    (8 CPU)

02:15:00 UTC  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:15:01 UTC  all    4.12    0.00    1.88    0.25    0.00    0.88    0.00    0.00    0.00   92.87
02:15:01 UTC    0    2.02    0.00    1.01    0.00    0.00    0.00    0.00    0.00    0.00   96.97
02:15:01 UTC    1    3.03    0.00    2.02    0.00    0.00    0.00    0.00    0.00    0.00   94.95
02:15:01 UTC    2    1.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.00
02:15:01 UTC    3   25.25    0.00   10.10    2.02    0.00    7.07    0.00    0.00    0.00   55.56
02:15:01 UTC    4    0.00    0.00    1.01    0.00    0.00    0.00    0.00    0.00    0.00   98.99
02:15:01 UTC    5    1.01    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   98.99
02:15:01 UTC    6    0.00    0.00    1.01    0.00    0.00    0.00    0.00    0.00    0.00   98.99
02:15:01 UTC    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

What It Does in Plain English

The mpstat command dissects processing activity into discrete operational categories—such as user-space calculations, kernel operations, hardware and software interrupt handling, and hypervisor-induced wait times—for each individual core independently.

By looking at individual rows rather than the global summary row (all), administrators can instantly detect when a single worker thread is redlining, when network traffic is bottlenecked on a single core's interrupt queue, or when virtualized cloud instances are being starved of CPU time by the host hypervisor.

The Underlying Kernel Architecture: How mpstat Interrogates the Silicon

To interpret mpstat accurately during high-pressure incidents, it helps to understand how the Linux kernel accounts for CPU time across symmetric multiprocessing (SMP) hardware. The tool does not directly query hardware registers; instead, it parses the virtual file /proc/stat, a pseudofilesystem interface through which the kernel exposes internal hardware accounting counters.

Every time the kernel's scheduler timer interrupt fires on a logical core—governed by the system's CONFIG_HZ frequency setting (typically between 250Hz and 1000Hz, translating to ticks every 1ms to 4ms)—the kernel increments internal counters for that specific CPU depending on the execution context in which the tick landed.

Under the Completely Fair Scheduler (CFS) and the Earliest Eligible Virtual Deadline First (EEVDF) scheduler, each CPU maintains its own runqueue structure (struct rq). When tasks transition across execution states, the kernel updates time slices expressed in raw clock ticks (or jiffies). The sysstat framework samples /proc/stat across user-defined time intervals $\Delta t$, computes the differential $\Delta \text{ticks}$ across all state categories for each core, and converts these values into clean percentages:

$$\% \text{State}{\text{CPU}_k} = \frac{\Delta \text{Ticks}{\text{State}, \text{CPU}k}}{\sum \Delta \text{Ticks}{\text{All States}, \text{CPU}_k}} \times 100$$

Furthermore, mpstat can correlate kernel state metrics with interrupt vector dispatch tables. When peripheral devices—such as Non-Volatile Memory Express (NVMe) controllers or 100GbE Network Interface Cards (NICs)—raise physical line signals or Message Signaled Interrupts (MSI/MSI-X), the kernel executes the immediate top-half hardware interrupt routine on the core assigned by the interrupt controller's affinity mask.

If execution exceeds microsecond thresholds, the kernel defers remaining processing to software interrupt routines (ksoftirqd/X bottom halves), monitored under the Linux Network Scaling Infrastructure. By tracking /proc/interrupts and /proc/softirqs, mpstat can correlate core execution states directly with hardware interrupt vectors and softirq subsystems.

Core Flags & Quick Start

The mpstat utility offers several flags to isolate specific processor behaviors across modern multi-core systems:

• -P { cpu [,...] | ON | ALL }: Specifies the logical processors to monitor. Passing ALL forces mpstat to report telemetry for every individual core alongside the global node average.
• -u: Reports standard CPU utilization metrics (default operational mode).
• -I { SUM | CPU | SCPU | ALL }: Emits detailed interrupt telemetry. CPU outputs per-core hardware interrupt rates; SCPU breaks down software interrupt handling (e.g., NET_RX, BLOCK, TASKLET, TIMER).
• -A: Combines all available telemetry outputs into a comprehensive matrix, equivalent to issuing -u -I ALL -P ALL.
• -o JSON: Directs mpstat to emit structured JSON streams, ideal for automated parsing, continuous integration harnesses, and daemonized telemetry agents.
• interval & count: Positional integer arguments supplied at the end of the command invocation defining the sampling epoch in seconds and the total number of reports to emit before terminating.

Telemetry Header Definitions

• %usr: Percentage of CPU time spent executing application code in user space (un-niced).
• %nice: Percentage of CPU time spent executing user-space code with an altered scheduling priority.
• %sys: Percentage of CPU time spent executing kernel-space code, system calls, and driver logic.
• %iowait: Percentage of time the CPU was idle while the system had pending synchronous block I/O requests.
• %irq: Percentage of CPU time consumed servicing hardware interrupts (top halves).
• %soft: Percentage of CPU time consumed servicing software interrupts (bottom halves like network ingress processing).
• %steal: Percentage of time spent in involuntary wait while the hypervisor serviced a different virtual core.
• %guest / %gnice: Percentage of time dedicated to running virtual processors for guest VMs.
• %idle: Percentage of time the CPU resided in an idle state without pending local disk I/O operations.

5 Real-World Production Use Cases

1. Isolating Single-Threaded Application Saturation Masked by Aggregate Averages

The Scenario

An in-memory caching tier running Redis alongside a Python asynchronous background worker is failing SLA benchmarks. While aggregate CPU utilization across the 8-core instance hovers around 13%, application logs indicate severe queue backpressure and request timeouts. The sysadmin suspects that a single-threaded runtime loop has pegged one core to 100%, but high-level monitoring tools averaging over all CPUs fail to trigger alerting thresholds.

Command Invocation

mpstat -P ALL 1 3

Terminal Telemetry

Linux 6.8.0-48-generic (cache-redis-02)     08/18/2026  _x86_64_    (8 CPU)

02:20:10 UTC  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:20:11 UTC  all   12.48    0.00    0.51    0.00    0.00    0.13    0.00    0.00    0.00   86.88
02:20:11 UTC    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
02:20:11 UTC    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
02:20:11 UTC    2   98.02    0.00    1.98    0.00    0.00    0.00    0.00    0.00    0.00    0.00
02:20:11 UTC    3    1.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.00
02:20:11 UTC    4    0.00    0.00    1.01    0.00    0.00    1.01    0.00    0.00    0.00   97.98
02:20:11 UTC    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
02:20:11 UTC    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
02:20:11 UTC    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

Line-by-Line Diagnostic Analysis

• all: The global aggregate row indicates an average %usr of 12.48% and an aggregate %idle of 86.88%. Standard monitoring systems reading aggregate /proc/stat will deem the host completely healthy and under-utilized.
• CPU 0, 1, 5, 6, 7: Logical cores are in complete 100.00% idle states, consuming minimal power.
• CPU 2: Demonstrates a classic single-core bottleneck: %usr is 98.02%, %sys is 1.98%, and %idle is precisely 0.00%. A single-threaded event loop (e.g., Redis or a Python process constrained by the Global Interpreter Lock) has fully saturated this core.

Remediation and Actionable Engineering Plan

1. Identify the Pinned Process: Execute pidstat -p ALL -u 1 1 | sort -k 8 -r or top -H -p $(pgrep redis-server) to identify the exact thread ID pinned to CPU 2.
2. Enforce Processor Affinity / Partitioning: If multiple single-threaded processes are contending for the same core, isolate and bind worker processes across distinct physical cores using taskset: bash taskset -c 0 /usr/bin/redis-server /etc/redis/redis-instance-0.conf taskset -c 1 /usr/bin/redis-server /etc/redis/redis-instance-1.conf
3. Horizontal Worker Scaling: For application runtimes (such as Node.js or Python), spawn additional worker instances under a process manager to utilize cores 0, 1, 4, 5, 6, and 7, distributing single-threaded constraints evenly across the SMP topology.

2. Auditing Hardware and SoftIRQ Bottlenecks Under Heavy Ingress Network Loads

The Scenario

An edge API proxy routing 950,000 packets per second (PPS) across dual 25GbE network interfaces begins reporting packet drops at the operating system socket buffer layer (rx_dropped in netstat -s). The proxy application itself is not consuming high user CPU, yet incoming TCP connections are failing handshake deadlines. The sysadmin suspects that incoming network packet interrupts are landing on a single processor core, generating a Software Interrupt (SoftIRQ) storm.

Command Invocation

mpstat -I SCPU -P ALL 1 2

Terminal Telemetry

Linux 6.8.0-48-generic (edge-proxy-lon01)   08/18/2026  _x86_64_    (8 CPU)

02:25:01 UTC  CPU       HI/s    TIMER/s   NET_TX/s   NET_RX/s    BLOCK/s  TASKLET/s    SCHED/s     HRT/s      RCU/s
02:25:02 UTC    0       0.00    1001.00       0.00  842109.00       0.00       0.00     250.00      0.00     120.00
02:25:02 UTC    1       0.00     998.00       0.00       0.00       0.00       0.00     248.00      0.00     118.00
02:25:02 UTC    2       0.00    1002.00       0.00       0.00       0.00       0.00     251.00      0.00     122.00
02:25:02 UTC    3       0.00    1000.00       0.00       0.00       0.00       0.00     249.00      0.00     119.00
02:25:02 UTC    4       0.00    1000.00       0.00       0.00       0.00       0.00     250.00      0.00     121.00
02:25:02 UTC    5       0.00    1003.00       0.00       0.00       0.00       0.00     252.00      0.00     120.00
02:25:02 UTC    6       0.00     999.00       0.00       0.00       0.00       0.00     247.00      0.00     117.00
02:25:02 UTC    7       0.00    1001.00       0.00       0.00       0.00       0.00     250.00      0.00     123.00

Cross-referencing this with standard CPU state time confirms the diagnosis:

mpstat -P ALL 1 1

02:25:05 UTC  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:25:06 UTC    0    0.00    0.00    2.94    0.00    1.96   95.10    0.00    0.00    0.00    0.00
02:25:06 UTC    1    8.00    0.00    1.00    0.00    0.00    0.00    0.00    0.00    0.00   91.00

Line-by-Line Diagnostic Analysis

• NET_RX/s on CPU 0: CPU 0 is servicing 842,109 network receive softirq transitions per second, while CPU 1 through CPU 7 process zero network receive interrupts.
• %soft on CPU 0: The standard utilization report shows CPU 0 spending 95.10% of its total time exclusively servicing software interrupts (%soft). The kernel worker ksoftirqd/0 has taken over Core 0.
• Root Cause: The physical NIC interrupt line (or default single queue) has its affinity bound exclusively to CPU 0. Under sustained high traffic, CPU 0 is overwhelmed by network packet processing, dropping frames before user-space applications can read them.

Remediation and Actionable Engineering Plan

1. Enable Multi-Queue NIC Distribution: Check hardware queue allocation via ethtool -l eth0 and increase combined queue channels to match the core count: bash ethtool -L eth0 combined 8
2. Deploy and Direct irqbalance: Ensure the interrupt balancing daemon is running to distribute MSI-X vectors across all CPU cores: bash systemctl restart irqbalance
3. Implement Software Receive Packet Steering (RPS): If the NIC does not support multiple hardware queues, spread software interrupt processing across cores by configuring the sysfs RPS bitmask. To distribute ingress traffic across all 8 cores (hex mask ff): bash echo "ff" > /sys/class/net/eth0/queues/rx-0/rps_cpus Re-running mpstat -I SCPU -P ALL 1 2 will confirm uniform NET_RX/s distribution across all cores.

3. Diagnosing Multi-Tenant Cloud Hypervisor Contention via Steal Time (%steal)

The Scenario

A mission-critical e-commerce application deployed on public cloud instances (such as AWS EC2 or GCP Compute Engine) suffers severe performance degradation during high-traffic sales events. Database response times jump by an order of magnitude. Internal monitoring reveals that while the application's user-space load (%usr) is only at 25%, wall-clock execution time has tripled. The team must determine whether the issue is inside the application or caused by CPU oversubscription on the host hypervisor (noisy neighbors).

Command Invocation

mpstat -P ALL 1 4

Terminal Telemetry

Linux 6.8.0-1017-aws (ip-10-0-4-82)     08/18/2026  _x86_64_    (4 CPU)

02:30:15 UTC  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:30:16 UTC  all   24.12    0.00    3.27    0.25    0.00    0.50   48.24    0.00    0.00   23.62
02:30:16 UTC    0   22.00    0.00    4.00    0.00    0.00    1.00   52.00    0.00    0.00   21.00
02:30:16 UTC    1   25.25    0.00    2.02    0.00    0.00    0.00   45.45    0.00    0.00   27.28
02:30:16 UTC    2   26.00    0.00    3.00    1.00    0.00    1.00   49.00    0.00    0.00   20.00
02:30:16 UTC    3   23.23    0.00    4.04    0.00    0.00    0.00   46.46    0.00    0.00   26.27

Line-by-Line Diagnostic Analysis

• %usr (22.00% – 26.00%): The guest operating system is executing standard application tasks, utilizing roughly a quarter of its compute budget.
• %steal (45.45% – 52.00%): Around half of all physical CPU cycles allocated to this virtual machine are being involuntarily withheld by the underlying hypervisor.
• Interpretation: The cloud hypervisor scheduler has oversubscribed the host physical CPU. The virtual CPUs assigned to this instance are placed into wait states while the hypervisor executes instructions for other co-located tenant VMs on the same hardware core. The latency spike is strictly external to the guest operating system.

Remediation and Actionable Engineering Plan

1. Examine Burst Credit Exhaustion: If operating on burstable instance families (such as AWS t3/t4g or GCP e2-standard), verify whether CPU burst credits have been depleted. If exhausted, enable unlimited credit mode or upgrade instance types: bash aws ec2 modify-instance-credit-specification --instance-id i-0123456789abcdef0 --cpu-credits-specification CpuCredits=unlimited
2. Migrate to Dedicated Compute Instances: For predictable production performance, migrate away from shared instances to dedicated tenancy, bare-metal instances, or compute-optimized instance types (such as AWS c6i/c7i or GCP c3-highcpu) where physical-to-virtual CPU ratios are pinned 1:1.
3. Implement Automated Steal-Time Alerting: Configure observability daemons to trigger an automated alert whenever %steal > 5.00% for longer than three consecutive minutes.

4. Profiling SMT/Hyper-Threading Contention and NUMA Node Imbalances

The Scenario

An enterprise PostgreSQL database running on a high-spec dual-socket server (64 physical cores, 128 logical threads across two NUMA nodes) experiences significant query latency regression under concurrent analytical workloads. A junior administrator suggests the host has plenty of headroom because global CPU usage sits at 40%. However, kernel profilers reveal extreme spinlock contention (native_queued_spin_lock_slowpath). The team suspects simultaneous multithreading (SMT) sibling contention and unbalanced NUMA memory access are degrading throughput.

Command Invocation

mpstat -P 0,1,64,65 1 2

(Where CPU 0 and CPU 64 are hardware SMT siblings on NUMA Node 0, and CPU 1 and CPU 65 are SMT siblings on NUMA Node 1).

Terminal Telemetry

Linux 6.8.0-48-generic (db-oltp-primary)    08/18/2026  _x86_64_    (128 CPU)

02:35:40 UTC  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:35:41 UTC    0   58.42    0.00   41.58    0.00    0.00    0.00    0.00    0.00    0.00    0.00
02:35:41 UTC   64   62.11    0.00   37.89    0.00    0.00    0.00    0.00    0.00    0.00    0.00
02:35:41 UTC    1    2.00    0.00    1.00    0.00    0.00    0.00    0.00    0.00    0.00   97.00
02:35:41 UTC   65    1.01    0.00    1.01    0.00    0.00    0.00    0.00    0.00    0.00   97.98

Line-by-Line Diagnostic Analysis

• CPU 0 & CPU 64: Both logical hyperthreads on the same physical core are pinned (0.00% idle), with %sys elevated to approximately 40% simultaneously.
• Microarchitectural Diagnosis: SMT thread-pairs share physical execution units (ALUs, FPUs) and L1/L2 caches. When sibling threads run heavy compute workloads simultaneously, execution pipelines stall. Furthermore, high %sys points to kernel spinlock contention across NUMA interconnects.
• CPU 1 & CPU 65: Sibling threads on the alternate NUMA socket sit essentially idle (~97% idle), confirming that database connections are heavily skewed toward Node 0.

Remediation and Actionable Engineering Plan

1. Pin Workloads Using NUMA-Aware Directives: Bind database processes to specific NUMA nodes and local memory allocations using numactl: bash numactl --cpunodebind=0 --membind=0 /usr/lib/postgresql/16/bin/postgres -D /var/lib/postgresql/16/main
2. Evaluate Runtime SMT Disabling for Latency-Critical Workloads: For transaction engines with heavy thread synchronization, disable SMT at runtime via the sysfs topology interface to eliminate sibling contention entirely: bash echo off > /sys/devices/system/cpu/smt/control Re-checking mpstat -P ALL 1 1 will show half the logical core count, with physical execution units operating without resource sharing stalls.

5. Automated Continuous Telemetry and I/O Wait Bottleneck Detection

The Scenario

During a distributed storage soak test, a Kafka broker cluster experiences periodic throughput collapses where message ingestion drops from 500 MB/s to zero for ten-second intervals. Engineers must ascertain whether the system is experiencing kernel lock stalls, memory allocation freezes, or underlying NVMe storage controller write-barrier stalls (%iowait). The task requires scripting automated telemetry capture to record per-core performance metrics in a machine-parsable format for post-test analysis.

Command Invocation

Deploy mpstat with high-frequency JSON-formatted emission directly into an automated observability ingest pipeline:

mpstat -P ALL -u 2 5 -o JSON > /var/log/telemetry/cpu_telemetry_soaktest.json

Simultaneously monitor real-time text output during the throughput collapse:

mpstat -P ALL 1 3

Terminal Telemetry

Linux 6.8.0-48-generic (kafka-broker-03)    08/18/2026  _x86_64_    (4 CPU)

02:40:02 UTC  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
02:40:03 UTC  all    2.50    0.00    4.25   82.25    0.00    0.25    0.00    0.00    0.00   10.75
02:40:03 UTC    0    3.03    0.00    5.05   84.85    0.00    1.01    0.00    0.00    0.00    6.06
02:40:03 UTC    1    2.02    0.00    4.04   79.80    0.00    0.00    0.00    0.00    0.00   14.14
02:40:03 UTC    2    3.00    0.00    4.00   81.00    0.00    0.00    0.00    0.00    0.00   12.00
02:40:03 UTC    3    1.98    0.00    3.96   83.17    0.00    0.00    0.00    0.00    0.00   10.89

Line-by-Line Diagnostic Analysis

• %iowait (79.80% – 84.85%): Across all logical CPUs, over 80% of time slices are spent in %iowait.
• Kernel Meaning of %iowait: Crucially, %iowait does not mean the CPU is actively performing disk operations. Rather, it means the CPU is completely idle, but execution is blocked because threads are lodged in the kernel's TASK_UNINTERRUPTIBLE state waiting for synchronous block I/O requests (such as fsync or journal commits) to complete on physical storage devices.
• %usr (<3%) & %sys (<5%): The compute engines are starved of work because application threads cannot progress until the storage controllers acknowledge persistent writes.

Automated Parsing Pipeline Script

To integrate this telemetry into automated alerting pipelines, use the following production-grade Python telemetry consumer to parse mpstat JSON output and flag latency anomalies:

#!/usr/bin/env python3
"""
Automated mpstat JSON Telemetry Parser for CI/CD Soak Testing
"""
import json
import sys

def analyze_telemetry(filepath: str, iowait_threshold: float = 20.0, steal_threshold: float = 5.0):
    with open(filepath, 'r') as f:
        data = json.load(f)

    hosts = data['sysstat']['hosts']
    for host in hosts:
        nodename = host['nodename']
        statistics = host['statistics']

        for record in statistics:
            timestamp = record.get('utc') or record.get('timestamp')
            cpu_load = record['cpu-load']

            for cpu_data in cpu_load:
                cpu_id = cpu_data['cpu']
                iowait = float(cpu_data['iowait'])
                steal = float(cpu_data['steal'])

                if iowait > iowait_threshold:
                    print(f"[ALERT - I/O REGRESSION] Node: {nodename} | Time: {timestamp} | "
                          f"CPU: {cpu_id} | %iowait: {iowait}% > Threshold ({iowait_threshold}%)")

                if steal > steal_threshold:
                    print(f"[ALERT - STEAL CONTINGENCY] Node: {nodename} | Time: {timestamp} | "
                          f"CPU: {cpu_id} | %steal: {steal}% > Threshold ({steal_threshold}%)")

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print(f"Usage: {sys.argv[0]} <path_to_mpstat_json>")
        sys.exit(1)
    analyze_telemetry(sys.argv[1])

Remediation and Actionable Engineering Plan

1. Investigate Storage Subsystem Latency: Correlate the mpstat %iowait event with disk metrics using iostat -xz 1 5 to identify which physical block device (nvme0n1, sda) is saturating write queues or exhibiting high await times.
2. Optimize Filesystem Flush Parameters: Mitigate synchronous write stalls by adjusting kernel dirty page writeback thresholds in /etc/sysctl.conf: bash sysctl -w vm.dirty_background_ratio=5 sysctl -w vm.dirty_ratio=10
3. Transition to Asynchronous File Subsystems: Configure the application runtime to utilize asynchronous I/O engines (such as modern Linux io_uring interfaces or asynchronous log appenders) to eliminate blocking synchronous fsync calls on worker threads.

Metric Analysis Matrix

To rapidly synthesize diagnostic outputs during a production incident, refer to this quick reference matrix:

What Can Go Wrong: Pitfalls, Sampling Biases, and Inaccuracies

When utilizing mpstat in production diagnostic workflows, engineers commonly encounter subtle pitfalls that lead to flawed conclusions:

1. The First-Epoch Statistical Distortion

A critical trap when executing mpstat without an interval argument (such as simply typing mpstat -P ALL) is that the output does not represent current real-time activity. The first line of output emitted by mpstat—or the sole output when executed without an interval—is a historical average of CPU time spent across every state since the operating system booted.

If a server has been running for 400 days with nominal 5% utilization, an acute catastrophic saturation event occurring right now will be invisible in that baseline row.

2. The Fallacy of %iowait as a Direct Metric of Processing Load

Engineers frequently misinterpret %iowait as a metric indicating that the CPU is working hard to process storage operations. As detailed in the kernel architecture analysis, a CPU core spending 90% of its time in %iowait is technically 100% idle.

Attempting to optimize CPU processing or adding more CPU capacity to resolve an %iowait issue will yield zero improvement. The bottleneck is strictly within the physical storage layer, the NVMe fabric, or the filesystem locking hierarchy.

3. Sampling Overhead and High-Frequency Jitter

While mpstat is lightweight, running high-frequency sampling (e.g., mpstat 0.01 or 100Hz polling) across dense multi-socket servers (e.g., 256 logical cores) induces non-trivial overhead. Parsing /proc/stat requires the kernel to iterate over all CPU data structures, lock internal state tables, and format ASCII strings in virtual memory.

Under extreme lock-sensitive production workloads, aggressive polling of /proc/stat can exacerbate spinlock contention and perturb the very latencies you are attempting to measure. Restrict production sampling intervals to no less than one second (1s) unless operating in an isolated test harness.

Today's Takeaway

The single most valuable habit you can adopt today is checking how evenly your system distributes work across its processor cores before trusting any aggregate CPU graph. Open a terminal on your primary development machine or application server and run mpstat -P ALL 1 5. Instead of looking at the overall summary row, scan the table for variance across individual cores: look for a lone core sitting at 0.00% %idle, check whether %soft interrupts are piling up exclusively on CPU 0, and verify that %steal remains flat at zero. Spending five minutes getting familiar with this per-core breakdown will ensure that hidden single-threaded bottlenecks and interrupt storms never take your production systems by surprise again.

Authoritative Documentation & Further Reading

• mpstat(1) — Linux Manual Page
• The Linux Kernel /proc Filesystem Documentation
• Linux Kernel Scheduler Architecture (CFS/EEVDF)
• Linux Kernel Scaling and SoftIRQ Networking Documentation
• Red Hat Enterprise Linux 9 Performance Tuning Guide
• ArchWiki Comprehensive System Performance Guide