Stress-ng: Benchmarking Kernel Subsystems, Validating Container Resource Quotas, and Stress-Testing Hardware Resilience in Production

The subsequent incident review inevitably uncovers an uncomfortable truth: a recently deployed kernel patch, container update, or hardware configuration behaved unpredictably when pushed to its limits. Subsystems that performed flawlessly during light development testing buckled the moment they faced real-world concurrency and lock contention. In systems engineering, hope is never an operational strategy. If you do not deliberately push your systems to their breaking points in a controlled environment, live user traffic will eventually do it for you at the worst possible moment.

The standard tool engineered to prevent these disasters is stress-ng, authored and maintained by Colin Ian King. It is a comprehensive, surgical workload synthesiser designed to evaluate the physical and architectural boundaries of operating systems, hardware caches, memory management units, and kernel subsystems. Rather than relying on unpredictable applications to see if a machine holds up, stress-ng generates deterministic computational storms that allow engineers to test hardware stability, tune kernel parameters, and verify resource quotas with mathematical precision before servers ever handle customer data.

To immediately evaluate a machine's baseline health across its processor, virtual memory, and storage controllers without risking an unrecoverable system freeze, the single most practical command to run is:

stress-ng --cpu 4 --vm 2 --vm-bytes 1G --io 2 --timeout 30s --metrics-brief

Running this thirty-second sanity check spawns four CPU workers dedicated to compute pipelines, two virtual memory workers allocating and exercising one gigabyte of RAM each, and two input/output workers exercising disk synchronization, returning a concise summary of system throughput:

stress-ng: info:  [28412] setting to a 30 second run per stressor
stress-ng: info:  [28412] dispatching hogs: 4 cpu, 2 io, 2 vm
stress-ng: info:  [28412] cache allocate: default cache size: 36864K
stress-ng: info:  [28412] successful run completed in 30.01s
stress-ng: metrc: [28412] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [28412]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [28412] cpu               18452     30.00     98.45      0.12       615.06         187.19
stress-ng: metrc: [28412] io                24810     30.01      0.08     18.92       826.72        1305.79
stress-ng: metrc: [28412] vm               142100     30.01     14.22     22.10      4735.08        3912.44

This telemetry confirms that the operating system scheduler evenly distributed the four CPU workers (accumulating 98.45 seconds of user-space compute time), the virtual memory stressors successfully dirtied and verified 142,100 memory chunks across their allocations, and the I/O stressors spent 18.92 seconds of kernel system time exercising file synchronization boundaries without triggering kernel lockups or storage stalls.

How Stress-ng Interacts with System Subsystems

The design of stress-ng revolves around modular stressor engines. Each stressor targets a specific hardware path or operating system layer by generating high-density system calls and memory operations.

Core Flags and Operational Mechanics

Understanding the fundamental flags allows administrators to construct precise load profiles suited to their specific validation needs:

• --cpu <N>: Spawns N concurrent worker threads dedicated to saturating processor execution pipelines using selectable computational algorithms.
• --vm <N>: Allocates and exercises virtual memory across N workers using memory-intensive stressors that invoke continuous page mapping, modification, and unmapping.
• --vm-bytes <size>: Specifies the exact chunk size of anonymous memory allocated by each virtual memory worker (e.g., 4G, 80%).
• --io-uring <N>: Spawns N asynchronous I/O workers exercising the high-throughput Linux kernel io_uring interface.
• --taskset <cpulist>: Enforces strict CPU affinity, binding spawned stressors to a defined range or comma-separated list of logical CPU cores.
• --timeout <time>: Enforces an absolute, deterministic execution duration (e.g., 60s, 10m), after which all worker processes terminate cleanly via timer signals.
• --metrics-brief: Emits a structured telemetry summary detailing operations executed, runtime duration, and standardised bogo-operations per second (bogo ops/s).
• --dry-run: Parses, validates, and displays the complete stress engine configuration without spawning processes or consuming system resources.

Five Real-World Production Use Cases

Synthetic stress testing provides maximum value when tailored to specific operational challenges. The five production topologies below demonstrate how to validate resource boundaries, measure hardware limits, and verify platform resilience.

1. Validating Container Cgroup Memory Limits & OOM Killer Policies

Scenario

A Kubernetes node runs multi-tenant workloads constrained by Linux Control Groups v2. You must ensure that when a rogue container exceeds its memory.max threshold, the kernel's asynchronous memory reclaimer invokes the cgroup Out-Of-Memory (OOM) killer to terminate only the misbehaving container processes, rather than locking the host kernel or triggering system-wide memory thrashing.

Exact Production Invocation

Encapsulate the execution within a transient systemd cgroup boundary with strict memory limits:

systemd-run --unit=cgroup-validator --scope \
  -p MemoryAccounting=true \
  -p MemoryMax=2G \
  -p MemoryHigh=1800M \
  stress-ng --vm 2 --vm-bytes 1400M --vm-method all --oomable --timeout 45s --metrics-brief

Realistic Terminal Output

Running as unit: cgroup-validator.scope
stress-ng: info:  [31055] setting to a 45 second run per stressor
stress-ng: info:  [31055] dispatching hogs: 2 vm
stress-ng: info:  [31055] cache allocate: default cache size: 36864K
stress-ng: fail:  [31057] stress-ng-vm: [31057] terminated on signal: 9 (Killed) (instance 0)
stress-ng: info:  [31055] stressor stress-ng-vm (instance 0) was killed by the OOM killer
stress-ng: metrc: [31055] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [31055]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [31055] vm                 8241      8.12      4.11      6.20      1014.88         800.09
stress-ng: info:  [31055] unsuccessful run completed in 8.12s

Line-by-Line Telemetry Analysis

• Running as unit: cgroup-validator.scope: Systemd binds the process hierarchy inside a dedicated cgroup v2 slice with a hard memory ceiling of 2.0 GiB and a throttling threshold at 1.8 GiB.
• dispatching hogs: 2 vm: The harness initializes two virtual memory stressors, each requesting 1,400 MiB of anonymous memory (2,800 MiB aggregate), intentionally overcommitting the 2.0 GiB cgroup boundary.
• --vm-method all: Cycles through every memory-touching mechanism (zeroing, byte-twiddling, reverse walking, random patterns) to ensure the kernel memory management subsystem cannot optimize allocations via duplicate zero pages.
• terminated on signal: 9 (Killed): The Linux kernel memory controller detected allocation exceeding memory.max, invoked mem_cgroup_out_of_memory(), and dispatched a non-catchable SIGKILL.
• unsuccessful run completed in 8.12s: The workload collapsed cleanly in 8.12 seconds, confirming that the OOM killer responded promptly without leaving processes stuck in an unkillable sleep state.

Sysadmin Next Actions

Inspect kernel cgroup event counters by reviewing /sys/fs/cgroup/system.slice/cgroup-validator.scope/memory.events or querying the journal via journalctl -k -g "oom_reaper". Verify that the oom_kill counter increments cleanly and that the host operating system experienced no broader memory pressure stalls.

2. Benchmarking CPU Thermal Throttling & NUMA Latency Under Load

Scenario

A dual-socket AMD EPYC server experiences sporadic latency spikes during database transactions. You suspect that memory allocations are crossing Non-Uniform Memory Access (NUMA) boundaries via the interconnect fabric, while sustained vector workloads are pushing package temperatures over thermal thresholds, causing CPU core downclocking.

Exact Production Invocation

stress-ng --cpu $(nproc) --cpu-method matrixprod \
  --matrix 8 --cache 8 --numa 4 \
  --taskset 0-63 \
  --thermal-zones --timeout 60s --metrics-brief

Realistic Terminal Output

stress-ng: info:  [40112] setting to a 60 second run per stressor
stress-ng: info:  [40112] dispatching hogs: 64 cpu, 8 matrix, 8 cache, 4 numa
stress-ng: info:  [40112] thermal zone: x86_pkg_temp 0: start 42.0 C, end 88.5 C (delta 46.5 C)
stress-ng: info:  [40112] thermal zone: x86_pkg_temp 1: start 41.0 C, end 76.0 C (delta 35.0 C)
stress-ng: metrc: [40112] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [40112]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [40112] cpu              891240     60.00   3780.20      2.10     14854.00        3777.67
stress-ng: metrc: [40112] matrix           412010     60.00    472.10      0.40      6866.83        1423.83
stress-ng: metrc: [40112] cache            182100     60.01    465.80      4.20      3034.49        387.45
stress-ng: metrc: [40112] numa              48110     60.01     82.10    152.40       801.70        342.34

Line-by-Line Telemetry Analysis

• --cpu-method matrixprod: Exercises CPU arithmetic logic units and vector floating-point units using square matrix multiplication, drawing peak thermal design power (TDP).
• thermal zone: x86_pkg_temp 0: start 42.0 C, end 88.5 C: Socket 0 sustained a temperature surge of +46.5°C, approaching the maximum thermal throttling ceiling (90°C).
• thermal zone: x86_pkg_temp 1: start 41.0 C, end 76.0 C: Socket 1 ran 12.5°C cooler, indicating an asymmetrical heatsink seating issue or fan zone airflow obstruction in the server chassis.
• numa ... sys time: 152.40: Elevated system time during NUMA testing confirms substantial kernel overhead in cross-node memory page migration (migrate_pages) and memory bus arbitration.

Sysadmin Next Actions

1. Monitor processor frequency scaling using turbostat --interval 1 or cpupower monitor during the test to verify whether Socket 0 dropped below its base clock speed.
2. Pin latency-sensitive database worker threads to dedicated NUMA domains using numactl --interleave=all or strict core binding (--cpunodebind=0 --membind=0) to eliminate cross-socket interconnect latency.

3. Stress-Testing Filesystem Locking & Asynchronous Block I/O

Scenario

Before deploying an enterprise transactional database engine onto a high-performance NVMe array formatted with XFS, you need to verify whether the storage subsystem, Virtual File System (VFS) journal layer, and the kernel's asynchronous io_uring implementation experience lock contention or queue stalls under heavy parallel write workloads.

Exact Production Invocation

stress-ng --io-uring 8 --fallocate 4 --sync-file 4 --hdd 4 \
  --hdd-bytes 4G --hdd-opts direct,sync \
  --temp-path /mnt/nvme-pool/test-harness \
  --timeout 60s --metrics-brief

Realistic Terminal Output

stress-ng: info:  [52109] setting to a 60 second run per stressor
stress-ng: info:  [52109] dispatching hogs: 4 hdd, 8 io-uring, 4 fallocate, 4 sync-file
stress-ng: info:  [52109] successful run completed in 60.02s
stress-ng: metrc: [52109] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [52109]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [52109] io-uring        1248900     60.00     12.10    142.80     20815.00        13429.03
stress-ng: metrc: [52109] fallocate        310450     60.01      1.02    182.10      5173.30         2798.11
stress-ng: metrc: [52109] sync-file         18210     60.01      0.40    198.30       303.45          91.64
stress-ng: metrc: [52109] hdd               45200     60.02      2.10     84.10       753.08          524.36

Line-by-Line Telemetry Analysis

• --temp-path /mnt/nvme-pool/test-harness: Confines temporary file creation and block allocation to the physical mount point under evaluation, protecting operating system partitions.
• --hdd-opts direct,sync: Enforces O_DIRECT and O_SYNC flags on disk operations, bypassing the Linux page cache to push every transaction directly to non-volatile controller hardware.
• io-uring ... bogo ops/s 20815.00: Validates that kernel submission queue entries (SQEs) and completion queue entries (CQEs) maintained high throughput with minimal user-space context-switching cost (12.10 seconds user time).
• sync-file ... sys time 198.30: The high kernel execution time in sync-file reflects the physical barrier latency of NVMe flush operations writing dirty metadata to the XFS write-ahead journal.

Sysadmin Next Actions

Track NVMe queue saturation using iostat -xz 1 during the test run. If device utilization reaches 100% while average wait times (await) diverge significantly between buffered and direct writes, adjust filesystem mount parameters (such as noatime,nodiratime,logbufs=8,logbsize=256k) to mitigate journal contention.

4. Inducing Synthetic IPC Contention & Context-Switch Spikes

Scenario

A microservices platform experiences intermittent latency anomalies when handling surges in inter-service RPC traffic. You need to benchmark the Linux scheduler under heavy lock contention across POSIX pipes, System V semaphores, and fast userspace mutexes (futex), while validating dynamic tracing instrumentation.

Exact Production Invocation

stress-ng --pipe 16 --sem-sysv 8 --futex 16 --switch 32 \
  --timeout 45s --metrics-brief

Realistic Terminal Output

stress-ng: info:  [61400] setting to a 45 second run per stressor
stress-ng: info:  [61400] dispatching hogs: 16 pipe, 8 sem-sysv, 16 futex, 32 switch
stress-ng: info:  [61400] successful run completed in 45.01s
stress-ng: metrc: [61400] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [61400]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [61400] pipe             894100     45.00     14.20    490.10     19868.88        3942.29
stress-ng: metrc: [61400] sem-sysv         412000     45.01      8.40    310.20      9153.52        1293.15
stress-ng: metrc: [61400] futex            654100     45.00     28.10    410.80     14535.55        1490.32
stress-ng: metrc: [61400] switch          1245000     45.00     12.00    980.50     27666.66        1254.40

Line-by-Line Telemetry Analysis

• --switch 32: Spawns 32 worker threads that continuously yield processor execution via sched_yield(), forcing the kernel runqueue to repeatedly recalculate task priorities and switch memory address spaces.
• switch ... sys time 980.50: The substantial accumulation of system time across workers demonstrates that CPU cores spent the majority of their cycles executing context transitions rather than application logic.
• futex ... bogo ops/s 14535.55: Exercises kernel hash buckets for sleeping threads. Disproportionately low throughput relative to available cores points to lock hash collisions within the kernel's internal futex queues.

Sysadmin Next Actions

Execute an eBPF off-CPU analysis tool such as offcputime-bpfcc or trace scheduler wait times with perf sched record -- sleep 10 followed by perf sched latency. This helps determine whether tuning kernel parameters such as kernel.sched_migration_cost_ns is necessary to prevent excessive thread migration across CPU cores.

5. Executing Bare-Metal Hardware Burn-In & Memory Integrity Verification

Scenario

You are commissioning a fleet of newly racked physical hypervisors equipped with multi-terabyte DDR5 ECC memory and multi-socket CPUs. You must perform an uncompromising pre-production burn-in to uncover defective memory modules, floating-point calculation errors, and power delivery issues before placing the hardware into the production cluster, following standard hardware qualification methodologies.

Exact Production Invocation

stress-ng --matrix-prod 0 --fp-error 0 --cache 0 \
  --vm 0 --vm-bytes 92% --verify \
  --thermal-zones --timeout 300s --metrics-brief

Realistic Terminal Output

stress-ng: info:  [78001] setting to a 300 second run per stressor
stress-ng: info:  [78001] dispatching hogs: 128 matrix-prod, 128 fp-error, 128 cache, 128 vm
stress-ng: info:  [78001] vm: allocated 1.84 TB of virtual memory
stress-ng: info:  [78001] thermal zone: x86_pkg_temp 0: start 38.0 C, end 82.0 C (delta 44.0 C)
stress-ng: info:  [78001] thermal zone: x86_pkg_temp 1: start 37.5 C, end 81.5 C (delta 44.0 C)
stress-ng: info:  [78001] successful run completed in 300.08s
stress-ng: metrc: [78001] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [78001]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [78001] matrix-prod     4812000    300.01  38110.20      8.10     16039.46         421.11
stress-ng: metrc: [78001] fp-error        8912400    300.02  38200.10      4.20     29706.01         777.60
stress-ng: metrc: [78001] cache           2140000    300.08  37910.40     82.10      7131.43         187.93
stress-ng: metrc: [78001] vm              1984500    300.08   4820.10   9810.40      6613.23         452.82

Line-by-Line Telemetry Analysis

• --matrix-prod 0 --fp-error 0 ...: Passing 0 automatically scales worker processes to match the online logical core count (128 threads across both CPU sockets).
• --vm-bytes 92%: Allocates and populates 92% of total physical RAM (1.84 TB), ensuring comprehensive physical address space coverage across all memory channels.
• --verify: Activates strict in-line mathematical verification. After every matrix calculation or memory pass, the stressor validates results against known invariants; any mismatched bit indicates memory corruption, voltage instability, or arithmetic logic drift.
• successful run completed in 300.08s: All 128 threads completed verification cycles across 300 seconds without encountering mathematical discrepancies or data integrity faults.

Sysadmin Next Actions

Check the Linux kernel's Machine Check Architecture (MCA) and Error Detection and Correction (EDAC) logging:

dmesg -T | grep -E -i "(edac|mce|hardware error)"
ras-mc-ctl --errors

If zero uncorrected errors (UE) or corrected errors (CE) are logged in the operating system logs or the Baseboard Management Controller (BMC) System Event Log, the machine is verified and ready for production provisioning.

What Can Go Wrong: Hazards, Safety Controls, and Recovery

Subjecting infrastructure to intentional saturation carries inherent operational risks. When applied without proper safeguards, synthetic workloads can induce host lockups, trigger disk exhaustion, or cause ungraceful shutdowns.

1. Unbounded Memory Allocation Leading to Host Lockups

• The Hazard: Invoking --vm without specifying explicit boundaries (or setting --vm-bytes 100%) causes aggressive memory exhaustion. The Linux kernel enters emergency synchronous page reclamation, freezing filesystem caches and rendering remote management connections like SSH unresponsive.
• The Defense: Always enforce bounded limits using explicit values (such as --vm-bytes 4G) or constrain executions within systemd cgroups. Always include the --oomable flag, which instructs stress-ng to gracefully record and handle process termination by the OOM killer rather than crashing.
• Dry-Run Verification: Validate the parsed memory footprint before spawning processes: bash stress-ng --vm 4 --vm-bytes 2G --dry-run

2. Disk and Inode Exhaustion on the Root Filesystem

• The Hazard: Running storage stressors (--hdd, --fallocate) without specifying a destination directory will write temporary files directly into /tmp. If /tmp resides on the root filesystem, this can consume all remaining disk space or inode tables, disrupting system logging, database processes, and network socket creation.
• The Defense: Always isolate file operations to a dedicated testing mount using --temp-path /path/to/scratch.
• Recovery Action: If a test run is aborted prematurely and temporary files remain, sweep the scratch directory: bash find /mnt/scratch/ -maxdepth 1 -name "stress-ng-*" -delete

3. Orphaned Stressors and Uninterruptible Sleep States

• The Hazard: Forcibly terminating stress-ng with ungraceful signals (such as kill -9 <pid>) can leave worker child processes orphaned. If these workers are waiting on unresponsive storage controllers in kernel space (the uninterruptible D sleep state), they cannot be terminated immediately and will continue consuming system resources.
• The Defense: Always execute workloads with an explicit --timeout <time>. The parent process maintains robust signal handlers designed to coordinate the graceful teardown of all child workers.
• Recovery Action: If processes become orphaned, signal the entire process group cleanly: bash killall --signal SIGALRM stress-ng Allow ten seconds for child processes to unwind kernel locks before issuing a forceful pkill -9 stress-ng.

Performance Telemetry Analysis Reference

The table below maps primary stress-ng stressor classes to their underlying kernel subsystems and key diagnostic telemetry indicators:

Today's Takeaway

System resilience and hardware reliability should never be matters of blind faith. Right now, log in to a development machine or staging server and run a safe, fully bounded 30-second subsystem baseline audit: stress-ng --cpu 2 --vm 1 --vm-bytes 512M --io 1 --timeout 30s --metrics-brief. This single command exercises CPU arithmetic logic, cycles half a gigabyte of virtual memory through the memory management unit, tests file synchronization barriers, and outputs a concrete performance metric (bogo ops/s). By incorporating deterministic, metric-driven load generation into your standard server provisioning and testing workflows, you systematically uncover hardware flaws, tune kernel bottlenecks, and validate resource boundaries long before they can trigger a 2am production emergency.