Df: Auditing Filesystem Capacity, Diagnosing Inode Exhaustion, and Resolving Storage Discrepancies in Production

The baffling part is that the storage array had several terabytes of headroom added just weeks prior. Hastily logging into the bastion host, the engineer runs a directory sweep using du to pinpoint the bloated log or runaway dump file, only to watch in disbelief as the tool reports that barely forty percent of the disk is consumed. The physical drive appears half empty, yet the operating system categorically refuses to write a single additional byte to the database journal.

This disconnect between what you think is on your drive and what the operating system will allow you to write is one of the most frustrating dilemmas in systems administration. The discrepancy arises because storage is not merely a bucket of raw bytes—it is a complex ecosystem of kernel accounting ledgers, metadata tables, and administrative reservations.

When outages strike, file-scanning utilities can take hours to crawl nested directory trees, only to return partial or misleading numbers. The utility engineered to pierce this confusion instantly, querying the kernel’s live storage ledger in constant time, is the venerable Unix tool df (disk free).

For immediate, uncluttered visibility across any Linux or Unix host, the single most practical command to run right away is:

df -hT

Filesystem     Type      Size  Used Avail Use% Mounted on
udev           devtmpfs   32G     0   32G   0% /dev
tmpfs          tmpfs     6.3G  1.8M  6.3G   1% /run
/dev/nvme0n1p2 ext4      916G  412G  458G  48% /
tmpfs          tmpfs      32G   64M   32G   1% /dev/shm
tmpfs          tmpfs     5.0M     0  5.0M   0% /run/lock
/dev/nvme0n1p1 vfat      511M  6.1M  505M   2% /boot/efi
/dev/nvme1n1p1 xfs       1.8T  1.2T  640G  66% /mnt/data

This baseline invocation scales raw byte counts into intuitive, human-readable gigabytes and terabytes (-h) while printing the underlying filesystem driver (-T), giving you a complete topological overview of your disks within milliseconds.

1. What It Does in Plain English

At its core, df reports the aggregate amount of available, consumed, and reserved space across all mounted filesystems. Unlike file-scanning utilities such as du (disk usage) that calculate consumption by painstakingly traversing every individual file and directory on disk, df bypasses the directory tree entirely.

Instead, df communicates directly with the operating system kernel to retrieve real-time, global filesystem state in constant time ($\mathcal{O}(1)$). It acts as the definitive arbiter of whether a storage volume can accept new write operations, tracking not only raw byte capacity, but also the structural metadata containers known as indexing nodes (inodes) and privileged reserve buffers.

2. Kernel Architecture and Storage Metrology

To understand why df responds instantaneously while du can grind your disks to a halt during heavy I/O, you have to look at the interface between the Unix Virtual Filesystem (VFS) and the underlying storage drivers.

The statvfs(2) Mechanics

When you invoke df, it issues the statvfs(2) or legacy statfs(2) system call against the target mount point or block device. The Linux kernel handles this request by reading the filesystem's in-memory superblock—a control block maintained dynamically by filesystem drivers like ext4, XFS, or Btrfs.

The struct statvfs buffer returned to user space contains the core metrological counters defined by the POSIX.1-2017 Specification for df:

struct statvfs {
    unsigned long  f_bsize;    /* Filesystem block size */
    unsigned long  f_frsize;   /* Fundamental allocation unit (fragment size) */
    fsblkcnt_t     f_blocks;   /* Total data blocks in filesystem */
    fsblkcnt_t     f_bfree;    /* Total free blocks (including reserved) */
    fsblkcnt_t     f_bavail;   /* Free blocks available to unprivileged users */
    fsfilcnt_t     f_files;    /* Total file nodes (inodes) */
    fsfilcnt_t     f_ffree;    /* Total free file nodes */
    fsfilcnt_t     f_favail;   /* Free file nodes available to unprivileged */
    unsigned long  f_fsid;     /* Filesystem ID */
    unsigned long  f_flag;     /* Mount flags (e.g., ST_RDONLY, ST_NOSUID) */
    unsigned long  f_namemax;  /* Maximum filename length */
};

Because these counters are updated atomically in the superblock during allocation and deallocation operations, df maintains an algorithmic time complexity of $\mathcal{O}(1)$. It does not parse individual directory entries; it reads the running ledger already kept by the kernel.

The Mathematical Formulas of Filesystem Capacity

The metrics displayed in terminal tables by df are calculated through precise algebraic formulas:

$$\text{Capacity}_{\text{Total}} = f_blocks \times f_frsize$$

$$\text{Capacity}_{\text{Free (Raw)}} = f_bfree \times f_frsize$$

$$\text{Capacity}_{\text{Available}} = f_bavail \times f_frsize$$

$$\text{Capacity}_{\text{Reserved}} = (f_bfree - f_bavail) \times f_frsize$$

The calculation of the percentage capacity utilized introduces a mathematical nuance that often catches engineers off guard. In standard GNU Coreutils df implementations, the percentage is not calculated relative to raw total blocks ($f_blocks$), but rather relative to the space actually accessible to unprivileged processes:

$$\text{Blocks}_{\text{Used}} = f_blocks - f_bfree$$

$$\text{Blocks}{\text{Resolvable}} = \text{Blocks}{\text{Used}} + f_bavail$$

$$\text{Percentage}{\text{Used}} = \left\lceil \frac{\text{Blocks}{\text{Used}}}{\text{Blocks}_{\text{Resolvable}}} \times 100 \right\rceil$$

Consequently, when $f_bavail$ drops to zero, the reported usage reaches 100%, even though $f_bfree$ may still contain hundreds of megabytes of blocks reserved exclusively for the root administrative user.

Inode Geometry and Exhaustion Physics

A Unix filesystem divides storage into two distinct realms: data blocks (which store the actual contents of files) and index nodes or inodes (which store metadata, permissions, ownership, timestamps, and pointers to physical data blocks).

In traditional filesystems like ext4, the total number of inodes ($f_files$) is fixed when the filesystem is created, based on a bytes-per-inode heuristic ($I_{\text{ratio}}$):

$$N_{\text{inodes}} = \frac{\text{Size}{\text{partition}}}{I{\text{ratio}}}$$

When applications produce millions of microscopic files—such as temporary session tokens, microservice trace dumps, or uncleaned email queues—every single inode can be consumed ($f_ffree = 0$) while data block utilization remains below 30%. When this happens, any new file creation fails instantly with ENOSPC, because the operating system cannot allocate an index node to register the file, regardless of how many gigabytes of raw block capacity remain empty.

3. Core Flags and Quick-Start Reference

The modern POSIX and GNU variants of df provide flags to format numbers, exclude virtual or synthetic mounts, and inspect inode tables.

4. Five Real-World Production Scenarios

Scenario 1: Filtering Ephemeral and Virtual Storage Noise in Containerised Fleets

The Scenario

On a busy Kubernetes worker node or Docker host running dozens of microservices, running an unfiltered df -h floods your terminal with dozens of synthetic RAM disks (tmpfs), loopback mounts (squashfs), and dynamic container overlays (overlay). This screen-filling noise obscures physical drive usage and slows down incident triage during an active outage.

Execution Command

df -hT -x tmpfs -x devtmpfs -x overlay -x squashfs

Terminal Output

Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda1      ext4   49G   38G  8.6G  82% /
/dev/sdb1      xfs   1.9T  1.4T  520G  73% /var/lib/containerd
/dev/sdc1      xfs   3.8T  3.1T  700G  82% /data/storage

Analytical Breakdown

• Filesystem: Isolates physical block devices (/dev/sda1, /dev/sdb1, /dev/sdc1), filtering out transient container layers and RAM disks.
• Type: Confirms that only persistent journaled storage engines (ext4, xfs) are being audited.
• Size vs Used vs Avail: Displays accurate hardware boundaries without mathematical distortions from nested container mount namespaces.
• Use%: Highlights the true storage pressure across the root OS, the container runtime directory, and persistent application volumes.

What the Administrator Does Next

1. If the container runtime volume (/var/lib/containerd) crosses 80% utilization, trigger an immediate container runtime garbage collection: bash crictl rmi --prune
2. Verify that application log rotation policies on /data/storage are actively truncating stale container logs.

Scenario 2: Triaging Silent ENOSPC Outages via Inode Auditing

The Scenario

An API gateway starts failing customer requests with HTTP 500 errors and logging java.io.IOException: No space left on device. The sysadmin runs df -h and sees that the root disk /dev/nvme0n1p2 has 320 Gigabytes of free space (only 38% used). Despite vast block reserves, the OS refuses to create new files.

Execution Command

df -i /

Terminal Output

Filesystem       Inodes   IUsed   IFree IUse% Mounted on
/dev/nvme0n1p2 61054976 61054976       0  100% /

Analytical Breakdown

• Inodes ($f_files$): Denotes the absolute physical ceiling of indexing nodes allocated when the filesystem was formatted (61,054,976 nodes).
• IUsed: Shows the total number of inodes currently allocated to files, directories, symlinks, and special sockets (61,054,976).
• IFree ($f_ffree$): The number of unallocated inodes has reached exactly zero (0).
• IUse%: Inode utilization is at 100%. Every new file creation request (open(..., O_CREAT), mkdir, touch) fails immediately with ENOSPC.

What the Administrator Does Next

1. Run a directory traversal to pinpoint which directory is hoarding millions of microscopic files without crossing into other mount points: bash find / -xdev -printf '%h\n' | sort | uniq -c | sort -k1 -nr | head -n 10
2. After discovering that an unconfigured mail or monitoring spooler has created 20 million tiny files in /var/spool/clientmqueue, purge the directory using find -delete to avoid hitting shell argument limits (ARG_MAX): bash find /var/spool/clientmqueue -type f -delete
3. Re-run df -i / to verify that IFree has recovered and inode pressure has subsided.

Scenario 3: Resolving Phantom Storage Discrepancies and Unlinked File Descriptors

The Scenario

A high-traffic web proxy triggers an urgent storage alert on /var/log. The administrator logs in and runs rm /var/log/nginx/access.log, expecting to free 40 GB of space. However, df -h /var/log stubbornly reports that the disk is still 100% full (0 bytes available), while du -sh /var/log insists that only 1.8 GB of files exist in the entire folder.

Diagnostic Verification Pipeline

df -h /var/log
du -sh /var/log
lsof +L1 /var/log

Terminal Output

# df -h /var/log
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        99G   94G     0 100% /var/log

# du -sh /var/log
1.8G    /var/log

# lsof +L1 /var/log
COMMAND   PID USER   FD   TYPE DEVICE   SIZE/OFF NLINK      NODE NAME
nginx   14209 root    7w   REG   8,17 98956042240     0 144179214 /var/log/nginx/access.log (deleted)

Analytical Breakdown

• The df vs du Discrepancy: Under POSIX standards, executing rm calls unlink(2), which deletes the filename from the directory structure. But if an active process still has that file open, the kernel cannot release the underlying data blocks until the process closes its file descriptor.
• NLINK (0): Confirms that no directory entries reference this inode.
• SIZE/OFF (98956042240): Confirms that 92 GB of disk space remains held hostage by process ID 14209 (nginx).
• (deleted): The filesystem marks the name as unlinked, but the superblock block counters will not decrease until the process descriptor is closed or truncated.

What the Administrator Does Next

1. Do not abruptly kill critical services with kill -9, which can corrupt state or drop live client connections.
2. Locate the file descriptor from the lsof(8) trace (in this case, file descriptor 7w under PID 14209).
3. Truncate the file descriptor safely in place through the kernel's /proc virtual interface, freeing the allocated blocks without restarting the service: bash : > /proc/14209/fd/7
4. Run df -h /var/log to confirm that the space has been instantly reclaimed.

Scenario 4: Constructing Resilient Storage Alerting via Structured Field Output

The Scenario

Traditional shell scripts often parse df output by piping tabular text into awk or sed. This approach is fragile: long device names cause table columns to wrap, differing operating system distributions shift output formats, and system locale settings alter header names. To build rock-solid monitoring watchdogs, administrators need deterministic, machine-readable output.

Execution Command

df -h --output=pcent,avail,target -x tmpfs -x devtmpfs

Terminal Output

Use% Avail Mounted on
 48%  458G /
  2%  505M /boot/efi
 66%  640G /mnt/data
 91%   18G /var/log

Analytical Breakdown

• --output=pcent,avail,target: Explicitly instructs GNU df to output only the requested columns in a deterministic sequence, regardless of console width or device name length.
• pcent: Emits the integer usage percentage followed by the % symbol for reliable regex matching.
• avail: Displays human-readable availability numbers.
• target: Emits the absolute path of the mount point, avoiding naming ambiguities with device mappers (/dev/mapper/...).

What the Administrator Does Next

Deploy a robust POSIX monitoring script that checks for physical mount points exceeding 90% utilization and logs structured critical events directly to the system journal:

#!/usr/bin/env bash
set -euo pipefail

THRESHOLD=90

df --output=pcent,target -x tmpfs -x devtmpfs -x overlay | tail -n +2 | while read -r pcent target; do
    numeric_pcent="${pcent%%%}"
    if [ "${numeric_pcent}" -ge "${THRESHOLD}" ]; then
        logger -p local0.crit -t STORAGE_MONITOR "Mount point ${target} has reached critical saturation: ${pcent} capacity."
    fi
done

Scenario 5: Auditing and Managing Reserved Root Space During Critical Logins

The Scenario

A dedicated logging partition mounted at /var/audit on an ext4 filesystem runs out of space. Applications report write failures, yet an administrative engineer can still log in, run diagnostics, and write test files. The operator must audit the root space reservation to maximize usable storage without endangering system stability.

Diagnostic Verification Pipeline

df -h /var/audit
tune2fs -l /dev/sdd1 | grep -E 'Block count|Reserved block count|Block size'

Terminal Output

# df -h /var/audit
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdd1       1.8T  1.8T     0 100% /var/audit

# tune2fs -l /dev/sdd1 | grep -E 'Block count|Reserved block count|Block size'
Block count:              488378624
Reserved block count:      24418931
Block size:                    4096

Analytical Breakdown

• Avail (0) vs Used (1.8T): df reveals that unprivileged application processes have zero bytes left for writes.
• tune2fs Superblock Inspection: Out of 488,378,624 total blocks, 24,418,931 blocks are reserved exclusively for the root superuser ($S_{\text{reserved}}$).
• The 5% Default: The standard ext4 filesystem configuration reserves a default 5% of all blocks for UID 0. On a modern multi-terabyte volume, a 5% buffer locks away nearly 100 GB of capacity ($24,418,931 \times 4096 \approx 100\text{ GB}$) from standard applications.

What the Administrator Does Next

1. On dedicated, non-root application and data partitions, reclaim this reserved space on live filesystems without taking the volume offline using tune2fs(8): bash tune2fs -m 1 /dev/sdd1
2. On pure archive or media partitions where system services do not depend on root emergency headroom, dial the reservation down to zero: bash tune2fs -m 0 /dev/sdd1
3. Re-check df -h /var/audit to confirm that nearly 100 GB of storage is immediately available for writes: text Filesystem Size Used Avail Use% Mounted on /dev/sdd1 1.8T 1.7T 98G 95% /var/audit

5. What Can Go Wrong: Failure Modes, Subtle Traps, and Mitigations

1. The Kernel D-State Lockup: Unresponsive Network Filesystems

The Hazard

Running a bare df command on a server with remote mounts (NFS, CIFS, or CephFS) during an upstream network outage can freeze your terminal completely. The df process enters an un-interruptible sleep state (labeled as D in process tables) while statvfs() waits indefinitely for an RPC response. In this state, the command cannot be killed—even with kill -9—and scheduled monitoring scripts will pile up, consuming process slots and driving up system load averages.

Mitigation Strategy

Always isolate local filesystems from network mounts. If remote filesystems must be evaluated, enforce a strict execution deadline:

# Audit local filesystems only
df -hl

# Enforce an execution deadline against remote network shares
timeout 5 df -h

2. Binary (-h) versus Decimal (-H) Discrepancies in Sizing Audits

The Hazard

Hard drive manufacturers advertise drive capacities using the International System of Units (SI), where $1\text{ Gigabyte (GB)} = 10^9\text{ bytes} = 1,000,000,000\text{ bytes}$. Operating system kernels, however, manage storage boundaries using base-2 binary units, where $1\text{ Gibibyte (GiB)} = 2^{30}\text{ bytes} = 1,073,741,824\text{ bytes}$.

When you inspect a newly installed 1 TB SSD with df -h, it reports roughly $931\text{ GiB}$ of total capacity. If an administrator mistakenly assumes that 70 GB of storage has vanished due to bad partition alignment, valuable time is wasted troubleshooting a phantom problem.

(The ~7.37% delta represents a difference in mathematical units, not missing disk space).

Mitigation Strategy

Use df -h when inspecting memory, filesystem buffers, and kernel structures. Use df -H when reconciling storage figures with cloud vendor billing invoices or hardware datasheets.

6. Comprehensive Flag and Behavior Matrix

7. Production Storage Sentinel Implementation

To eliminate fragile manual parsing and protect your infrastructure against cascading disk outages, deploy the following production-ready shell sentinel. It audits block saturation, inode depletion, and deleted descriptor leaks, emitting structured JSON logs to standard output:

#!/usr/bin/env bash
# ==============================================================================
# SCRIPT: storage_sentinel.sh
# PURPOSE: Continuous filesystem block, inode, and descriptor health telemetry.
# COMPLIANCE: POSIX / Modern GNU Environments
# ==============================================================================
set -euo pipefail

BLOCK_THRESHOLD=85
INODE_THRESHOLD=85

log_alert() {
    local severity="$1"
    local check_type="$2"
    local mount="$3"
    local value="$4"
    local message="$5"

    printf '{"timestamp":"%s","severity":"%s","check":"%s","mount":"%s","utilization":"%s","message":"%s"}\n' \
        "$(date --iso-8601=seconds)" "${severity}" "${check_type}" "${mount}" "${value}" "${message}"
}

audit_storage() {
    # Extract block and inode metrics deterministically
    df -P -h --output=target,pcent,ipcent -x tmpfs -x devtmpfs -x overlay -x squashfs | tail -n +2 | while read -r target pcent ipcent; do
        block_usage="${pcent%%%}"
        inode_usage="${ipcent%%%}"

        # Evaluate Block Capacity
        if [ "${block_usage}" -ge "${BLOCK_THRESHOLD}" ]; then
            log_alert "CRITICAL" "BLOCK_SATURATION" "${target}" "${pcent}" "Block allocation exceeded threshold."
        fi

        # Evaluate Inode Capacity
        if [ "${inode_usage}" != "-" ] && [ "${inode_usage}" -ge "${INODE_THRESHOLD}" ]; then
            log_alert "CRITICAL" "INODE_SATURATION" "${target}" "${ipcent}" "Inode table allocation exceeded threshold."
        fi
    done
}

audit_phantom_leaks() {
    if command -v lsof >/dev/null 2>&1; then
        # Audit unlinked files consuming > 500MB
        lsof +L1 -F size -F name -F pid 2>/dev/null | awk '
            /^p/ { pid=substr($0,2) }
            /^s/ { size=substr($0,2) }
            /^n/ { 
                name=substr($0,2);
                if (size > 524288000) {
                    printf "{\"timestamp\":\"%s\",\"severity\":\"WARNING\",\"check\":\"PHANTOM_LEAK\",\"pid\":\"%s\",\"size_bytes\":%s,\"file\":\"%s\"}\n", 
                        strftime("%Y-%m-%dT%H:%M:%SZ"), pid, size, name
                }
            }
        '
    fi
}

main() {
    audit_storage
    audit_phantom_leaks
}

main "$@"

8. Today's Takeaway

To immediately improve the health and visibility of your own systems, open a terminal on your primary workstation or server right now and run:

df -hT -x tmpfs -x devtmpfs -x overlay

Review the output to establish a clean, noise-free baseline of your true physical block usage. Follow it up immediately with df -i / to confirm that your root filesystem’s inode tables are not quietly filling up under the weight of forgotten application caches, package manager remnants, or temporary files. Storage capacity is multidimensional: a disk with hundreds of gigabytes of unallocated space remains completely unusable if its metadata tables are exhausted, its root reservation is misconfigured, or deleted log files are still held open by background processes. Mastering df ensures you will never be caught off guard when system availability is on the line.