Lslocks: Auditing Kernel Byte-Range Locks, Isolating Contended File Descriptors, and Resolving Storage Deadlocks in Production

The graphs show a bewildering paradox. Processor utilisation sits near zero percent; available system memory is virtually untouched; the network interfaces are idling along cleanly without a single dropped packet; and the storage subsystem reports lightning-fast sub-millisecond response times. On paper, the server is running effortlessly. In reality, dozens of transaction workers have ground to a dead halt, new processes crash the instant they spawn, and payments are blocked across the entire platform.

Standard diagnostic tools such as top, vmstat, and iostat offer no insight because the operating system is not starved of compute or storage capacity. Instead, it is caught in a silent, invisible traffic jam. Deep inside the Linux Virtual Filesystem (VFS), an orphaned background worker has crashed without relinquishing its exclusive lock on a shared data file. Every subsequent process has lined up behind it in an uninterruptible sleep state, waiting indefinitely for a green light that will never come.

When your systems are paralyzed by invisible contention, the single most practical diagnostic command you can run to cut through the fog is:

lslocks -u -o COMMAND,PID,TYPE,MODE,START,END,PATH

In a single stroke, this command filters out routine background reservations and surfaces only the contested locks causing the bottleneck, handing you the offending command name, its process ID, the locking mechanism, and the exact file path under contention. To master this critical diagnostic discipline and resolve complex storage deadlocks, systems engineers rely on lslocks(8).

What It Does in Plain English

Distributed as a core component of the foundational util-linux package, lslocks is an administrative utility that inspects, parses, and formats the Linux kernel's internal file-locking tables.

While familiar utilities like lsof examine file descriptors from the perspective of user-space processes—a heavyweight operation that requires crawling through thousands of process directories—lslocks interfaces directly with the kernel's Virtual Filesystem locking registry via the /proc/locks interface. This allows it to construct an immediate, system-wide snapshot of every active file lock, byte-range reservation, Open File Description lock, and network storage lease without imposing meaningful overhead on a struggling production system.

Core Flags and Quick Start

Before diving into complex production failures, it is helpful to understand the operational flags that make lslocks so versatile for both human operators and automated monitoring pipelines:

• -u, --unlocked: Filters the output to display exclusively those locks that currently block or conflict with other lock requests, immediately isolating contended resources.
• -J, --json: Formats the entire lock registry as structured JSON, ideal for ingestion by observability agents, telemetry pipelines, and automated triage scripts.
• -o, --output <list>: Specifies a custom, comma-delimited selection of columns (such as COMMAND,PID,TYPE,SIZE,MODE,M,START,END,PATH,INODE).
• -p, --pid <PID>: Restricts the inspection scope exclusively to file locks held by a designated process identifier.
• -r, --raw: Emits unformatted, space-delimited raw column data, preventing truncated file paths and column wrapping in narrow terminal windows.
• -n, --noheadings: Suppresses the column header row, making it straightforward to pipe output into text-processing tools like awk, sed, or grep.
• -b, --bytes: Displays file sizes and byte-range boundaries as exact integer byte counts rather than human-readable units (for example, 1048576 instead of 1M).

Quick Start: Baselining System Locks

To inspect all active kernel file locks across the operating system, execute lslocks with standard default formatting:

lslocks

COMMAND           PID  TYPE  SIZE MODE  M START          END PATH
systemd             1 FLOCK   48K READ  0     0            0 /run/systemd/seats/seat0
dockerd          1420 FLOCK    0K WRITE 0     0            0 /var/lib/docker/containerd/daemon.lock
dockerd          1420 FLOCK    0K WRITE 0     0            0 /run/docker.pid
postgres         2105 POSIX  128M WRITE 0    48           48 /var/lib/postgresql/16/main/global/pg_control
mysqld           2480 POSIX   64M WRITE 0     0            0 /var/lib/mysql/ibdata1
sshd             3892 POSIX  1.2M READ  0     0            0 /var/log/lastlog

This baseline view immediately reveals key diagnostic columns: the governing command (COMMAND), its process ID (PID), the locking mechanism (TYPE), file size (SIZE), access permissions (MODE), mandatory locking status (M), the specific byte boundaries (START and END), and the canonical filesystem location (PATH).

Architectural Breakdown: The Linux File Locking Subsystem

To interpret lock state during an outage with confidence, it helps to understand how the Linux Virtual Filesystem (VFS) handles concurrency under the hood, as outlined in the official docs.kernel.org Locking Documentation.

When an application initiates a locking system call, the kernel allocates an internal struct file_lock structure and attaches it to the target inode's lock chain. The Linux kernel recognises four primary locking paradigms:

1. Classic POSIX Byte-Range Locks (fcntl)

Invoked through the fcntl(2) system call using commands such as F_SETLK and F_SETLKW, POSIX locks allow processes to lock granular, contiguous byte segments within a file (reflected in the START and END columns). However, POSIX locks carry two critical architectural quirks: * Bound to Process ID: The lock is tied directly to a specific PID and inode pair. * Automatic Release on Close: If a process closes any open file descriptor pointing to an inode, all POSIX locks held by that process on that inode are instantly dropped—even if other independent descriptors remain open. POSIX locks are also not inherited across fork(2) calls.

2. BSD-Style Whole-File Locks (flock)

Originating in BSD Unix and exposed via flock(2), these locks apply strictly to the entire file (START: 0, END: 0). Unlike POSIX locks, BSD locks attach to the kernel's Open File Description (struct file). As a result, they are preserved across fork(2) boundaries—allowing parent and child workers to share the lock—and are not inadvertently released when an unrelated descriptor to the same file is closed.

3. Open File Description (OFD) Locks (fcntl)

Introduced in modern Linux kernels (3.15+) to solve the thread-safety issues of POSIX locks, OFD locks are created using fcntl commands like F_OFD_SETLK. They combine the granular byte-range precision of POSIX locks with the safer lifecycle semantics of BSD locks: they belong to the file description rather than the process ID, preventing unexpected lock loss in multi-threaded applications.

4. Kernel File Leases (fcntl Leases)

Configured using fcntl(fd, F_SETLEASE, ...), a lease allows a caching process (such as a local cache manager or network server daemons like Samba and NFS) to be notified via signals (SIGIO) whenever another process attempts to open, truncate, or overwrite the file. The kernel temporarily stalls the competing process for a configurable window (typically 45 seconds via /proc/sys/fs/lease-break-time), allowing the leaseholder to flush write caches or invalidate local data before yielding access.

Advisory versus Mandatory Locks

By default, Linux file locking is advisory. Processes must voluntarily check for locks before accessing files; an errant script or binary can bypass advisory locks entirely using standard read() or write() calls. Conversely, mandatory locks—flagged with a 1 under the M column in lslocks—instruct the VFS layer to intercept and block any conflicting read or write operation across the entire system. Enabling mandatory locking requires mounting filesystems with the -o mand option and applying specific file permissions (chmod g+s,g-x <file>).

Kernel-to-User State Mapping: /proc/locks

The lslocks binary functions by reading the /proc/locks virtual interface, where the kernel exposes raw locking records:

1: POSIX  ADVISORY  WRITE 2105 08:01:1048580 48 48
2: FLOCK  ADVISORY  WRITE 1420 08:01:2097153 0 EOF
3: OFD    ADVISORY  READ  5432 08:01:3145729 1024 2048

Each row maps to specific kernel metadata: 1. Global lock ordinal. 2. Lock family (POSIX, FLOCK, OFD, LEASE). 3. Enforcement regime (ADVISORY or MANDATORY). 4. Access mode (READ or WRITE). 5. Owner Process ID (PID). 6. Major/Minor device numbers and inode identifier (MAJOR:MINOR:INODE). 7. Start offset in bytes. 8. End offset in bytes (EOF indicates the end of the file).

lslocks reads this table, resolves device and inode numbers back to human-readable paths by cross-referencing mount tables and /proc file descriptors, and formats the output into clean, structured reports.

5 Real-World Production Scenarios

Scenario 1: Isolating Lock Contention in High-Concurrency Database Engines

The Context

An event ingestion microservice using an embedded SQLite database in Write-Ahead Logging (WAL) mode begins throwing continuous SQLITE_BUSY: database is locked exceptions. While read queries complete normally, all write transactions stall. The engineering team must determine whether the issue is caused by reader starvation or a stuck write transaction locking the shared-memory index file (-shm).

The Command

Run lslocks with byte-level reporting, custom columns, and the unblocking filter:

lslocks -u -b -o COMMAND,PID,TYPE,MODE,START,END,PATH

Realistic Terminal Output

COMMAND           PID TYPE  MODE   START    END PATH
telemetry_ingest 8124 POSIX WRITE    120    120 /var/data/telemetry.db-shm
telemetry_worker 8431 POSIX READ     120    120 /var/data/telemetry.db-shm
telemetry_worker 8432 POSIX READ     120    120 /var/data/telemetry.db-shm
telemetry_worker 8435 POSIX READ     120    120 /var/data/telemetry.db-shm

Line-by-Line Explanation

• telemetry_ingest (PID 8124): Holds an exclusive POSIX WRITE lock on byte 120 of /var/data/telemetry.db-shm. In SQLite WAL mode, byte 120 manages the write transaction lock for shared memory.
• telemetry_worker (PIDs 8431, 8432, 8435): Multiple background workers are blocked trying to obtain shared READ locks on byte 120 to validate index headers.
• START: 120 / END: 120: The lock covers exactly one byte, confirming fine-grained database index contention rather than an operating system-level whole-file freeze.

What the Admin Does Next

Inspect the call stack of the blocking process (PID 8124) to identify why its transaction has stalled:

cat /proc/8124/stack

If the stack trace indicates that the process is hung on an unhandled network socket or application deadlock, terminate the offending worker gracefully:

kill -15 8124

Once PID 8124 exits, the kernel immediately frees byte 120, allowing queued worker threads to resume normal execution.

Scenario 2: Triaging Deadlocked System Services and Scheduled Cron Jobs

The Context

A scheduled configuration management agent (puppet-agent or ansible-pull) fails to run across several application servers, reporting Active: failed (Result: exit-code). Manual restart attempts stall indefinitely because the agent's PID lockfile in /run/lock remains locked by an earlier, aborted execution.

The Command

Query lslocks and filter specifically for lock files under /run/lock and /var/run:

lslocks -o COMMAND,PID,TYPE,SIZE,MODE,M,START,END,PATH | grep -E "(/run/lock|/var/run)"

Realistic Terminal Output

COMMAND           PID  TYPE  SIZE MODE  M START END PATH
puppet           45091 FLOCK    0B WRITE 0     0   0 /run/lock/puppet-agent.lock
flock            49102 FLOCK    0B WRITE 0     0   0 /run/lock/backup-batch.lock

Line-by-Line Explanation

• puppet (PID 45091): An orphaned Puppet agent process holds an advisory whole-file (START: 0, END: 0) write lock via flock(2) on /run/lock/puppet-agent.lock.
• flock (PID 49102): A secondary shell wrapper holds an exclusive lock on /run/lock/backup-batch.lock, preventing subsequent backup cron jobs from running.
• M: 0: Confirms both locks are advisory.

What the Admin Does Next

Check whether process 45091 is active or stuck in an unworkable state:

ps -fp 45091 -o pid,ppid,stat,time,cmd

  PID  PPID STAT     TIME CMD
45091     1 D    00:12:44 /usr/bin/ruby /usr/bin/puppet agent --no-daemonize

The process is in an uninterruptible sleep state (STAT: D), stuck waiting on an unresponsive storage or network request. Check the wait channel and kernel stack:

cat /proc/45091/wchan
cat /proc/45091/stack

Once the underlying storage hang is addressed, terminate the stalled process to clear the flock reservation and restore scheduled runs:

kill -9 45091
systemctl restart puppet

Scenario 3: Auditing Clustered and Network Filesystem Leases

The Context

A fleet of application servers shares a clustered export over NFSv4 or Samba/CIFS. During peak traffic, write operations against shared configuration files stall. The team suspects that a client node acquired an exclusive file lease (F_SETLEASE) that was never cleanly broken, causing the local VFS layer to block subsequent write attempts while waiting on a lease-break acknowledgement.

The Command

Query the kernel lock registry specifically for active LEASE entries:

lslocks -o COMMAND,PID,TYPE,MODE,START,END,PATH | grep LEASE

Realistic Terminal Output

COMMAND           PID   TYPE  MODE  START END PATH
smbd             184902 LEASE WRITE     0   0 /mnt/shared_storage/configs/cluster_manifest.json
nfsd             190114 LEASE READ      0   0 /mnt/shared_storage/assets/catalog_index.bin

Line-by-Line Explanation

• smbd (PID 184902): The local Samba daemon holds an exclusive WRITE lease on /mnt/shared_storage/configs/cluster_manifest.json. Any other process attempting to access this file locally will be suspended for up to 45 seconds while the kernel attempts to notify the remote client.
• nfsd (PID 190114): The NFS daemon holds a shared READ lease on /mnt/shared_storage/assets/catalog_index.bin. Readers can access the file unimpeded, but any write or truncate attempt will trigger a lease-break sequence.
• START: 0 / END: 0: Leases apply exclusively to entire files.

What the Admin Does Next

Review the active kernel lease-break timeout setting:

cat /proc/sys/fs/lease-break-time

If a client daemon fails to respond to lease-break requests during an active incident, temporarily reduce the timeout window or signal the daemon:

# Temporarily drop lease break timeout during triage to 5 seconds
sysctl -w fs.lease-break-time=5

# Signal the holding daemon to force delegation processing
kill -SIGIO 184902

Scenario 4: Programmatic Telemetry Ingestion with JSON Output

The Context

Site Reliability Engineers need an automated health check to continuously monitor shared storage volumes. The watchdog daemon must detect any file lock held continuously for more than 300 seconds and emit structured telemetry events to observability platforms like Datadog, Prometheus, or Elasticsearch.

The Command

Run lslocks with complete JSON formatting and exact byte reporting:

lslocks -J -b -o COMMAND,PID,TYPE,SIZE,MODE,M,START,END,PATH,INODE

Realistic Terminal Output

{
   "locks": [
      {
         "command": "containerd",
         "pid": 982,
         "type": "FLOCK",
         "size": 0,
         "mode": "WRITE",
         "m": false,
         "start": 0,
         "end": 0,
         "path": "/run/containerd/containerd.pid",
         "inode": 1049
      },
      {
         "command": "state_engine",
         "pid": 58210,
         "type": "OFD",
         "size": 1073741824,
         "mode": "WRITE",
         "m": false,
         "start": 4096,
         "end": 8191,
         "path": "/var/lib/engine/state.dat",
         "inode": 4194308
      }
   ]
}

Line-by-Line Explanation

• "type": "OFD": Indicates that state_engine uses modern Open File Description locking, tying the lock to the file description rather than the process ID lifecycle.
• "start": 4096, "end": 8191: The engine has locked exactly one 4KB memory page (bytes 4096 through 8191), permitting concurrent reads and writes to all other segments of /var/lib/engine/state.dat.
• "inode": 4194308: Provides the explicit filesystem inode for cross-referencing with storage metrics.

What the Admin Does Next

Integrate lslocks into an automated triage pipeline with jq to isolate long-running write locks on critical state files:

lslocks -J -b -o COMMAND,PID,TYPE,MODE,START,END,PATH | jq '
  .locks[] | select(.mode == "WRITE" and (.path | test("/var/lib/engine")))
'

When an anomalous lock is detected, the automated pipeline triggers a high-priority alert with the exact PID, file path, and byte boundaries.

Scenario 5: Inode-to-Path Resolution and Target Process Isolation

The Context

An automated failover script attempting to unmount a storage volume (/srv/storage) fails with the error umount: /srv/storage: target is busy. Running lsof generates thousands of noisy lines referencing mapped shared libraries, making it difficult to find the specific process actively locking the volume.

The Command

Run lslocks filtered specifically for the target mount directory:

lslocks -o COMMAND,PID,TYPE,MODE,PATH | grep "/srv/storage"

Realistic Terminal Output

COMMAND           PID TYPE  MODE  PATH
data_sync       77102 POSIX WRITE /srv/storage/archive/wal_journal.bin
log_collector   77440 FLOCK READ  /srv/storage/incoming/syslog.pipe

Line-by-Line Explanation

• data_sync (PID 77102): Holds an exclusive POSIX WRITE lock on /srv/storage/archive/wal_journal.bin.
• log_collector (PID 77440): Holds a shared BSD FLOCK READ lock on /srv/storage/incoming/syslog.pipe.
• Because these active locks are registered directly in the kernel lock table, any attempt to run umount /srv/storage will be blocked with EBUSY.

What the Admin Does Next

Inspect the open file descriptors of the locking process in /proc:

ls -l /proc/77102/fd | grep "/srv/storage"

lrwx------ 1 root root 64 Aug 19 02:28 7 -> /srv/storage/archive/wal_journal.bin

Process 77102 holds the target file open on file descriptor 7. Carry out a clean termination and unmount sequence:

# 1. Gracefully terminate the holding processes
kill -SIGTERM 77102 77440

# 2. Verify that locks have cleared
lslocks -p 77102,77440

# 3. Safely unmount the volume
umount /srv/storage

Comparative Matrix: Tooling Selection for Storage & Concurrency Triage

When triaging storage contention and process deadlocks, engineers often choose between lslocks, lsof, fuser, and flock. Each serves a distinct purpose:

When to Use Which Tool

• Use lslocks when diagnosing deadlocks, lock contention, database WAL stalls, and byte-range conflicts.
• Use lsof(8) when identifying which process is holding an unlinked file open to reclaim deleted disk space.
• Use fuser(1) when needing to quickly list or kill all processes with open handles on a specific mount point.
• Use flock(1) inside cron jobs and systemd unit scripts to prevent overlapping executions of batch tasks.

What Can Go Wrong: Pitfalls, Misconceptions, and Failure Modes

1. The POSIX close() Invalidation Trap

The most frequent source of confusion when debugging lock lifecycles involves the aggressive invalidation semantics of classic POSIX locks (F_SETLK).

Engineers observing missing locks in lslocks often suspect kernel issues, when in reality a secondary application thread performed a quick stat() or read check and closed its descriptor, unintentionally stripping the locks held by its sibling threads. Modern codebases resolve this by migrating to Open File Description (OFD) locks (F_OFD_SETLK).

2. Path Resolution Failures (Deleted Files and Unnamed Inodes)

When an application opens an anonymous temporary file with open(..., O_TMPFILE) or acquires a lock on a file that is subsequently deleted (rm), lslocks will either show an empty string or append (deleted) to the path:

COMMAND           PID TYPE  MODE  START END PATH
cache_mgr        9912 POSIX WRITE     0   0 /tmp/cache.tmp (deleted)
worker_blob     10411 OFD   WRITE     0   0 

Diagnostic Recovery: When the path column is blank, the kernel can only expose the Major:Minor device and inode numbers in /proc/locks. To identify the target file, include the inode in your output (lslocks -o +INODE) and use find to map the inode back to the filesystem:

find /target/mountpoint -xdev -inum <INODE_NUMBER>

3. NFS and Clustered Filesystem Blindspots

Because lslocks reads the local kernel's /proc/locks table, its visibility in networked environments (NFSv3/NFSv4, CephFS, GlusterFS) has distinct boundaries: * Locks obtained by a remote NFS client via the Network Lock Manager (NLM) or NFSv4 state protocol appear on the NFS server's lslocks output under the local daemon process (rpc.statd, lockd, or nfsd). * However, if a remote NFS server experiences a lock conflict between two other client nodes, a local client's lslocks invocation will show nothing. Debugging clustered file locks requires querying the storage server's /proc/fs/nfsd/ state files or using nfsstat.

Hardened Cheat Sheet for High-Uptime Environments

# 1. Display all active locks with comprehensive, unambiguous columns
lslocks -o COMMAND,PID,TYPE,SIZE,MODE,M,START,END,PATH

# 2. Immediately isolate contested or blocking locks across the system
lslocks -u

# 3. Restrict inspection to a specific suspected process ID
lslocks -p <PID>

# 4. Display exact byte offsets for fine-grained database debugging
lslocks -b -o COMMAND,PID,TYPE,START,END,PATH

# 5. Output structured JSON for monitoring scripts and telemetry agents
lslocks -J -o COMMAND,PID,TYPE,SIZE,MODE,START,END,PATH

# 6. Run headless in continuous watch loops without repeating header lines
lslocks -n -o COMMAND,PID,TYPE,MODE,PATH

Today's Takeaway

File locking bottlenecks are uniquely frustrating because they never appear as CPU spikes, memory leaks, or saturated disk queues. Instead, they manifest as silent freezes where processes wait on invisible kernel primitives. Open a terminal right now and run lslocks -o COMMAND,PID,TYPE,MODE,START,END,PATH. Inspecting your machine's baseline locking state will immediately demystify how core services—from systemd and container runtimes to local databases—coordinate access to storage behind the scenes. Keep this command close at hand; the next time a production service stalls with zero resource usage, lslocks will pinpoint the exact process, file, and byte offset holding your infrastructure hostage.