Lsns: Querying Linux Namespaces, Auditing Container Isolation Topologies, and Inspecting Process Boundary Hierarchies in Production

You open your terminal and begin the standard resuscitation ritual, only to find yourself trapped in an operational ghost story. The orchestration dashboard insists that the offending container died forty minutes ago, terminated by an automated health check. Yet the port it once occupied remains aggressively blocked, upstream proxies are timing out, and local storage disks refuse to unmount. To make matters worse, the cluster engine refuses to schedule a replacement workload on the node because the virtual network adapter refuses to detach.

Under normal circumstances, you would reach for familiar diagnostic tools like ps, top, or netstat. But tonight, they offer no help at all. When you search for the rogue process ID, the process table reports that it does not exist. When you inspect the local network stack, the port appears completely empty. Traditional Unix utilities are fundamentally blinkered: they can only observe the operating system through the specific lens of whichever environment they were launched in. They cannot peer across the invisible partitions that modern Linux systems use to separate workloads into isolated, parallel universes.

To untangle these phantom boundaries and reveal what is actually happening inside the kernel, system administrators rely on lsns(8)—the canonical Linux utility built specifically to discover, enumerate, and inspect kernel namespaces.

Instead of forcing you to manually dig through thousands of cryptic directories under /proc, a single invocation of lsns scans the entire process tree and delivers an immediate, host-wide census of every isolated boundary currently active:

lsns -o NS,TYPE,NPROCS,PID,USER,COMMAND

        NS TYPE   NPROCS   PID USER     COMMAND
4026531834 time      182     1 root     /usr/lib/systemd/systemd --switched-root --system
4026531835 cgroup    182     1 root     /usr/lib/systemd/systemd --switched-root --system
4026531837 user      182     1 root     /usr/lib/systemd/systemd --switched-root --system
4026532411 uts         1 38201 10001    /app/service-worker --worker-id=3
4026532412 ipc         1 38201 10001    /app/service-worker --worker-id=3
4026532413 mnt         1 38201 10001    /app/service-worker --worker-id=3
4026532414 pid         1 38201 10001    /app/service-worker --worker-id=3
4026532415 net         1 38201 10001    /app/service-worker --worker-id=3
4026532577 mnt        12 41092 root     /usr/bin/containerd-shim-runc-v2 -namespace k8s.io
4026532578 uts        12 41092 root     /usr/bin/containerd-shim-runc-v2 -namespace k8s.io
4026532579 ipc        12 41092 root     /usr/bin/containerd-shim-runc-v2 -namespace k8s.io
4026532580 pid        12 41092 root     /usr/bin/containerd-shim-runc-v2 -namespace k8s.io
4026532581 net        12 41092 root     /usr/bin/containerd-shim-runc-v2 -namespace k8s.io

With this single command, the opaque abstractions of modern container runtimes evaporate. You can see precisely which namespaces exist, how many processes are corralled inside them, who owns them, and the root process responsible for their creation.

1. What It Does in Plain English

At its core, lsns provides a comprehensive bird's-eye map of your operating system's isolation topology. Rather than treating containers as opaque black boxes, lsns queries the kernel to determine how system resources—such as network cards, mount tables, user accounts, and process IDs—have been carved up among running tasks.

Kernel Mechanics: Inodes, References, and the Eightfold Boundary

To understand why lsns is essential during outages, one must look at how the Linux kernel handles namespaces(7). Namespaces are not physical partitions or emulated virtual hardware; they are lightweight kernel data structures that encapsulate global system resources into virtualized domains.

Every running task in Linux is represented by a struct task_struct. Inside this structure sits a pointer to user credentials (task_struct->cred->user_ns) and a pointer to an nsproxy table (task_struct->nsproxy), which ties the process to its specific virtualization boundaries:

The Linux kernel instantiates eight distinct namespace subsystems, created using system call flags via clone(2) or unshare(2):

Inside the Virtual Filesystem (VFS), every active namespace is assigned an inode on the synthetic nsfs pseudo-filesystem. When you inspect a process directory under /proc/[pid]/ns/, each entry appears as a specialized symlink pointing to its unique inode number:

/proc/41092/ns/cgroup -> cgroup:[4026531835]
/proc/41092/ns/ipc    -> ipc:[4026532579]
/proc/41092/ns/mnt    -> mnt:[4026532577]
/proc/41092/ns/net    -> net:[4026532581]
/proc/41092/ns/pid    -> pid:[4026532580]
/proc/41092/ns/time   -> time:[4026531834]
/proc/41092/ns/user   -> user:[4026531837]
/proc/41092/ns/uts    -> uts:[4026532578]

The Inode Lifecycle and Garbage-Collection Mechanics

A frequent source of production outages is a misunderstanding of how the kernel cleans up namespaces. Each namespace structure contains an atomic reference counter (refcnt). The kernel retains the namespace in memory and preserves its synthetic inode as long as refcnt > 0.

This counter increases whenever: 1. A running task or thread references the namespace via its nsproxy or cred struct. 2. An external process holds an open file descriptor pointing to /proc/[pid]/ns/<type>. 3. A persistent VFS bind-mount pins /proc/[pid]/ns/<type> to a filesystem path (such as /var/run/netns/<name> created by ip netns). 4. A nested child namespace holds an active reference to its parent.

When every process inside a container terminates, the kernel attempts to invoke the relevant cleanup destructor (such as put_net() for network stacks or put_mnt_ns() for mount tables). However, if an external monitoring agent, tracing script, or rogue daemon leaves a file descriptor open to that namespace's /proc entry, the reference count never hits zero. The processes vanish from the process table, but the namespace—along with its IP addresses, virtual ethernet interfaces, and mount references—remains locked in kernel memory.

lsns resolves this by reading all numeric directories in /proc, resolving the target links in /proc/[pid]/ns/*, and constructing a complete in-memory graph. It groups processes by inode number, highlights the namespace leaders, and pinpoints orphaned boundaries.

2. Core Flags & Command Reference

Running lsns without arguments outputs a table of every discovered namespace on the system. To adapt the tool for precise diagnostics and automated scripting, several core flags are available.

Essential Flag Reference

3. Five Real-World Production Use Cases

The true diagnostic strength of lsns appears when container abstractions break down, security boundaries need validation, or low-level kernel resources leak silently.

Use Case 1: Auditing Container Runtime Boundaries on Kubernetes Nodes

Scenario

During a cluster security audit, an administrator must verify that no untrusted third-party tenant pods running on a shared Kubernetes worker node have been misconfigured with hostNetwork: true or hostIPC: true. These settings inadvertently grant containers direct access to host network interfaces, loopback communications, and shared memory segments.

Execution Command

lsns -t net,ipc -o NS,TYPE,NPROCS,PID,USER,COMMAND | grep -E "(systemd|containerd-shim|kubelet)"

Terminal Output

        NS TYPE   NPROCS   PID USER COMMAND
4026531992 net       142     1 root /usr/lib/systemd/systemd --switched-root --system
4026531993 ipc       142     1 root /usr/lib/systemd/systemd --switched-root --system
4026532810 net         2 18920 root /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id a9f1...
4026532811 ipc         2 18920 root /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id a9f1...
4026531992 net        34 22401 root /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id ff82...
4026532940 ipc         4 22401 root /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id ff82...

Line-by-Line Technical Analysis

• Lines 1–2: The root host namespaces (net:[4026531992] and ipc:[4026531993]) are owned by PID 1 (systemd), managing 142 host processes.
• Lines 3–4: Container shim PID 18920 correctly isolates its processes inside a dedicated network namespace (net:[4026532810]) and IPC namespace (ipc:[4026532811]).
• Line 5: Container shim PID 22401 has 34 processes bound directly to net:[4026531992]—the root host network namespace.
• Line 6: The same container shim PID 22401 uses an isolated IPC namespace (ipc:[4026532940]), proving that network isolation was explicitly disabled while IPC isolation remained enabled.

What the Administrator Does Next

The administrator maps container ID ff82... back to its Kubernetes pod manifest:

kubectl get pods --all-namespaces -o json | jq '.items[] | select(.status.containerStatuses[]?.containerID | contains("ff82")) | .metadata.name'

After identifying the offending workload, they update the pod manifest to disable hostNetwork: true and enforce cluster-wide admission policies to prevent unauthorized host namespace sharing.

Use Case 2: Pinpointing Orphaned File Descriptor Leaks Holding Stale Namespaces

Scenario

A Kubernetes node fails to tear down a terminated pod's virtual network interface (veth*). The network plugin reports failed to delete sandbox network: device or resource busy. The container processes are completely gone, but the network namespace remains trapped in kernel memory, blocking IP recycling and network cleanup.

Execution Command

lsns -t net -o NS,NPROCS,PID,COMMAND

        NS NPROCS   PID COMMAND
4026531992    148     1 /usr/lib/systemd/systemd --switched-root --system
4026533104      0     - <orphaned/pinned>
4026533215      4 31204 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io

To locate the external process holding open file handles to the zero-process namespace net:[4026533104]:

find /proc/[1-9]*/fd -lname "*4026533104*" 2>/dev/null

Terminal Output

/proc/9842/fd/7 -> /proc/28411/ns/net:[4026533104]

To inspect the offending process:

ps -fp 9842

UID        PID  PPID  C STIME TTY          TIME CMD
root      9842     1  0 01:15 ?        00:00:02 /opt/security-agent/bin/network-tracer --daemon

Line-by-Line Technical Analysis

• lsns Output: The namespace net:[4026533104] shows NPROCS: 0 and PID: -. No active processes remain inside, but the kernel destructor put_net() has not freed the memory.
• find Output: File descriptor 7 of PID 9842 holds an open handle pointing to net:[4026533104].
• ps Output: A third-party security tracing daemon (network-tracer, PID 9842) opened a handle to the container's network namespace on startup but failed to close the descriptor when the container stopped.

What the Administrator Does Next

The administrator restarts the leaking daemon (systemctl restart network-tracer). As soon as the lingering file descriptor closes, the reference counter drops to zero, the kernel automatically garbage-collects the namespace, and the virtual network interface detaches cleanly.

Use Case 3: Extracting Namespace Leaders for In-Flight nsenter Diagnostics

Scenario

A mission-critical payment processing service running inside an ultra-minimal, distroless container image (with no shell, no curl, and no packet capture utilities) is returning intermittent HTTP 502 errors. The engineer needs to run tcpdump directly on the container's loopback and interface traffic without restarting the workload or altering its filesystem.

Execution Command

lsns -t net -p $(pgrep -f "order-processor --port=8443") -o NS,TYPE,PID,NPROCS,COMMAND

Terminal Output

        NS TYPE     PID NPROCS COMMAND
4026533812 net    51208      3 /app/order-processor --port=8443

With the host leader PID (51208) identified, the engineer executes nsenter(1) to attach host-native debugging tools into the container's network namespace:

nsenter -t 51208 -n tcpdump -nn -vv -i any port 8443 -c 3

tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
02:48:12.104921 eth0 In  IP (tos 0x0, ttl 64, id 41201, offset 0, flags [DF], proto TCP (6), length 60)
    10.244.3.1.58492 > 10.244.3.45.8443: Flags [S], cksum 0x1a2b (correct), seq 1098234120, win 64240, options [mss 1460,sackOK,TS val 3829102 ecr 0], length 0
02:48:12.105103 eth0 Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    10.244.3.45.8443 > 10.244.3.1.58492: Flags [R.], cksum 0x8f21 (correct), seq 0, ack 1098234121, win 0, length 0

Line-by-Line Technical Analysis

• lsns Resolution: Finds that order-processor runs inside network namespace net:[4026533812] led by host PID 51208.
• nsenter Execution: Switches only the network namespace (-n) to match PID 51208 while preserving the host mount namespace. This allows the host's /usr/sbin/tcpdump binary to run against the container's virtual network interface.
• Packet Trace: The capture reveals an immediate Flags [R.] (TCP Reset) response from the local socket, indicating that the application's connection backlog is saturated rather than failing due to upstream routing issues.

What the Administrator Does Next

Instead of performing an unguided container restart, the administrator increases the socket backlog limit (net.core.somaxconn) and expands the application worker thread pool in the service configuration.

Use Case 4: Automating Security Audits via JSON Pipelines to Detect Privilege Escalation

Scenario

Unprivileged user namespaces (CLONE_NEWUSER) allow non-root users to map UID 0 inside their private container namespace. While critical for rootless containerization, unprivileged user namespaces expand the kernel attack surface. The security operations team requires an automated daily scan to detect unauthorized user namespaces created outside approved container engines.

Execution Command

lsns -J -t user -o NS,PID,USER,UID,NPROCS,COMMAND | jq '.namespaces[] | select(.uid != 0 and .uid != 999 and (.command | test("containerd|crio|podman") | not))'

Terminal Output

{
  "ns": 4026534910,
  "pid": 62114,
  "user": "web-deployer",
  "uid": 1002,
  "nprocs": 2,
  "command": "/tmp/.hidden_exploit/unshare_payload --spawn-user-ns"
}

Line-by-Line Technical Analysis

• lsns -J -t user: Extracts all active user namespaces across the host in structured JSON format.
• jq Filter: Excludes system root (UID 0) and container runtime daemons (UID 999), as well as approved container binaries (containerd, crio, podman).
• Result Payload: Flags an unauthorized binary running out of /tmp under standard user web-deployer (UID 1002), which has spawned an unprivileged user namespace (user:[4026534910]).

What the Administrator Does Next

The security team terminates the rogue process (kill -9 62114), isolates the compromised account, and restricts unprivileged user namespace creation system-wide:

sysctl -w kernel.unprivileged_userns_clone=0

Use Case 5: Diagnosing Mount Propagation Leaks Across Container Hierarchies

Scenario

A storage automation worker dynamically mounts host backup directories into container volumes. Over time, storage unmount operations fail with EBUSY (Device or resource busy). The administrator suspects that mount events are propagating recursively between the host and container namespaces due to misconfigured mount propagation flags (MS_SHARED versus MS_SLAVE).

Execution Command

lsns -t mnt -o NS,NPROCS,PID,USER,COMMAND

        NS NPROCS   PID USER COMMAND
4026531840    160     1 root /usr/lib/systemd/systemd --switched-root --system
4026534120      2 27810 root /usr/bin/containerd-shim-runc-v2 -namespace k8s.io
4026534188      2 28450 root /usr/bin/containerd-shim-runc-v2 -namespace k8s.io
4026534299      2 29100 root /usr/bin/storage-sync-agent --mount-path=/mnt/data

To cross-examine the mount propagation table of the suspicious agent (PID 29100):

findmnt -N 29100 -o TARGET,SOURCE,FSTYPE,PROPAGATION

Terminal Output

TARGET                 SOURCE            FSTYPE  PROPAGATION
/                      /dev/nvme0n1p2    ext4    shared:1
  /mnt/data            /dev/nvme1n1p1    xfs     shared:48
    /mnt/data/vol-001  /dev/nvme2n1p1    ext4    shared:52
    /mnt/data/vol-002  /dev/nvme3n1p1    ext4    shared:53
  /proc                proc              proc    shared:1

Line-by-Line Technical Analysis

• lsns -t mnt: Identifies all active mount namespaces, showing PID 29100 (storage-sync-agent) running inside isolated mount namespace mnt:[4026534299].
• findmnt -N 29100: Inspects the mount hierarchy directly through the mount namespace of PID 29100.
• Propagation Output: The storage directories /mnt/data/vol-001 and /mnt/data/vol-002 are flagged as shared:52 and shared:53. Because they are marked as shared rather than slave or private, filesystem operations created inside the container propagate back to the host, preventing host unmounts because the container maintains an active reference.

What the Administrator Does Next

The administrator changes the container mount propagation setting from rshared to rslave (or runs mount --make-rslave /mnt/data inside the container namespace). This ensures mounts propagate strictly unidirectionally from the host to the container without locking host storage devices upon unmount.

4. What Can Go Wrong: Pitfalls, Dangers, and Mitigations

While lsns is a read-only diagnostic utility, misinterpreting its output in production can lead to severe operational mistakes.

Pitfall 1: The PID Context Inversion Trap

A dangerous mistake occurs when an operator runs lsns from inside a container and attempts to use the reported PIDs in host-level commands like kill -9 <PID> or nsenter -t <PID>.

Inside a container PID namespace, processes are renumbered starting at PID 1. If an operator mistakes an internal container PID for a host PID and executes a kill command from the host shell, they may send the signal to an entirely unrelated host process, potentially crashing critical system daemons.

Mitigation: Always verify which PID namespace your current shell occupies before acting on numerical PIDs. Run: bash readlink /proc/1/ns/pid && readlink /proc/self/ns/pid If the two inode numbers differ, your current shell is running inside a child namespace, and its local PIDs do not correspond to host PIDs.

Pitfall 2: High-Density /proc Scanning Overhead

On dense server nodes hosting thousands of containers and tens of thousands of threads, invoking lsns causes the tool to open, read, and parse /proc/[pid]/ns/* for every numerical directory in /proc.

Although /proc is an in-memory virtual filesystem, reading thousands of entries requires acquiring read locks across internal kernel task lists. Running lsns in tight polling loops (e.g., every 500ms via custom monitoring scripts) causes measurable lock contention, driving up system CPU utilization.

Mitigation: When monitoring a known workload, avoid scanning the whole host. Always constrain discovery using the process filter flag -p <PID>: bash lsns -p <TARGET_PROCESS_PID> For automated metric collection, scrape at reasonable intervals (e.g., every 30–60 seconds) and parse the output using the JSON interface (lsns -J).

Pitfall 3: PID Recycling Race Conditions

In high-churn container environments—such as continuous integration runners or serverless hosts—processes spawn and terminate within milliseconds. A PID identified by lsns as a namespace leader may terminate immediately after the scan completes. If the kernel recycles that PID for a newly spawned process, subsequent commands like nsenter -t <PID> could attach to an unintended namespace.

Mitigation: Before executing operations against a discovered PID, verify that the active namespace inode matches your target: bash TARGET_NS="4026533812" CURRENT_NS=$(readlink /proc/<PID>/ns/net | tr -dc '0-9') if [ "$TARGET_NS" = "$CURRENT_NS" ]; then nsenter -t <PID> -n <command> fi

5. Today's Takeaway

To truly understand containerized infrastructure, you must inspect your systems through the kernel's actual isolation boundaries rather than relying solely on high-level orchestration abstractions. In the next five minutes, open a terminal on any Linux machine with administrative privileges and run:

lsns -t net,mnt -o NS,TYPE,NPROCS,PID,USER,COMMAND

Take note of how your operating system partitions system daemons and containerized workloads across distinct synthetic inode numbers on the nsfs filesystem. Check which processes share network stacks, locate your container leaders, and verify that no orphaned namespaces are silently lingering in memory. Making lsns your first diagnostic reflex turns opaque container abstractions into a clear, debuggable map of kernel reality.

Authoritative References & Documentation

• lsns(8) Linux Manual Page — util-linux namespace enumeration documentation.
• namespaces(7) Linux Overview — Deep architectural specification of Linux kernel namespace primitives.
• nsenter(1) Manual Page — Tool for entering process namespaces.
• mount_namespaces(7) Manual Page — Comprehensive guide to mount tables, shared subtrees, and mount propagation.
• user_namespaces(7) Manual Page — Security models, UID/GID mappings, and capabilities inside unprivileged containers.
• pid_namespaces(7) Manual Page — Process ID hierarchies and init process lifecycle in nested namespaces.
• clone(2) Kernel System Call Documentation — Low-level interface for spawning tasks into new namespace boundaries.
• Linux Kernel Administrative Guide to Namespaces — Official kernel documentation on virtualized subsystem boundaries.