Nethogs: Auditing Per-Process Network Bandwidth, Isolating Saturated Socket Sinks, and Triaging Cloud Egress Spikes in Production

The natural instinct is to reach for familiar diagnostic standbys like iftop, bmon, or vnstat. Within seconds, these tools confirm what the billing alert already made painfully obvious: fifty gigabits of unmetered data are rushing out of your physical network card toward an unfamiliar external IP address. Yet these traditional utilities hit an immediate dead end. They are brilliant at showing you which network cables are overheating and where the traffic is heading, but they remain utterly blind to the human-scale question that actually resolves an outage: which specific running program, rogue background job, or compromised container is responsible for sending the data?

In a sprawling modern server hosting dozens of microservices, database workers, and scheduled scripts, tracking down a connection from an IP address alone is like trying to identify an errant tenant in a crowded skyscraper solely by looking at the water meter on the street. You are left manually cross-referencing ephemeral port numbers across shifting namespaces while the clock ticks down.

This is where nethogs turns chaos into clarity. Instead of breaking traffic down by protocols, subnets, or hardware interfaces, nethogs groups network bandwidth consumption directly by Process ID (PID), program name, and user account. It gives you a real-time, top-style live scoreboard of exactly which programs are eating your bandwidth.

To immediately identify the bandwidth hog on your primary network interface (eth0), launch nethogs with superuser privileges:

sudo nethogs eth0

NetHogs version 0.8.7-cb4e82b

PID USER     PROGRAM                                            DEV        SENT      RECEIVED       
  12844 www-data /usr/sbin/nginx                                    eth0     1245.210    84.112 KB/sec
   4912 postgres /usr/lib/postgresql/16/bin/postgres                eth0       12.804   982.410 KB/sec
   1042 root     /usr/sbin/sshd: root@pts/0                         eth0        4.102     0.210 KB/sec
      ? root     unknown TCP                                                    0.000     0.000 KB/sec

TOTAL                                                                      1262.116  1066.732 KB/sec

Within a fraction of a second, the mystery evaporates. Rather than presenting an impenetrable wall of hexadecimal socket addresses, the terminal displays an orderly list of running binaries alongside live transfer speeds. You can instantly see whether a routine database dump has run amok, a web server worker is caught in an infinite redirect loop, or a rogue script is quietly exfiltrating gigabytes to an external server.

1. What It Does in Plain English

Traditional network diagnostic utilities inspect the flow of traffic passing through a network interface card (NIC), categorizing data streams strictly by hostnames, IP addresses, and protocol port numbers. In contrast, nethogs groups network bandwidth consumption directly by the running applications and processes generating that traffic.

Rather than reporting that an unknown remote endpoint is receiving forty megabytes per second on port 443, nethogs indicates precisely that /usr/bin/python3 running under PID 18402 owned by user app-runner is responsible. It provides a real-time, top-like dynamic visual interface for process-level network throughput, allowing systems engineers to instantly pinpoint and terminate bandwidth-hogging processes during active incidents.

2. Core Architectural Mechanics: From Libpcap to /proc Inode Resolution

To appreciate why nethogs succeeds where traditional packet sniffers stop short, it helps to understand its internal operational model. Unlike diagnostic tools that rely on dedicated kernel modules (such as eBPF-based socket tracers or SystemTap probes), nethogs functions entirely in user space through a clever two-stage correlation engine.

Stage 1: Promiscuous Header Interception via Libpcap

Upon invocation on a target interface (such as eth0 or bond0), nethogs initializes an AF_PACKET socket via the Tcpdump libpcap library. It captures Layer 3 and Layer 4 headers traversing the interface without reading the actual payload data, minimizing memory copies and CPU consumption. From these headers, nethogs extracts the fundamental transport 5-tuple:

$$\text{Flow} = (\text{Protocol}, \text{Source IP}, \text{Source Port}, \text{Destination IP}, \text{Destination Port})$$

Stage 2: Procfs Socket and Inode Introspection

To link this 5-tuple to a binary path, nethogs inspects the kernel's network state tables exported via the virtual filesystem: * /proc/net/tcp and /proc/net/tcp6 * /proc/net/udp and /proc/net/udp6 * /proc/net/raw and /proc/net/raw6

As documented in the Linux Kernel proc_net_tcp Documentation, the kernel represents active network connections with hexadecimal encoding of IP addresses, ports, connection states, and critically, the Virtual File System (VFS) socket inode number.

  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode
   1: 0100007F:1F90 00000000:0000 0A 00000000:00000000 00:00000000 00000000  1000        0 84920 ...

In the entry above, a local connection bound to 127.0.0.1:8080 (hexadecimal 0100007F:1F90) in listening state (0A) is assigned inode 84920.

Stage 3: File Descriptor Mapping and Process Identification

nethogs maintains an internal lookup table mapping (IP, Port) tuples to their corresponding VFS socket inodes. To map an inode to a physical process, it scans the directory entries of all active processes in /proc/[pid]/fd/.

Every open socket in a Linux process exists as a file descriptor symlink pointing to the kernel socket subsystem formatted as socket:[<inode>]. When nethogs resolves /proc/18402/fd/3 pointing to socket:[84920], it matches the inode from /proc/net/tcp with the process file descriptor table. Finally, it reads /proc/18402/cmdline and /proc/18402/exe to extract the full binary path and argument string.

Per-Process vs. Host-Level Attribution Matrix

Understanding the operational distinction between different telemetry tools prevents deploying the wrong utility during production incidents:

3. Core Flags & Quick Start Reference

The nethogs CLI binary adheres to standard POSIX conventions, accepting interface selectors, view formatters, and timing intervals as documented in the Linux man7 nethogs(8) manual.

Primary Command-Line Switches

• -t (Trace/Batch Mode): Disables the ncurses interactive terminal interface and emits newline-delimited textual records sequentially. Crucial for log shipping and scripted telemetry.
• -d <seconds> (Refresh Delay): Configures the sampling cycle interval (default: 1 second). Controls the granularity of the rolling rate calculation.
• -v <mode> (View/Unit Mode): Selects the display metric for bandwidth reporting:
• 0: Kilobytes per second (KB/s) — Default
• 1: Total Kilobytes transferred (KB)
• 2: Megabytes per second (MB/s)
• 3: Total Megabytes transferred (MB)
• -c <count> (Cycle Limit): Specifies the exact number of sampling iterations to capture before nethogs terminates automatically.
• -a (Monitor All Interfaces): Directs the capture engine to monitor all available interfaces, including loopback (lo) and unmanaged interfaces.
• -p (Promiscuous Mode): Forces the designated capture interface into promiscuous packet sniffing mode.
• -b (Bug Compatibility): Enables promiscuous mode with raw IP capture fallbacks for virtual interfaces lacking standard link-layer headers.

Interactive Keyboard Controls

When operating in interactive ncurses mode, the following runtime keys alter the view without restarting the process: * m: Cycle between throughput rate units (KB/s, B/s, MB/s, b/s). * r: Sort process table by received traffic (Download). * s: Sort process table by sent traffic (Upload). * q: Cleanly terminate execution, release raw socket bindings, and restore terminal state.

4. Five Production-Grade Field Scenarios

Use Case 1: Triaging Runaway Cloud Egress on Bonded Interfaces

Scenario

An Amazon Web Services (AWS) or Google Cloud Platform (GCP) egress billing alert fires: an internal application worker host with a dual-interface active-backup bonded link (bond0) is transmitting continuous outbound data at 950 Mbps, threatening both bandwidth quotas and upstream switch ports. Systems administrators must immediately identify which process or containerized microservice is pushing this unmetered egress.

Execution Command

sudo nethogs -v 2 -s bond0

Flags explained: -v 2 renders values directly in human-readable Megabytes per second (MB/s); -s enforces a strict descending sort based on outbound traffic (SENT); bond0 targets the aggregated Linux bonding interface.

Terminal Output

NetHogs version 0.8.7-cb4e82b

PID USER     PROGRAM                                            DEV        SENT      RECEIVED       
  31092 svc-app  /opt/venvs/worker/bin/python3 /opt/app/drain.py    bond0     108.450     0.142 MB/sec
   1402 root     /usr/bin/dockerd                                   bond0       0.412     0.088 MB/sec
    891 systemd- /lib/systemd/systemd-journal-upload                bond0       0.018     0.001 MB/sec
  28410 promethe /usr/bin/node_exporter                             bond0       0.004     0.002 MB/sec

TOTAL                                                                       108.884     0.233 MB/sec

Line-by-Line Technical Teardown

1. 31092 svc-app /opt/venvs/worker/bin/python3 /opt/app/drain.py: nethogs intercepted the outbound TCP segments on bond0, looked up the socket inodes in /proc/net/tcp, mapped inode 1948201 to PID 31092, and parsed /proc/31092/cmdline.
2. The process is generating 108.450 MB/s (approximately 867.6 Mbps) of outbound traffic while pulling a negligible 0.142 MB/s inbound.
3. The remaining processes (dockerd, systemd-journal-upload, node_exporter) exhibit baseline management traffic totaling less than 0.5 MB/s.

Remediation and Next Operational Steps

1. The administrator inspects the active network sockets for the specific process: bash sudo ss -tpn | grep 31092
2. Identify the remote IP addresses receiving the data to establish whether this represents a malicious exfiltration attempt or an uncontrolled telemetry crash dump.
3. Terminate or pause the rogue worker gracefully: bash sudo kill -15 31092
4. Check the systemd unit or orchestration definition that manages /opt/app/drain.py to prevent automated respawning until the underlying egress loop is patched.

Use Case 2: Non-Interactive Telemetry & Incident Auditing via Structured Batch Ingestion

Scenario

During intermittent performance degradations that occur during unmonitored off-peak hours (such as nightly batch processing windows), interactive consoles are useless. Network engineers require automated, non-interactive captures that output timestamped per-PID network usage directly into structured log processors, parsing scripts, or monitoring agents like Fluentbit and Vector.

Execution Command

sudo nethogs -t -d 2 -c 3 eth0

Flags explained: -t disables the ncurses buffer and activates linear stream mode; -d 2 captures over a stable two-second sliding window; -c 3 limits the execution to exactly three sample updates before automatically exiting.

Terminal Output

Refreshing:
/usr/bin/redis-server/10.0.1.15:6379-10.0.1.50:41280/1842/1001  24.120  1850.410
/usr/bin/mongod/10.0.1.15:27017-10.0.1.52:58102/2104/1002   180.500 12.110
/usr/sbin/sshd/10.0.1.15:22-192.168.1.100:54122/4910/0  0.812   0.140
unknown TCP/0.0.0.0:0-0.0.0.0:0/0/0 0.000   0.000

Refreshing:
/usr/bin/redis-server/10.0.1.15:6379-10.0.1.50:41280/1842/1001  31.840  2104.920
/usr/bin/mongod/10.0.1.15:27017-10.0.1.52:58102/2104/1002   194.200 14.800
/usr/sbin/sshd/10.0.1.15:22-192.168.1.100:54122/4910/0  0.410   0.080
unknown TCP/0.0.0.0:0-0.0.0.0:0/0/0 0.000   0.000

Refreshing:
/usr/bin/redis-server/10.0.1.15:6379-10.0.1.50:41280/1842/1001  28.910  1998.150
/usr/bin/mongod/10.0.1.15:27017-10.0.1.52:58102/2104/1002   175.110 11.900
/usr/sbin/sshd/10.0.1.15:22-192.168.1.100:54122/4910/0  0.210   0.040
unknown TCP/0.0.0.0:0-0.0.0.0:0/0/0 0.000   0.000

Line-by-Line Technical Teardown

1. Each record in trace mode prints in a deterministic format: $$\text{Binary Path} / \text{Local IP}:\text{Port} - \text{Remote IP}:\text{Port} / \text{PID} / \text{UID} \quad \text{Sent KB/s} \quad \text{Recv KB/s}$$
2. /usr/bin/redis-server/10.0.1.15:6379-10.0.1.50:41280/1842/1001 24.120 1850.410: * Binary path: /usr/bin/redis-server * Local socket: 10.0.1.15:6379 connected to client 10.0.1.50:41280 * Operating context: PID 1842, running under UID 1001 (redis) * Network performance: Pushing 24.120 KB/s sent, receiving an intense 1,850.410 KB/s ingress.
3. This tabular stream is structured for direct parsing via standard CLI utilities or log forwarders.

Scripted Pipeline Integration

To transform this live stream into a continuous CSV auditor for downstream SIEM ingestion, pipe nethogs through gawk:

sudo nethogs -t -d 5 eth0 | awk -F'[\t/]' '
BEGIN { print "timestamp,program,local_socket,remote_socket,pid,uid,sent_kb_s,recv_kb_s" }
/^Refreshing:/ { next }
NF >= 6 {
    ts = strftime("%Y-%m-%dT%H:%M:%SZ", systime(), 1);
    printf "%s,%s,%s,%s,%s,%s,%s,%s\n", ts, $1, $2, $3, $4, $5, $6, $7
}' >> /var/log/nethogs_audit.csv

Use Case 3: Auditing Multi-Interface Gateway Traffic for Asymmetrical Tunnel Leaks

Scenario

An enterprise security gateway hosts a public WAN physical interface (eth0), a corporate site-to-site WireGuard overlay tunnel (wg0), and an OpenVPN client connection (tun0). Network engineers observe unexpected transit bandwidth spikes on the public interface, suggesting that encrypted tunnel traffic or internal database replication is leaking outside the overlay networks due to asymmetrical routing errors.

Execution Command

sudo nethogs -a eth0 wg0 tun0

Flags explained: -a guarantees that all specified interfaces (eth0, wg0, tun0) are simultaneously bound by the libpcap sniffer engine within a single consolidated process monitoring table.

Terminal Output

NetHogs version 0.8.7-cb4e82b

PID USER     PROGRAM                                            DEV        SENT      RECEIVED       
  14502 backup   /usr/bin/rsync -avz /data/ rsync://192.168.10.5/   eth0      42150.12   120.450 KB/sec
   3201 root     /usr/bin/wireguard-go                              wg0         810.40   940.112 KB/sec
   8914 openvpn  /usr/sbin/openvpn --config client.ovpn             tun0        104.22    88.310 KB/sec
   1004 root     /usr/sbin/sshd                                     eth0          2.11     0.450 KB/sec

TOTAL                                                                       43066.85  1149.322 KB/sec

Line-by-Line Technical Teardown

1. 14502 backup /usr/bin/rsync ... eth0 42150.12 KB/sec: The rsync replication process targeting 192.168.10.5 is transmitting at approximately 42.15 MB/s over eth0 (the unencrypted public WAN interface) instead of routing through wg0 or tun0.
2. wireguard-go (wg0) and openvpn (tun0) are operating under minimal baseline throughput (< 1 MB/s).
3. This diagnosis confirms a routing table misconfiguration: the subnet 192.168.10.0/24 lacks a static route pointing to the wg0 interface gateway, resulting in default-route egress over unencrypted public infrastructure.

Remediation and Next Operational Steps

1. Immediately pause the data transmission: bash sudo kill -STOP 14502
2. Verify the kernel routing table for the destination IP address: bash ip route get 192.168.10.5
3. Restore the missing kernel route through the WireGuard interface according to the ArchWiki Network Monitoring and Routing Guide: bash sudo ip route add 192.168.10.0/24 dev wg0
4. Resume the transfer process securely over the encrypted overlay: bash sudo kill -CONT 14502

Use Case 4: High-Precision Bandwidth Profiling for Ephemeral Backup Daemons

Scenario

A PostgreSQL primary node exhibits severe transaction write latency degradation during scheduled maintenance windows. Database administrators suspect that the background physical backup agent (pg_basebackup or wal-g) saturates local network interfaces, consuming TCP buffer space and causing lock contention on replication connections. The administrator must establish the precise bandwidth footprint of the backup execution in total transferred megabytes.

Execution Command

sudo nethogs -v 3 -d 1 eth0

Flags explained: -v 3 configures the cumulative throughput mode, reporting total cumulative Megabytes (MB) transferred since execution rather than instantaneous rates; -d 1 enforces high-frequency 1-second sampling accuracy.

Terminal Output

NetHogs version 0.8.7-cb4e82b

PID USER     PROGRAM                                            DEV        SENT      RECEIVED       
   9481 postgres wal-g backup-push /var/lib/postgresql/data         eth0     18492.410     4.120 MB      
   2101 postgres postgres: walreceiver process                      eth0        12.450  1450.812 MB      
   2099 postgres postgres: checkpointer process                     eth0         0.000     0.000 MB      
   1105 root     /usr/lib/systemd/systemd-journald                  eth0         1.210     0.000 MB

TOTAL                                                                      18506.070  1454.932 MB      

Line-by-Line Technical Teardown

1. In cumulative transfer mode (-v 3), the values represent total volume transferred during the active nethogs profiling session.
2. wal-g backup-push (PID 9481) has transmitted 18,492.410 MB (approximately 18.5 GB) outbound to object storage with negligible inbound reception.
3. postgres: walreceiver process (PID 2101) has received 1,450.812 MB of replication stream data while sending 12.450 MB of acknowledgments.
4. The backup agent is operating without an egress limit, monopolizing the network interface card's hardware transmit queues (Tx) and starving the replication receiver thread.

Remediation and Next Operational Steps

1. Apply a native application throttling configuration or cgroups limit to the backup daemon: yaml # For WAL-G, configure the bandwidth limiter in /etc/wal-g/wal-g.yaml: WALG_BWLIMIT: "50MB"
2. Alternatively, apply immediate Linux traffic control (tc) policies to throttle traffic originating from the backup process: bash sudo tc qdisc add dev eth0 root handle 1: htb default 12 sudo tc class add dev eth0 parent 1: classid 1:1 htb rate 500mbit

Use Case 5: Container & Namespace Egress Attribution Across Virtual Ethernet (veth) Bridges

Scenario

On a shared, multi-tenant Kubernetes or Docker container host running over a Linux bridge (cbr0 or docker0), one tenant container saturates the physical node's upstream interface. Because containers execute inside dedicated network namespaces, running standard host utilities shows traffic on virtual ethernet peer interfaces (veth*), but fails to resolve container names. The engineer must associate the virtual interface traffic with the offending container namespace.

Execution Command

sudo nethogs -a veth3b91a0 cbr0 eth0

Flags explained: Monitors the specific virtual interface (veth3b91a0), the container bridge (cbr0), and the parent physical adapter (eth0).

Terminal Output

NetHogs version 0.8.7-cb4e82b

PID USER     PROGRAM                                            DEV            SENT      RECEIVED       
   8194 100054   /usr/local/bin/node /app/scraper.js                veth3b91a0   9812.450    412.110 KB/sec
   8194 100054   /usr/local/bin/node /app/scraper.js                cbr0         9812.450    412.110 KB/sec
   8194 100054   /usr/local/bin/node /app/scraper.js                eth0         9812.450    412.110 KB/sec
   4192 100010   /usr/bin/python3 /app/server.py                    veth7a41c2      1.120      0.450 KB/sec

TOTAL                                                                         19626.020    824.670 KB/sec

Line-by-Line Technical Teardown

1. 8194 100054 /usr/local/bin/node /app/scraper.js: nethogs tracks the traffic passing through the virtual interface pair (veth3b91a0), bridged across cbr0, and forwarded out of eth0.
2. Although the process executes inside a network namespace, nethogs maps the socket back to the host-level PID 8194 and host UID 100054 because /proc on the host maintains visibility over all processes across child PID namespaces.
3. The process is consuming approximately 9.81 MB/s of continuous bandwidth on veth3b91a0, identifying it as the noisy neighbor.

Remediation and Next Operational Steps

1. Trace the host PID 8194 back to its Kubernetes container and Pod manifest: bash cat /proc/8194/cgroup | grep -o 'pod.*' # Alternatively query crictl: sudo crictl ps --pid 8194
2. Locate the Pod name and namespace (for example, analytics-scraper-7b8f99-x21z in namespace data-platform).
3. Apply a Kubernetes NetworkPolicy or egress bandwidth limit using standard resource annotations: yaml apiVersion: v1 kind: Pod metadata: name: analytics-scraper annotations: kubernetes.io/egress-bandwidth: "2M"
4. Evict or delete the offending pod to restore cluster stability: bash kubectl delete pod analytics-scraper-7b8f99-x21z -n data-platform --grace-period=0 --force

5. Operational Pathology: Edge Cases, Kernel Traps, and Failure Modes

While nethogs is indispensable for real-time process attribution, running packet capture utilities in mission-critical environments introduces operational nuances and trade-offs.

1. The Ephemeral Connection Race Condition ("unknown TCP")

A common artifact in nethogs outputs is the emergence of ? root unknown TCP or unknown UDP entries. This occurs due to an inherent race condition between libpcap header arrival and /proc filesystem polling.

If an application establishes a TCP connection, transfers a small payload (such as a DNS lookup or an HTTP health probe), and issues a close() system call within a few milliseconds, libpcap successfully captures the packet headers. However, by the time nethogs queries /proc/net/tcp and traverses /proc/[pid]/fd/, the kernel has already torn down the socket data structures and freed the inode.

Mitigation: For environments dominated by short-lived, ephemeral micro-bursts, do not rely on user-space procfs scrapers. Deploy eBPF-based socket lifecycle trackers (such as bcc-tools/tcptop or bpftrace) which hook directly into kernel tracepoints (sock:inet_sock_set_state).

2. High Packet-Per-Second (PPS) Saturation and CPU Overhead

nethogs is designed for diagnostic troubleshooting, not permanent background monitoring across high-throughput nodes. When an interface experiences high packet rates (>500,000 PPS on 10GbE or 40GbE interfaces), capturing packets via user-space libpcap introduces significant context switching overhead between kernel space and user space.

Danger: Running nethogs on an interface undergoing a distributed denial-of-service (DDoS) attack or handling massive ingress traffic can consume 100% of an assigned CPU core, dropping packets and compounding network degradation.

Guardrail: * Never invoke nethogs on high-throughput interfaces without setting a sample interval limit (for example, -d 2 or -d 5). * Terminate the process immediately if CPU consumption spikes: bash sudo pkill -9 nethogs

3. Privilege Escalation vs. Capability Sandboxing

Executing diagnostic utilities as full root violates the principle of least privilege. While nethogs requires access to raw sockets and /proc structures, granting complete superuser access exposes the system to unnecessary risk.

Under the Linux Capabilities System, nethogs requires only two distinct kernel capabilities: * CAP_NET_RAW: Allows opening AF_PACKET raw sockets for network packet capture. * CAP_NET_ADMIN: Allows placing network interfaces into promiscuous mode.

Systems administrators should configure file-based capabilities on the binary, permitting operations engineers to run the tool without full sudo privileges:

# Set specific capabilities on the nethogs binary
sudo setcap 'cap_net_admin,cap_net_raw+ep' $(which nethogs)

# Verify the assigned capabilities
getcap $(which nethogs)
# Output: /usr/sbin/nethogs = cap_net_admin,cap_net_raw+ep

# Execute safely as an unprivileged user
nethogs eth0

6. Today's Takeaway

Network saturation is rarely an abstract infrastructure failure; it is almost always the direct consequence of an identifiable, running process executing unthrottled I/O system calls. To prepare for your next production incident, grant minimal capabilities to the binary (sudo setcap 'cap_net_admin,cap_net_raw+ep' $(which nethogs)), run nethogs -v 2 on your primary development or staging interface, and observe how your background processes, container runtimes, and local daemons interact with the network. Mastering process-level socket attribution converts frantic midnight guesswork into precise, methodical engineering remediation.