Bpftool: Inspecting Loaded eBPF Programs, Auditing Kernel Map Topologies, and Managing Subsystem Objects in Production

When traditional tools claim everything is working while your services fail in plain sight, you have run headlong into the modern Linux kernel. Today's operating system is no longer a static switchboard passing data between hardware and user applications. Networking policies, security guardrails, and observability probes now run dynamic bytecode directly inside the kernel's execution path via extended Berkeley Packet Filter (eBPF) programs. When one of these internal programs misbehaves or silently discards data, conventional tools like netstat, ss, and tcpdump remain completely oblivious.

To see what is actually occurring inside ring 0, systems administrators rely on bpftool. This utility serves as the unified inspection lens and control plane for the kernel's eBPF runtime subsystem. Much like ps and top illuminate running userspace processes, bpftool translates raw kernel memory structures, bytecode instructions, and event hooks into human-readable data—letting you pinpoint rogue filters, inspect live data tables, and restore traffic without rebooting the server.

The quickest way to cut through the darkness during an outage and audit every eBPF program running on your host is a single baseline command:

sudo bpftool prog show --pretty

Running this command produces an immediate, structured snapshot of all resident kernel programs and the exact processes responsible for them:

[
    {
        "id": 248,
        "type": "sched_cls",
        "name": "tc_ingress_flt",
        "tag": "d3b4a2e5f8910c21",
        "gpl_compatible": true,
        "loaded_at": 1723504800,
        "uid": 0,
        "bytes_xlated": 528,
        "jited": true,
        "bytes_jited": 312,
        "bytes_memlock": 4096,
        "map_ids": [
            112,
            115
        ],
        "btf_id": 84,
        "pids": [
            {
                "pid": 38412,
                "comm": "cilium-agent"
            }
        ]
    }
]

In an instant, this output reveals the program's unique ID (248), its kernel attachment type (sched_cls for traffic control classification), its compiled machine code memory footprint (bytes_jited), its linked memory structures (map_ids), and the exact userspace process (cilium-agent at PID 38412) maintaining an open handle on the object.

Core Flags and Command-Line Reference

Before diving into targeted incident diagnostics, familiarise yourself with the global switches that control how bpftool interfaces with the kernel and serializes diagnostic information.

The Linux eBPF Architecture: Verification, BTF, and Filesystem Persistence

To wield bpftool effectively, it helps to understand how modern Linux kernels load, verify, and execute sandboxed code inside ring 0.

When an application supplies an eBPF program to the kernel via the bpf(2) system call, the kernel does not run it blindly. The bytecode must first pass through the in-kernel verifier, an automated static analysis engine that proves program safety. The verifier examines every potential branch, preventing out-of-bounds pointer arithmetic, uninitialized memory reads, and infinite loops to guarantee that the program cannot crash or deadlock the host operating system.

Once mathematically verified, the bytecode is compiled into native host machine instructions by the Just-In-Time (JIT) compiler, enabling execution speeds identical to native kernel drivers.

To bridge the gap between compiled bytecode and human understanding, the kernel uses BPF Type Format (BTF). Detailed in the Linux Kernel BTF Documentation, BTF embeds rich type metadata—such as C struct definitions, function signatures, and field offsets—directly into the kernel and compiled objects. This metadata underpins Compile Once – Run Everywhere (CO-RE), allowing programs to adapt dynamically to differing kernel struct layouts at runtime.

By default, an eBPF program lives only as long as the userspace process that loaded it holds an open file descriptor. When that process exits, the kernel garbage-collects the program. To ensure telemetry collectors, firewalls, and routing tables survive software restarts, objects can be pinned to the BPF Virtual Filesystem (bpffs), usually mounted at /sys/fs/bpf. Pinning increments the kernel reference counter, anchoring programs and maps in memory across daemon lifecycles.

5 Real-World Production Use Cases

1. Auditing and Disassembling Rogue Kernel Programs

Scenario

A microservices node experiences intermittent microsecond latency spikes across CPU scheduling paths. You suspect an unmanaged monitoring agent has attached an unoptimized dynamic probe (kprobe) to the kernel's context-switch routine (finish_task_switch), stalling CPU threads during high-frequency workload transitions.

Command

List all running programs with full details, then dump the translated assembly and JIT-compiled machine code for the suspect program ID (412):

sudo bpftool prog show --details
sudo bpftool prog dump xlated id 412 opcodes
sudo bpftool prog dump jited id 412

Realistic Terminal Output

# bpftool prog show id 412
412: kprobe  name trace_sched_switch  tag b12e456ad8c0ffee  gpl_compatible
    loaded_at 2026-08-18T23:10:04+0000  uid 0
    xlated 320B  jited 198B  memlock 4096B  map_ids 89
    btf_id 142
    pids rogue_agent(91823)

# bpftool prog dump xlated id 412 opcodes
   0: (79) r1 = *(u64 *)(r1 +0)
      bpf_probe_read_kernel: 79 11 00 00 00 00 00 00
   1: (b7) r2 = 16
      16: b7 02 00 00 10 00 00 00
   2: (bf) r3 = r10
      r3 = r10: bf a3 00 00 00 00 00 00
   3: (07) r3 += -16
      r3 += -16: 07 03 00 00 f0 ff ff ff
   4: (85) call bpf_probe_read_kernel#-89216
      call: 85 00 00 00 c0 a3 fe ff
   5: (18) r1 = 0xffff888104b28e00
      load map: 18 01 00 00 00 8e b2 04 81 88 ff ff
   7: (85) call bpf_map_lookup_elem#-90144
      call: 85 00 00 00 20 a0 fe ff
   8: (15) if r0 == 0x0 goto pc+3
      jnz: 15 00 03 00 00 00 00 00
   9: (79) r1 = *(u64 *)(r0 +0)
      r1 = *(u64 *)(r0 + 0): 79 01 00 00 00 00 00 00
  10: (07) r1 += 1
      r1 += 1: 07 01 00 00 01 00 00 00
  11: (7b) *(u64 *)(r0 +0) = r1
      writeback: 7b 10 00 00 00 00 00 00
  12: (b7) r0 = 0
      exit: b7 00 00 00 00 00 00 00
  13: (95) exit
      exit: 95 00 00 00 00 00 00 00

Line-by-Line Technical Analysis

• 412: kprobe name trace_sched_switch: Confirms program ID 412 is attached via the dynamic kprobe tracing hook under the function name trace_sched_switch.
• pids rogue_agent(91823): Connects the kernel program directly to userspace process ID 91823 executing rogue_agent.
• 0: (79) r1 = *(u64 *)(r1 +0): Loads the initial probe context pointer into eBPF register r1.
• 1-4: call bpf_probe_read_kernel: Copies memory safely from the kernel task frame onto the program's local stack space (r10 - 16).
• 5-7: call bpf_map_lookup_elem: Queries storage map ID 89 using the static kernel memory address 0xffff888104b28e00.
• 8-11: *(u64 *)(r0 +0) = r1: Performs an unbuffered, non-atomic 64-bit integer increment on a shared memory counter rather than using bpf_atomic_add, inducing severe cacheline bouncing across multi-core systems.

What the Admin Does Next

Terminate the offending daemon (sudo kill -9 91823), verify with sudo bpftool prog show id 412 that the kernel has unloaded the probe, and establish an alert policy to prevent unauthorized tracing scripts on critical hosts.

2. Inspecting and Modifying Live Kernel Maps

Scenario

An in-kernel edge firewall is rejecting valid customer API calls because an operational rate-limiting map was provisioned with a strict limit of 100 requests per second. You must update this threshold in live kernel memory immediately without restarting the firewall process or terminating active TCP sessions.

Command

Query the map metadata, inspect the current key-value pair, and update the entry using bpftool map:

sudo bpftool map show name ratelimit_config
sudo bpftool map dump name ratelimit_config
sudo bpftool map update name ratelimit_config key hex 0a 00 00 01 value hex 00 40 07 00

Realistic Terminal Output

# bpftool map show name ratelimit_config
89: hash  name ratelimit_config  flags 0x0
    key_size 4  value_size 4  max_entries 1024  memlock 89128B
    btf_id 142
    pids edge_fw(1042)

# bpftool map dump name ratelimit_config
key: 0a 00 00 01  value: 64 00 00 00
Found 1 element

# bpftool map update name ratelimit_config key hex 0a 00 00 01 value hex 00 40 07 00

# bpftool map lookup name ratelimit_config key hex 0a 00 00 01
key: 0a 00 00 01  value: 00 40 07 00

Line-by-Line Technical Analysis

• 89: hash name ratelimit_config: Identifies Map ID 89 as a hash table structure named ratelimit_config.
• key_size 4 value_size 4: Denotes that both the lookup keys and storage values are 4-byte (32-bit) unsigned integers.
• key: 0a 00 00 01 value: 64 00 00 00: Displays the stored hexadecimal pair in little-endian format. The key 0a 00 00 01 represents IPv4 address 10.0.0.1, and the value 64 00 00 00 equals decimal 100 operations per second.
• bpftool map update ... value hex 00 40 07 00: Writes a new limit directly to the kernel hash table. The little-endian hex 00 40 07 00 translates to 0x00074000, setting the new allowance to 475,136 requests per second.
• bpftool map lookup: Reads the key back from kernel space to verify that the live memory update succeeded.

What the Admin Does Next

The new limit takes effect instantaneously inside the running filter. Monitor ingress traffic to ensure packet drops stop, and update your configuration repository so the change persists across future software deployments.

3. Auditing Network Hooks and XDP Packet Drops

Scenario

A 100GbE network interface (eth0) is dropping incoming jumbo frames. Standard routing utilities and firewall rule counters report zero drops, yet incoming frames vanish before reaching user applications. You need to inspect the interface driver to verify if an eXpress Data Path (XDP) filter is discarding frames at the network card level.

Command

Inspect network attachments across the interface using bpftool net:

sudo bpftool net show dev eth0

Realistic Terminal Output

xdp:
eth0(2) drv id 501 act XDP_DROP

tc:
eth0(2) clsact/ingress proto all pref 49152 handle 0x1 bpf_ingress.o:[action_eval] id 505 default_action pipe
eth0(2) clsact/egress proto all pref 49152 handle 0x1 bpf_egress.o:[redirect_flow] id 509 default_action pipe

flow_dissector:

Line-by-Line Technical Analysis

• xdp: eth0(2) drv id 501 act XDP_DROP: Shows an active XDP filter (ID 501) executing directly inside the network card driver (drv mode). The fallback action XDP_DROP discards unmatched frames before memory allocation (sk_buff) or firewall evaluation occurs.
• tc: eth0(2) clsact/ingress ... id 505: Confirms an ingress Traffic Control classifier (program ID 505, action_eval) processes packets that pass through the XDP layer.
• clsact/egress ... id 509: Shows an egress TC classifier steering outbound flows via program ID 509.

What the Admin Does Next

Detach the faulty XDP driver filter using iproute2, then disassemble program 501 to locate the flawed boundary check:

sudo ip link set dev eth0 xdp off
sudo bpftool prog dump xlated id 501

Traffic will immediately normalize across eth0. You can then patch the eBPF program's MTU validation logic and safely reload it.

4. Diagnosing Container Network Isolation in Cgroups

Scenario

A container running inside a Kubernetes pod fails with EPERM (Operation Not Permitted) whenever it attempts to connect to an external financial payment gateway on port 8443. You must determine whether an inherited cgroup socket policy is blocking outbound connections.

Command

Audit the cgroup v2 hierarchy using bpftool cgroup:

sudo bpftool cgroup tree /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/pod_9c8d1e

Realistic Terminal Output

/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/pod_9c8d1e
sock_addr
    connect4 id 732 name restrict_egress_v4 attach_flags override
    connect6 id 733 name restrict_egress_v6 attach_flags override
sock_ops
    sock_opt_monitor id 734 name tcp_rtt_tracker
device
    dev_enforce id 735 name block_raw_sockets

Line-by-Line Technical Analysis

• /sys/fs/cgroup/.../pod_9c8d1e: Target cgroup directory path for the container workload.
• sock_addr: connect4 id 732 name restrict_egress_v4: An active socket-address hook (ID 732) intercepts every IPv4 connect(2) system call issued within the pod to evaluate destination addresses and ports.
• attach_flags override: Confirms this container policy overrides any default policies set by parent cgroup slices.
• device: dev_enforce id 735: Shows a device control program enforcing restrictions on raw socket instantiation.

What the Admin Does Next

Dump the bytecode of program 732 to review its port filtering rules:

sudo bpftool prog dump xlated id 732

The disassembled instructions will confirm that outbound traffic is restricted to ports 80 and 443. Update the pod's network security profile to allow egress on port 8443, resolving the issue cleanly without opening host-wide firewall exceptions.

5. Pinning Objects for Zero-Downtime Daemon Upgrades

Scenario

A mission-critical metrics daemon must be upgraded to a new release. The daemon tracks system latency histograms inside kernel maps. Restarting the daemon normally would close its file descriptors, causing the kernel to purge the maps and destroy all historical performance data. You need to pin the maps into the BPF filesystem, restart the service, and link the new binary to the existing kernel memory structures.

Command

Load the upgraded program object, map its memory handles to the existing pins under /sys/fs/bpf, and create a persistent execution link:

sudo bpftool prog load telemetry_v2.o /sys/fs/bpf/telemetry_v2 \
    map name latency_hist pinned /sys/fs/bpf/maps/latency_hist \
    type tracepoint

sudo bpftool link create id $(bpftool prog show name process_metrics -j | jq '.[0].id') \
    type tracepoint name sched_process_exec \
    pin /sys/fs/bpf/links/link_telemetry_v2

Realistic Terminal Output

# Verifying filesystem pins
# ls -la /sys/fs/bpf/maps/
total 0
drwxr-xr-x 2 root root 0 Aug 18 23:30 .
drwxr-xr-x 4 root root 0 Aug 18 23:00 ..
-rw------- 1 root root 0 Aug 18 23:15 latency_hist
-rw------- 1 root root 0 Aug 18 23:15 connection_tracker

# Confirming map retention across process exit
# bpftool map show pinned /sys/fs/bpf/maps/latency_hist
112: hash  name latency_hist  flags 0x0
    key_size 8  value_size 64  max_entries 65536  memlock 4718592B
    pinned /sys/fs/bpf/maps/latency_hist

Line-by-Line Technical Analysis

• bpftool prog load telemetry_v2.o /sys/fs/bpf/telemetry_v2: Compiles, verifies, and loads the new binary into kernel space, pinning its reference to /sys/fs/bpf/telemetry_v2.
• map name latency_hist pinned ...: Attaches the existing pinned map directly to the new program, preserving accumulated historical metrics without reallocating memory.
• bpftool link create ... pin ...: Establishes an atomic bpf_link handle that keeps the tracepoint active throughout daemon restart cycles.
• bpftool map show pinned ...: Confirms the kernel map remains resident in memory (memlock 4718592B) after userspace disconnection.

What the Admin Does Next

Restart the monitoring daemon. The new process will bind to the pinned map at /sys/fs/bpf/maps/latency_hist and resume metric collection without losing a single data point. Once confirmed, clean up deprecated pin handles:

sudo rm /sys/fs/bpf/telemetry_v1

Verifier Diagnostics, BTF Inspection, and Kernel Hardening

When deploying custom eBPF programs or troubleshooting failed loads, bpftool provides direct insight into verifier errors, kernel data layouts, and host security parameters.

Debugging Verifier Rejections

If a program fails verification, supply the -d (debug) flag to view the verifier's step-by-step register analysis:

sudo bpftool prog load bad_program.o /sys/fs/bpf/bad_program -d

libbpf: prog 'filter_traffic': BPF program load failed: Permission denied
libbpf: -- BEGIN PROG LOAD LOG --
0: R1=ctx(id=0,off=0,imm=0) R10=fp0
; int filter_traffic(struct __sk_buff *skb)
0: (79) r2 = *(u64 *)(r1 +104)   ; R2_w=pkt(id=0,off=0,r=0,imm=0)
1: (79) r3 = *(u64 *)(r1 +112)   ; R3_w=pkt_end(id=0,off=0,imm=0)
; if (r2 + 14 > r3)
2: (bf) r4 = r2
3: (07) r4 += 14
4: (2d) if r4 > r3 goto pc+5
; u16 proto = *(u16 *)(r2 + 12);
5: (69) r0 = *(u16 *)(r2 + 12)
invalid access to packet, off=12 size=2, R2(id=0,off=0,r=0)
R2 offset is outside of packet bounds
-- END PROG LOAD LOG --

The log points directly to instruction 5: the program attempted to read two bytes at offset 12 of the packet buffer (r2) without verifying that r2 + 14 did not exceed pkt_end (r3), prompting the verifier to reject the unsafe memory read.

Reconstructing Kernel Data Structures with BTF

Using bpftool btf, you can dump the running kernel's exact C struct declarations to confirm field alignments across different kernel releases:

sudo bpftool btf dump id 1 format c | head -n 25

This output reconstructs live kernel struct declarations (such as task_struct and sock), giving developers the precise struct layouts needed to write portable, multi-version programs.

Security Capabilities and Host Hardening

Managing eBPF programs in production requires specific Linux capabilities:

Enforce these kernel sysctl settings on production hosts to prevent unauthorized bytecode execution and protect against speculative execution attacks:

# Permanently restrict eBPF loading to privileged administrative processes
sudo sysctl -w kernel.unprivileged_bpf_disabled=2

# Enable and harden the JIT compiler against branch-target injection attacks
sudo sysctl -w net.core.bpf_jit_enable=1
sudo sysctl -w net.core.bpf_jit_harden=2

# Enable runtime BPF performance statistics collection
sudo sysctl -w kernel.bpf_stats_enabled=1

Common Pitfalls and How to Avoid Them

1. Memory Leaks via Orphaned Filesystem Pins

• The Trap: Crashed automation scripts or ungraceful daemon terminations can leave pinned maps behind in /sys/fs/bpf. Because these pins increment the kernel reference counter, the allocated memory remains locked (memlock) in non-swappable kernel space indefinitely.
• The Fix: Routinely audit your pinned directories and delete obsolete files: bash sudo ls -lR /sys/fs/bpf sudo rm /sys/fs/bpf/orphaned_map_name

2. Live Map Corruption via Endianness Errors

• The Trap: Supplying raw hexadecimal values to bpftool map update without accounting for architecture byte order will write incorrect values. On an x86_64 host (little-endian), inputting 0x00000001 in big-endian order stores decimal 16,777,216 instead of 1.
• The Fix: Verify every manual write immediately using bpftool map lookup, and script updates using structured JSON payloads where possible: bash sudo bpftool map lookup id <MAP_ID> key hex <KEY_BYTES>

3. Service Disruption via Abrupt Hook Removal

• The Trap: Detaching an active network eBPF filter on a high-throughput interface can immediately drop client traffic if that program handles stateful NAT or packet decapsulation.
• The Fix: Always inspect existing interface bindings with bpftool net show before making changes, and use atomic replacement workflows with bpf_link instead of detaching and reattaching live hooks.

Today's Takeaway

Log into one of your core Linux servers right now and run sudo bpftool prog show followed by sudo bpftool net show. In under five minutes, you will uncover the security policies, performance monitors, and network hooks running quietly inside your kernel—giving you complete transparency into what is actually happening beneath your applications.

Authoritative Documentation and References

• bpftool(8) Linux Manual Page
• Linux Kernel BPF Subsystem Documentation
• BPF Type Format (BTF) Specification
• ArchLinux eBPF Architecture and Tooling Guide
• Linux Kernel XDP Reference Guide