Logrotate: Managing High-Throughput Log Truncation, Automating Compression Pipelines, and Preventing Storage Saturation in Production

What you are witnessing is a rite of passage for systems engineers: an orphaned file descriptor. Hours earlier, someone deleted an oversized log file with a hasty rm command, but because the running application was never told to let go of the file, the operating system kernel continued pouring data into an invisible ghost handle. As the node collapses under its own unmanaged chatter, an inescapable reality becomes clear: production infrastructure rarely dies from elegant architectural flaws; it suffocates because someone forgot to take out the digital rubbish.

The standard tool standing between round-the-clock reliability and a storage-driven outage is logrotate(8). Operating as an automated custodian, logrotate cycles active log files, compresses older archives, prunes obsolete records, and coordinates with running services so they seamlessly swap files without dropping a single active network connection.

If there is one command every administrator should commit to memory before touching a live server, it is logrotate's non-destructive dry-run mode:

logrotate -d /etc/logrotate.conf

reading config file /etc/logrotate.conf
including /etc/logrotate.d
reading config file /etc/logrotate.d/rsyslog
reading config file /etc/logrotate.d/nginx
Allocating hash table for state file, size 64 entries

Handling 2 logs

rotating pattern: /var/log/nginx/*.log  after 1 days (14 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/nginx/access.log
  log does not need rotating (log has been rotated at 2026-08-18 00:00, that is not day ago yet)
considering log /var/log/nginx/error.log
  log does not need rotating (log has been rotated at 2026-08-18 00:00, that is not day ago yet)

Running logrotate -d reads your entire configuration tree, parses every directive, checks the timestamps recorded in the state file, and prints an exact blow-by-blow itinerary of what it would do—without moving, altering, or compressing a single byte on disk. It is the ultimate sanity check before applying any policy change.

What It Does in Plain English

Every running service on a Linux server—from web servers like Nginx to custom backend APIs—spews continuous operational telemetry. Left unchecked, a single chatty service can fill hundreds of gigabytes of storage within weeks, choking the host filesystem.

logrotate steps in to manage this lifecycle. At scheduled intervals or when specific file size thresholds are crossed, it renames the current log file, instructs the generating application to start writing to a fresh file, compresses historical records with gzip or zstd to save disk space, and eventually deletes expired archives once they exceed a defined retention limit.

Operating System Mechanics: How Files and Inodes Actually Work

To understand why simple log deletion fails and how logrotate avoids data loss, one must examine how the Linux Virtual File System (VFS) handles files.

When a program opens a log path like /var/log/application.log using the open(2) system call, the kernel translates the friendly path string into an index node—an inode—which points directly to the raw data blocks on the storage drive. The running program receives a numeric handle known as a file descriptor (fd).

From this point forward, every write(2) call from the program bypasses the file's name and writes directly to the underlying inode.

If an operator runs rm /var/log/application.log, the kernel executes unlink(2), removing the filename from the directory structure and decreasing the inode's link count (i_nlink). However, physical disk blocks are never released back to the free pool until every running process referencing that file descriptor closes its handle (f_count reaches zero). Because the program keeps writing into the open handle, the disk stays full while standard directory inspection tools like du see nothing.

logrotate resolves this challenge using two primary operational strategies:

1. Atomic Renaming with Signal Re-notification (create): logrotate renames the active log file (for example, moving access.log to access.log.1). The running application continues writing without interruption to the renamed inode. logrotate creates a brand-new, empty log file with appropriate permissions at the original path. Finally, via a postrotate script, logrotate sends an asynchronous POSIX signal (such as SIGHUP or SIGUSR1) telling the daemon to close its old file descriptor and re-open the original file path. The service smoothly attaches to the new inode without dropping a connection.
2. In-Place Byte Truncation (copytruncate): Designed for legacy applications or container runtimes that cannot reload file descriptors on command. logrotate makes a copy of the active log file to an archive path and then immediately truncates the original file in place using ftruncate(2). The inode remains identical, the file pointer resets to zero, and the application continues writing without a restart.

Core Command-Line Flags

Five Real-World Production Implementations

1. Zero-Downtime Edge Web Server Log Rotation

The Scenario: An Nginx reverse proxy processes tens of thousands of requests per second. Restarting the master process would terminate active customer TLS connections and drop requests. Logs must be rotated and compressed daily without dropping a single write.

Configuration File: /etc/logrotate.d/nginx

/var/log/nginx/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 $(cat /var/run/nginx.pid)
        fi
    endscript
}

Execution & Validation Command

logrotate -vf /etc/logrotate.d/nginx

Realistic Terminal Output

reading config file /etc/logrotate.d/nginx
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Handling 1 logs

rotating pattern: /var/log/nginx/*.log  forced from command line (30 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/nginx/access.log
  log is forcedly rotated
considering log /var/log/nginx/error.log
  log is forcedly rotated
rotating log /var/log/nginx/access.log, log->rotateCount is 30
dateext suffix '-20260818'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/log/nginx/access.log to /var/log/nginx/access.log.1
creating new /var/log/nginx/access.log mode = 0640 uid = 33 gid = 4
rotating log /var/log/nginx/error.log, log->rotateCount is 30
renaming /var/log/nginx/error.log to /var/log/nginx/error.log.1
creating new /var/log/nginx/error.log mode = 0640 uid = 33 gid = 4
running postrotate script
running script with arg /var/log/nginx/*.log : "
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 $(cat /var/run/nginx.pid)
        fi
"
compressing log with: /bin/gzip

Line-by-Line Explanation

• renaming /var/log/nginx/access.log to ...access.log.1: logrotate renames the file on disk instantly. Nginx worker processes continue streaming log lines to this renamed file without pause.
• creating new /var/log/nginx/access.log mode = 0640 uid = 33 gid = 4: A fresh file is created, owned by the web server user www-data (UID 33) and group adm (GID 4), ensuring secure permissions (0640).
• sharedscripts ... endscript: Ensures that the postrotate block runs only once after all matching wildcard log files have been renamed, rather than once per file.
• kill -USR1 $(cat /var/run/nginx.pid): Sends SIGUSR1 to Nginx. Nginx handles this signal by seamlessly re-opening its log files using their original paths, binding workers to the new inode without terminating active client connections.
• delaycompress: Postpones gzip compression of access.log.1 until the next rotation cycle, preventing race conditions if a worker is still finishing an active write buffer during the signal reload.

What the Admin Does Next

Verify that Nginx has successfully rebound its file descriptors to the new file by running:

lsof -p $(pgrep -f "nginx: master") | grep log

Confirm that the open descriptor points to the new inode number shown by stat -c %i /var/log/nginx/access.log.

2. In-Place Log Truncation for Legacy and Containerized Daemons

The Scenario: A proprietary Java billing service runs inside a secured container. The application does not support POSIX signals; sending a SIGHUP causes the runtime to terminate abruptly. It keeps an exclusive lock on its log file. You need to rotate the log without crashing the application.

Configuration File: /etc/logrotate.d/billing-engine

/opt/billing/logs/engine.log {
    weekly
    rotate 12
    copytruncate
    nocreate
    missingok
    compress
    size 1G
}

Execution & Validation Command

logrotate -vf /etc/logrotate.d/billing-engine

Realistic Terminal Output

reading config file /etc/logrotate.d/billing-engine
Reading state from file: /var/lib/logrotate/status
Handling 1 logs

rotating pattern: /opt/billing/logs/engine.log  forced from command line (12 rotations)
empty log files are not rotated, old logs are removed
considering log /opt/billing/logs/engine.log
  log is forcedly rotated
rotating log /opt/billing/logs/engine.log, log->rotateCount is 12
renaming /opt/billing/logs/engine.log.12.gz to /opt/billing/logs/engine.log.13.gz (rotatecount 12, logname /opt/billing/logs/engine.log)
old log /opt/billing/logs/engine.log.13.gz deleted
copying /opt/billing/logs/engine.log to /opt/billing/logs/engine.log.1
truncating /opt/billing/logs/engine.log
compressing log with: /bin/gzip

Line-by-Line Explanation

• copying /opt/billing/logs/engine.log to /opt/billing/logs/engine.log.1: logrotate copies the contents of the live file to the archive target.
• truncating /opt/billing/logs/engine.log: logrotate calls ftruncate(2) on the live file. The kernel resets the file size to 0 and reclaims the allocated storage blocks while leaving the file open and its inode intact.
• nocreate: Tells logrotate not to create a new file, since the existing file was truncated in place.
• compress: Compresses the copied backup file (engine.log.1) into engine.log.1.gz.

What the Admin Does Next

Check the log directory using ls -lh /opt/billing/logs/ to confirm that engine.log has shrunk to zero bytes while the Java process remains healthy and running under the same PID.

3. Size-Driven Emergency Thresholding with High-Frequency Dispatch

The Scenario: A high-velocity data ingestion engine experiences sudden, unpredictable bursts of traffic. If a surge occurs midday, waiting for a once-a-day cron job could result in an out-of-space failure. You need an automated policy that rotates logs whenever they exceed 500MB, evaluated on an hourly schedule.

Configuration File: /etc/logrotate.d/telemetry-burst

/var/log/telemetry/*.log {
    hourly
    maxsize 500M
    rotate 48
    compress
    delaycompress
    missingok
    notifempty
    create 0644 telemetry telemetry
    postrotate
        /usr/bin/systemctl kill -s HUP telemetry-collector.service
    endscript
}

High-Frequency Systemd Timer Setup

Modern Linux distributions manage scheduled jobs using systemd.timer(5). You can override the default daily schedule to run hourly:

mkdir -p /etc/systemd/system/logrotate.timer.d/
cat << 'EOF' > /etc/systemd/system/logrotate.timer.d/hourly.conf
[Timer]
OnCalendar=
OnCalendar=*-*-* *:00:00
AccuracySec=1m
EOF

systemctl daemon-reload
systemctl restart logrotate.timer

Execution & Validation Command

logrotate -v /etc/logrotate.d/telemetry-burst

Realistic Terminal Output

reading config file /etc/logrotate.d/telemetry-burst
Reading state from file: /var/lib/logrotate/status
Handling 1 logs

rotating pattern: /var/log/telemetry/*.log  hourly (48 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/telemetry/ingest.log
  log size is 524288001 bytes, maxsize is 524288000 --> rotating log
rotating log /var/log/telemetry/ingest.log, log->rotateCount is 48
renaming /var/log/telemetry/ingest.log to /var/log/telemetry/ingest.log.1
creating new /var/log/telemetry/ingest.log mode = 0644 uid = 1002 gid = 1002
running postrotate script
running script with arg /var/log/telemetry/*.log : "
        /usr/bin/systemctl kill -s HUP telemetry-collector.service
"
compressing log with: /bin/gzip

Line-by-Line Explanation

• hourly: Configures logrotate to evaluate rotation intervals with hourly granularity.
• log size is 524288001 bytes, maxsize is 524288000 --> rotating log: logrotate queries the file metadata with stat(). Because the file size exceeds the 500MB maxsize threshold, rotation triggers immediately.
• systemctl kill -s HUP telemetry-collector.service: Sends a SIGHUP signal across the systemd service control group, causing the telemetry daemon and all child workers to refresh their log handles.

What the Admin Does Next

Confirm that the new hourly schedule is active by inspecting the system timer list:

systemctl list-timers logrotate.timer

Ensure that the NEXT column shows the top of the upcoming hour.

4. Multi-Tenant Security Hardening with Explicit Isolation

The Scenario: A financial transaction engine writes audit records containing sensitive customer information to /var/log/audit-app/. The service runs under an unprivileged user called secops. If log rotation runs with default root permissions or lax file masks, local users could exploit symlink races or access sensitive customer data.

Configuration File: /etc/logrotate.d/compliance-audit

/var/log/audit-app/transaction.log {
    su secops secops
    monthly
    rotate 84
    create 0600 secops secops
    dateext
    dateformat -%Y%m%d-%s
    compress
    missingok
    notifempty
    sharedscripts
    prerotate
        /usr/local/bin/verify-audit-signature.sh /var/log/audit-app/transaction.log
    endscript
    postrotate
        /usr/bin/pkill -HUP -u secops audit-daemon
    endscript
}

Execution & Validation Command

logrotate -vf /etc/logrotate.d/compliance-audit

Realistic Terminal Output

reading config file /etc/logrotate.d/compliance-audit
Reading state from file: /var/lib/logrotate/status
Handling 1 logs

rotating pattern: /var/log/audit-app/transaction.log  forced from command line (84 rotations)
empty log files are not rotated, old logs are removed
switching euid to 1005 and egid to 1005
considering log /var/log/audit-app/transaction.log
  log is forcedly rotated
running prerotate script
running script with arg /var/log/audit-app/transaction.log : "
        /usr/local/bin/verify-audit-signature.sh /var/log/audit-app/transaction.log
"
rotating log /var/log/audit-app/transaction.log, log->rotateCount is 84
dateext suffix '-20260818-1755486118'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]-[0-9]*'
renaming /var/log/audit-app/transaction.log to /var/log/audit-app/transaction.log-20260818-1755486118
creating new /var/log/audit-app/transaction.log mode = 0600 uid = 1005 gid = 1005
running postrotate script
running script with arg /var/log/audit-app/transaction.log : "
        /usr/bin/pkill -HUP -u secops audit-daemon
"
compressing log with: /bin/gzip
switching euid to 0 and egid to 0

Line-by-Line Explanation

• switching euid to 1005 and egid to 1005: logrotate drops superuser privileges before touching the target directory, switching effective user and group to secops. This completely neutralizes symlink escalation vulnerabilities (CVE-2011-1098) where an unprivileged user points a symlink to a sensitive path like /etc/shadow.
• prerotate ... endscript: Executes an integrity verification script before touching the filesystem. If this script exits with an error code, logrotate immediately aborts the pipeline, preserving the log in its untouched state.
• dateext and dateformat -%Y%m%d-%s: Replaces default numeric extensions (.1, .2) with explicit calendar dates and Unix epoch timestamps, making archives easy to cross-reference with audit logs.
• create 0600 secops secops: Restricts permissions so that only the secops user can read or write to the file (chmod 600), blocking all other system users.
• switching euid to 0 and egid to 0: logrotate safely restores its privileged context after finishing the job.

What the Admin Does Next

Inspect the resulting file attributes in the audit directory:

ls -l /var/log/audit-app/

Verify that all rotated archives are restricted to 0600 permissions and owned by secops:secops.

5. Non-Destructive Configuration Auditing and Force Testing

The Scenario: You are reviewing an automated infrastructure deployment that updated multiple custom rotation rules across /etc/logrotate.d/. You need to audit configuration syntax, check last-rotated timestamps, and safely test a single rule without altering production state.

Auditing the System State File

The /var/lib/logrotate/status file acts as the single source of truth for rotation timestamps:

head -n 10 /var/lib/logrotate/status

logrotate state -- version 2
"/var/log/nginx/access.log" 2026-8-18-0:0:0
"/var/log/nginx/error.log" 2026-8-18-0:0:0
"/var/log/syslog" 2026-8-18-0:0:0
"/var/log/auth.log" 2026-8-1-0:0:0
"/var/log/telemetry/ingest.log" 2026-8-18-3:0:0
"/var/log/audit-app/transaction.log" 2026-8-1-0:0:0

Diagnostic Execution (Dry-Run Debug)

Run a system-wide dry run to evaluate rules and status records without touching files:

logrotate -d /etc/logrotate.conf

reading config file /etc/logrotate.conf
including /etc/logrotate.d
reading config file /etc/logrotate.d/compliance-audit
reading config file /etc/logrotate.d/nginx
reading config file /etc/logrotate.d/telemetry-burst
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries

Handling 3 logs

rotating pattern: /var/log/audit-app/transaction.log  monthly (84 rotations)
empty log files are not rotated, old logs are removed
switching euid to 1005 and egid to 1005
considering log /var/log/audit-app/transaction.log
  log does not need rotating (log has been rotated at 2026-8-1 0:0:0, that is not month ago yet)
switching euid to 0 and egid to 0

rotating pattern: /var/log/telemetry/*.log  hourly (48 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/telemetry/ingest.log
  log size is 104857600 bytes, maxsize is 524288000 --> log does not need rotating

Line-by-Line Explanation

• -d / --debug: Completely disables filesystem writes. No files are moved, no scripts are executed, and /var/lib/logrotate/status remains unmodified.
• Reading state from file: /var/lib/logrotate/status: Reads recorded timestamps to calculate elapsed time against configuration directives.
• log does not need rotating (... that is not month ago yet): logrotate confirms that the required one-month interval has not yet passed.
• log size is 104857600 bytes, maxsize is 524288000 --> log does not need rotating: Confirms that the current 100MB file has not breached the 500MB emergency limit.

What the Admin Does Next

If you want to perform a live rotation test on a single configuration without modifying production tracking records, supply an isolated temporary state file:

logrotate -v -f -s /tmp/test-logrotate.status /etc/logrotate.d/target-config

This forces execution of target-config while writing its status records harmlessly to /tmp/test-logrotate.status, leaving your production state file untouched.

What Can Go Wrong: Failure Modes and Edge Cases

1. The copytruncate Race Condition and Data Loss

While copytruncate is convenient for stubborn applications that cannot reload log handles, it suffers from a fundamental design limitation under heavy traffic: copying a file and truncating it are separate, non-atomic operations.

During the time logrotate spends copying a multi-gigabyte file, the application keeps writing new log entries to the end of that file. When logrotate completes the copy and calls ftruncate(), any data written between the start of the copy and the moment of truncation is permanently lost.

2. Phantom Inode Leaks and the df/du Discrepancy

When an unmanaged log file threatens to fill a disk, an engineer's first instinct might be to remove it with rm /var/log/huge-service.log.

This removes the name from the directory tree, but as long as the service holds the file descriptor open, the storage blocks remain allocated. The du command reports that the directory is empty, yet df continues to report that the volume is 100% full.

To locate these hidden open files across your system, inspect unlinked file handles with lsof:

lsof +L1 /var

COMMAND      PID USER   FD   TYPE DEVICE   SIZE/OFF NLINK      NODE NAME
bad-daemon 14592 root    3w   REG  254,1 94371840000     0 134217730 /var/log/unmanaged-service.log (deleted)

3. Insecure Directory Permissions and su Aborts

If a log directory is configured with overly permissive access (such as a shared directory configured with chmod 0777), logrotate will refuse to run and log an explicit error:

error: skipping "/var/log/shared/app.log" because parent directory has insecure permissions (IT HAS GROUP OR OTHERS WRITABLE PERMISSIONS)

This security mechanism prevents local unprivileged users from replacing log files with symbolic links to sensitive system files (such as /etc/shadow) right before logrotate runs with root privileges.

Today's Takeaway

Log management is not an afterthought—it is a core safeguard for system uptime and storage reliability. Open a terminal right now and run logrotate -d /etc/logrotate.conf. This harmless, read-only audit will show you exactly how your system handles log growth, reveal any syntax warnings in your /etc/logrotate.d/ directory, and verify your state tracking file at /var/lib/logrotate/status. Taking five minutes today to audit your rotation policies will ensure you never find yourself debugging an invisible, 100%-full disk at two in the morning.

Authoritative References & Further Reading

• Linux Programmer's Manual: logrotate(8)
• Linux Programmer's Manual: open(2) and File Flags
• Linux Programmer's Manual: unlink(2) and Inode Lifecycle
• Systemd Reference Manual: systemd.timer(5)
• ArchWiki Comprehensive Guide to Logrotate
• Nginx Administration: Controlling Process Logging Signals