Prlimit: Dynamically Adjusting Process Resource Limits, Overriding File Descriptor Caps, and Mitigating Outages in Production

Under normal circumstances, the standard fix is straightforward but painful: update a configuration file and reboot the service. But at peak volume, restarting a core system is the nuclear option. A restart instantly severs every active connection, dumps cached encryption handshakes, and risks triggering a cascading stampede of reconnecting clients that can easily knock offline the rest of the infrastructure. The engineer is trapped between a failing service and a cure that might kill the system entirely.

What is needed in this high-stakes moment is surgical precision: a way to step inside the running Linux system and widen the boundaries of the live process without interrupting it for even a microsecond.

This is the exact problem solved by prlimit(1). With a single targeted command, an administrator can dynamically raise the ceiling on open files for an active process ID (PID) on the fly:

sudo prlimit --pid 14820 --nofile=65536:131072

The instant this command executes, the operating system raises the active limit to 65,536 files with a maximum headroom of 131,072. The error log goes quiet, the backlog clears, and live traffic continues flowing uninterrupted—all without a single second of downtime.

How Linux Sets Boundaries: The Kernel's Hidden Ceilings

To understand how prlimit performs this live rescue, it helps to look at how Unix systems protect themselves from runaway programs. For decades, POSIX operating systems have used resource limits to ensure that a single buggy script or runaway process cannot hog all available memory, monopolise CPU cycles, or exhaust file handles.

Historically, these boundaries were managed through the getrlimit(2) and setrlimit(2) system calls. However, these older mechanisms suffered from a frustrating blind spot: a program could only alter its own resource limits or configure defaults for child processes it was about to launch. Modifying the boundaries of an already running, independent process from the outside was impossible without attaching a heavy debugger like GDB—a risky, intrusive procedure that could freeze or crash production systems.

Linux resolved this limitation with the introduction of the prlimit64(2) system call. The prlimit command-line utility provides a direct, non-invasive interface to this capability:

Soft Ceilings vs. Hard Ceilings

Inside the Linux kernel, every process is tracked by an internal structure known as task_struct. Its resource limits live in an associated signal_struct, shared across all threads within the application:

struct rlimit {
    rlim_t rlim_cur;  /* Soft limit: the current operational ceiling */
    rlim_t rlim_max;  /* Hard limit: the maximum ceiling unprivileged code can reach */
};

The kernel treats these two values very differently: * The Soft Limit (rlim_cur): This is the working operational ceiling. When a process exceeds this limit, the kernel intervenes immediately. For file descriptors (RLIMIT_NOFILE), any new request to open a file or socket returns -EMFILE. For CPU time (RLIMIT_CPU), the kernel sends a warning SIGXCPU signal. * The Hard Limit (rlim_max): This acts as a safety barrier. An unprivileged program can adjust its own soft limit up or down, but it can never exceed its hard limit. Raising the hard ceiling requires administrative elevation.

Syscall Mechanics and Security

When you invoke prlimit, the kernel executes an atomic read-modify-write operation using prlimit64:

int prlimit64(pid_t pid, int resource, 
              const struct rlimit64 *new_limit, 
              struct rlimit64 *old_limit);

Before applying any change, the kernel performs strict security checks governed by capabilities(7). To modify another process, the caller must either share the same user ID or hold the CAP_SYS_RESOURCE capability (standard for the root user). If authorised, the kernel updates the internal table instantly and atomically.

You can inspect these limits at any time in read-only form through the proc(5) filesystem by viewing /proc/[pid]/limits. However, while /proc is strictly a window for reading state, prlimit allows bidirectional inspection and live mutation.

Core Syntax and Baseline Inspection

The syntax of prlimit is designed to be straightforward for both interactive rescue work and automated scripts.

How to Assign Values

• --nofile=4096:8192 sets the soft limit to 4,096 and the hard limit to 8,192.
• --nofile=1024: updates only the soft limit to 1,024, leaving the hard limit unchanged.
• --nofile=:65535 leaves the soft limit alone and raises the hard ceiling to 65,535.
• --nofile=unlimited removes the ceiling entirely.

Auditing a Live Process

To see the full operational matrix of any running service (for instance, PID 1420), run:

prlimit --pid 1420

RESOURCE   DESCRIPTION                             SOFT      HARD UNITS
AS         address space limit                unlimited unlimited bytes
CORE       max core file size                         0 unlimited bytes
CPU        CPU time                           unlimited unlimited seconds
DATA       max data size                      unlimited unlimited bytes
FSIZE      max file size                      unlimited unlimited bytes
LOCKS      max file locks count               unlimited unlimited 
MEMLOCK    max locked-in-memory address space     65536     65536 bytes
MSGQUEUE   max bytes in POSIX mqueues            819200    819200 bytes
NICE       max nice prio allowed to raise             0         0 
NOFILE     max number of open files                1024      4096 files
NPROC      max number of processes                62788     62788 processes
RSS        max resident set size              unlimited unlimited bytes
RTPRIO     max real-time priority                     0         0 
RTTIME     timeout for real-time tasks        unlimited unlimited microsecs
SIGPENDING max number of pending signals          62788     62788 signals
STACK      max stack size                       8388608 unlimited bytes

Five Real-World Production Case Studies

Use Case 1: Auditing and Unblocking a Throttling Database

The Scenario: A core PostgreSQL database instance (PID 4182) begins rejecting background analytics tasks and worker processes. Server memory and CPU graphs look healthy, but database logs report worker spawn errors. The administrator needs to inspect the live process limits to locate the bottleneck.

Execution:

sudo prlimit --pid 4182 --output RESOURCE,DESCRIPTION,SOFT,HARD,UNITS

Terminal Output:

RESOURCE   DESCRIPTION                             SOFT      HARD UNITS
AS         address space limit                unlimited unlimited bytes
CORE       max core file size                         0 unlimited bytes
CPU        CPU time                           unlimited unlimited seconds
DATA       max data size                      unlimited unlimited bytes
FSIZE      max file size                      unlimited unlimited bytes
LOCKS      max file locks count               unlimited unlimited 
MEMLOCK    max locked-in-memory address space  67108864  67108864 bytes
MSGQUEUE   max bytes in POSIX mqueues            819200    819200 bytes
NICE       max nice prio allowed to raise             0         0 
NOFILE     max number of open files                1024      4096 files
NPROC      max number of processes                  256       512 processes
RSS        max resident set size              unlimited unlimited bytes
RTPRIO     max real-time priority                     0         0 
RTTIME     timeout for real-time tasks        unlimited unlimited microsecs
SIGPENDING max number of pending signals         128100    128100 signals
STACK      max stack size                       8388608  16777216 bytes

Line-by-Line Breakdown: * NPROC max number of processes 256 512 processes: This reveals the root cause. Under Linux, RLIMIT_NPROC limits the total processes and threads for the entire system user. Background jobs have exhausted the allocation of 256, blocking any new forks. * NOFILE max number of open files 1024 4096 files: The default soft limit of 1,024 leaves almost no margin for complex queries touching multiple tables simultaneously. * MEMLOCK max locked-in-memory address space 67108864 67108864 bytes: Memory locking is constrained to 64 MiB, limiting database performance optimisations like huge pages.

What the Administrator Does Next: The engineer immediately raises both process and file handle limits on the live daemon:

sudo prlimit --pid 4182 --nproc=4096:8192 --nofile=65536:65536

To ensure the fix persists across future service restarts, the administrator then adds LimitNPROC=8192 and LimitNOFILE=65536 to the database service unit in systemd.

Use Case 2: Expanding File Descriptors (NOFILE) Under Connection Spikes

The Scenario: An NGINX reverse proxy worker (PID 14820) receives an unexpected surge of traffic during a major promotion. The error logs start filling with accept4() failed (24: Too many open files). Restarting the proxy would drop thousands of active TLS connections.

Execution:

sudo prlimit --pid 14820 --nofile=65536:131072

Terminal Output:

RESOURCE DESCRIPTION                 SOFT   HARD UNITS
NOFILE   max number of open files   65536 131072 files

Line-by-Line Breakdown: * prlimit issues an atomic system call directly to kernel space for process 14820. * The kernel updates the descriptor bounds in memory, lifting the soft ceiling to 65,536 and the hard maximum to 131,072. * Subsequent accept4() and socket() requests from the proxy loop succeed immediately, as the kernel checks file descriptors against the new 65,536 ceiling.

What the Administrator Does Next: The engineer verifies the change directly in /proc:

grep "Max open files" /proc/14820/limits

Max open files            65536                131072               files     

The error rates drop to zero, and the proxy handles the surge smoothly.

Use Case 3: Imposing Virtual Memory Ceilings (AS) on a Leaking Process

The Scenario: A background data-processing worker written in Python (PID 29104) develops an uncontained memory leak. Left unchecked, it will consume all available host RAM and provoke the Linux Out-Of-Memory (OOM) killer into terminating critical neighboring databases. The engineer needs to put the runaway worker in a safety cage while it finishes its queue.

Execution:

# Soft ceiling: 4 GiB (4294967296 bytes), Hard ceiling: 5 GiB (5368709120 bytes)
sudo prlimit --pid 29104 --as=4294967296:5368709120

Terminal Output:

RESOURCE DESCRIPTION                 SOFT       HARD UNITS
AS       address space limit   4294967296 5368709120 bytes

Line-by-Line Breakdown: * RLIMIT_AS bounds the total virtual memory footprint (code, stack, heap, and mapped assets). * By setting the soft boundary to 4 GiB, any attempt by Python to allocate additional memory beyond this threshold causes the kernel to return -ENOMEM. * Rather than suffering a sudden crash or taking down the entire machine, the Python runtime catches this allocation failure cleanly as a MemoryError, allowing it to log its state and perform a controlled shutdown.

What the Administrator Does Next: The engineer monitors /proc/29104/status to ensure virtual memory stabilizes below the 4 GiB threshold, and files a bug report for the development team to patch the leak.

Use Case 4: Enabling Live Crash Dumps (CORE) for Intermittent Faults

The Scenario: A compiled C++ backend service (PID 8831) experiences intermittent crashes (SIGSEGV) once every few days under specific workloads. Because production servers default to disabling crash dumps (RLIMIT_CORE = 0), no diagnostic memory snapshot is generated when it fails. The engineer must enable crash dumping on the running service without restarting it.

Execution:

sudo prlimit --pid 8831 --core=unlimited:unlimited

Terminal Output:

RESOURCE DESCRIPTION                 SOFT      HARD UNITS
CORE     max core file size     unlimited unlimited bytes

Line-by-Line Breakdown: * RLIMIT_CORE is adjusted from 0 to unlimited. * When a segmentation fault next occurs, the kernel verifies the core limit. Finding no restriction, it writes a complete memory image to disk according to /proc/sys/kernel/core_pattern.

What the Administrator Does Next: The engineer checks the configured crash dump receiver:

cat /proc/sys/kernel/core_pattern

|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h

When the application next crashes, the full core dump is captured and ready for immediate forensic debugging:

gdb /opt/microservices/bin/ordersvc /var/lib/systemd/coredump/core.ordersvc.8831

Use Case 5: Imposing a CPU Leash on Runaway Calculations

The Scenario: A compute-heavy scientific simulation (PID 33100) occasionally enters infinite mathematical loops that peg a CPU core at 100%. Service agreements specify that no single computation should consume more than 5 minutes (300 seconds) of processing time, with a 60-second grace window to clean up before hard termination.

Execution:

# Soft limit: 300 seconds, Hard limit: 360 seconds
sudo prlimit --pid 33100 --cpu=300:360

Terminal Output:

RESOURCE DESCRIPTION           SOFT  HARD UNITS
CPU      CPU time               300   360 seconds

Line-by-Line Breakdown: * RLIMIT_CPU tracks pure processing time consumed on the CPU, rather than wall-clock elapsed time. * Soft Limit Enforcement (300s): At 300 seconds of active calculation, the kernel delivers a SIGXCPU signal. The program can catch this signal, save its current progress to disk, and exit cleanly. * Hard Limit Enforcement (360s): If the program ignores the signal or remains trapped in a tight computational loop, the kernel sends an uncatchable SIGKILL at 360 seconds to free the processor.

What the Administrator Does Next: The administrator tracks the process via ps:

ps -o pid,user,time,cputime,state,cmd -p 33100

If the job exceeds its allocation, it terminates cleanly, returning exit code 152 (SIGXCPU) or 137 (SIGKILL).

Architectural Context: Limits, Inheritance, and Cgroups

Understanding how limits move through the operating system is critical to using them effectively.

How Child Processes Inherit Limits

When a process calls fork(), the child process inherits an identical copy of the parent’s resource table. Any change made with prlimit to a running parent process applies only to future children spawned after the change; siblings and children that were already running retain their original limits. When execve() is called to run a new binary, those resource limits remain intact across the transition unless security boundaries (like setuid binaries) intervene.

POSIX Limits vs. Control Groups (cgroups v2)

Modern Linux administrators often ask how POSIX resource limits relate to Control Groups (cgroups). While both enforce boundaries, they work at completely different levels:

Making Runtime Changes Permanent

Because prlimit modifies kernel state in memory, its changes do not survive a server reboot or service restart. To make your changes permanent, record them in the configuration layer:

• For Systemd Services: Add resource limits directly to the service unit file, as documented in systemd.exec(5): ini [Service] LimitNOFILE=65536:131072 LimitAS=5368709120 LimitCORE=infinity LimitCPU=300:360
• For User Sessions: Edit /etc/security/limits.conf to set baseline boundaries for interactive user logins: ``` # postgres soft nofile 65536 postgres hard nofile 131072
  • soft core unlimited ```

Operational Pitfalls to Avoid

1. The One-Way Hard Limit Trap: An unprivileged process can lower its own hard limit at will, but it can never raise it again—even back to what it was. If a program accidentally drops its own hard limit, only an administrator with CAP_SYS_RESOURCE can restore it using prlimit.
2. Unit Mismatches: Different resources use different units of measurement:
   • AS, CORE, DATA, FSIZE, MEMLOCK, and STACK are measured in bytes.
   • CPU is measured in seconds.
   • NOFILE, NPROC, LOCKS, and SIGPENDING are raw counts.
   • NICE represents priority scale offsets ($0 \dots 39$).

Always verify your target units using --output RESOURCE,SOFT,HARD,UNITS before modifying a live production workload.

Quick Reference Guide: System Resources and Directives

The following matrix maps Linux resource options to their kernel constants, units, and corresponding configuration keys in systemd, based on the GNU C Library Resource Limits Manual and Linux Kernel Documentation.

Today's Takeaway

You do not need to wait for a 2am outage to put prlimit to work. Open your terminal right now and run:

prlimit --pid $$

This simple command reveals the exact resource envelope governing your current shell session (represented by the special $$ variable). Look down the list at NOFILE (open files) and NPROC (maximum child processes). Now, test giving your shell extra headroom by running prlimit --pid $$ --nofile=2048:4096. Run prlimit --pid $$ once more, and you will see your new soft and hard limits active immediately. In under five minutes, you have learned how to inspect the kernel's process boundaries and adjust them on the fly—a vital skill that turns potential production emergencies into quick, non-disruptive fixes.