How to Tune a Linux Server That Falls Over Only at Peak

There is a kind of outage that survives every dashboard you own. Load average under one, CPU half idle, memory fine, and for four minutes at eight in the evening the site returns errors. Nothing on that dashboard measures the thing that broke, because the limits that bite at peak are queues and counters rather than utilisation. Once you know where those counters live you can watch the failure approach for a week before it lands.

Find the overflow before you touch a sysctl

When a TCP connection arrives and the application has not accepted it yet, the kernel parks it in the accept queue. If that queue is full the kernel drops the SYN, and the Linux SNMP counter documentation describes exactly what gets recorded. "When kernel receives a SYN from a client, and if the TCP accept queue is full, kernel will drop the SYN and add 1 to TcpExtListenOverflows. At the same time kernel will also add 1 to TcpExtListenDrops."

# absolute values since boot, including counters still sitting at zero
nstat -az TcpExtListenOverflows TcpExtListenDrops TcpExtTCPSynRetrans

The -a option dumps absolute values rather than the increment since the last run, and -z shows counters still sitting at zero so you can see the name before it moves. Run it now, note the numbers, and run it again after the next peak. If TcpExtListenOverflows moved, you have found your outage. TcpExtListenDrops moves on its own too, since the kernel increments it for any packet dropped on a listening socket, a memory allocation failure included.

There is a reason this failure looks like a timeout rather than an error. The kernel documents tcp_abort_on_overflow as "If listening service is too slow to accept new connections, reset them", and its "Default state is FALSE". Nothing is reset, nothing is logged, the SYN is simply dropped and the client retries.

The two numbers behind the accept queue

The application asks for a backlog when it calls listen(), and the kernel caps that request. From the manual page, "If the backlog argument is greater than the value in /proc/sys/net/core/somaxconn, then it is silently capped to that value." The same page gives the default and the version that changed it, "Since Linux 5.4, the default in this file is 4096; in earlier kernels, the default value is 128."

That is why raising somaxconn on its own often changes nothing. nginx asks for a backlog of 511 on Linux by default and keeps asking for 511 however high the cap goes, so both numbers have to move.

cat /proc/sys/net/core/somaxconn

# /etc/nginx/sites-available/example.conf
server {
    listen 443 ssl backlog=16384;
    server_name example.com;
}

net.ipv4.tcp_max_syn_backlog is a different queue and a common confusion. It holds the "Maximal number of remembered connection requests (SYN_RECV), which have not received an acknowledgment from connecting client", and its documented minimum is 128 on low memory machines, rising in proportion to RAM. A SYN flood fills that queue, a slow application fills the accept queue, and the two need different responses.

PHP-FPM is usually the real ceiling

On a PHP stack the accept queue is rarely the first thing to fill. The worker pool is, and PHP-FPM says so in words worth grepping for right now (the log path differs by distribution).

grep -E "reached pm.max_children|listening queue is not empty" /var/log/php*-fpm.log

The two lines the pool emits are "server reached pm.max_children setting (N), consider raising it" and "listening queue is not empty, #N requests are waiting to be served, consider raising pm.max_children setting (N)". Either one, timestamped at your peak hour, answers the whole question.

The live view is better than the log. Turn the status page on and read it during peak.

; /etc/php/8.3/fpm/pool.d/www.conf
pm = static
pm.max_children = 40
pm.status_path = /fpm-status

On that page, max listen queue is "The maximum number of requests seen in the listen queue at any one time" and max children reached answers "Has the maximum number of processes ever been reached?" as a count. Both are cumulative, so they say what the five minute graphs cannot.

Do not copy the 40. It is a budget rather than a setting, so measure what one worker actually costs on your codebase and divide the memory you are willing to hand PHP by that figure.

ps --no-headers -o rss= -C php-fpm8.3 | awk '{s+=$1; n++} END {print s/n/1024, "MB average per worker"}'

Ephemeral ports, and why a proxy runs out first

Every outbound connection consumes a local port, and the kernel documents the range with "The default values are 32768 and 60999 respectively", a little over 28,000 ports. That sounds like plenty until you remember the constraint applies per destination tuple, so a reverse proxy talking to one backend on one port, or a PHP tier talking to a single database host, burns through them fastest.

sysctl net.ipv4.ip_local_port_range
ss -tan state time-wait | wc -l

Widening the range buys time. Reusing connections removes the problem, and for nginx that means keepalive to the upstream, which needs three directives together and does nothing with only the first.

# /etc/nginx/conf.d/upstream.conf
upstream app_backend {
    server 127.0.0.1:8080;
    keepalive 16;
}

server {
    location / {
        proxy_pass http://app_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

net.ipv4.tcp_tw_reuse comes up in every thread on this subject. It takes 0 to disable, 1 for global enable and 2 for loopback traffic only, it already defaults to 2, and the kernel documentation says it "should not be changed without advice/request of technical experts".

Connection tracking, the limit nobody remembers enabling

If the host carries any stateful firewall rule, every flow takes a slot in the connection tracking table, and once that table is full new connections are dropped no matter how idle the CPU looks.

echo "$(cat /proc/sys/net/netfilter/nf_conntrack_count) of $(cat /proc/sys/net/netfilter/nf_conntrack_max)"
conntrack -S
conntrack -C

nf_conntrack_count is read-only and documented as the "Number of currently allocated flow entries", while nf_conntrack_max is the "Size of connection tracking table" whose "Default value is nf_conntrack_buckets value * 4".

Traffic is often not the reason the table fills. nf_conntrack_tcp_timeout_established defaults to 432000 seconds, which is five days, so a connection that ended badly can hold its slot for most of a week.

nginx has two limits of its own

worker_connections defaults to 512, and the documentation carries a warning people skip. "It should be kept in mind that this number includes all connections (e.g. connections with proxied servers, among others), not only connections with clients." A proxied request uses two. The same paragraph adds that the real ceiling is the open file limit, which worker_rlimit_nofile raises.

# /etc/nginx/nginx.conf
worker_processes auto;
worker_rlimit_nofile 65535;

events {
    worker_connections 8192;
}

keepalive_requests now defaults to 1000 requests per keep-alive connection, up from 100 before nginx 1.19.10, so a config that still pins it low makes every client re-handshake far more often than it needs to.

Make the changes survive a reboot

# /etc/sysctl.d/99-peak.conf
net.core.somaxconn = 16384
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_local_port_range = 10240 65535
net.netfilter.nf_conntrack_max = 262144

sudo sysctl --system
sudo sysctl net.core.somaxconn net.ipv4.ip_local_port_range

Those values are starting points to be sized against the counters you just read, not numbers to adopt on faith. The nf_conntrack_max line only applies once the nf_conntrack module is loaded, so on a host with no firewall rules it is ignored at boot and quietly appears later.

Read the counters again after the next peak. If TcpExtListenOverflows is flat and max children reached is still climbing, the queue simply moved to PHP, and that is where the next change belongs.

If you would rather have this measured against your own traffic than worked through as a checklist, that is what server setup and optimization covers, and keeping it correct as the traffic grows is servers management.