How to Block an Attack Without Blocking Your Own Client

The rate limit that finally stops the credential stuffing is also the one that locks out your customer's head office at five past nine on Monday, or quietly drops a payment provider's callback and leaves a hundred orders sitting unpaid. Both failures come from the same mistake, which is deciding who to block before you can reliably tell who is who.

The setup below stops abuse and leaves a documented, testable path for every address that must never be touched.

Get the real client address right before anything else

If a CDN or a load balancer sits in front of nginx, every request arrives from the proxy's address. Rate limit on that and you throttle the CDN, which means you throttle everybody at once.

nginx's real IP module fixes it, and it is not compiled in by default, so check before you rely on it.

nginx -V 2>&1 | tr ' ' '\n' | grep realip

Behind Cloudflare the header to trust is CF-Connecting-IP, documented as "The client IP address connecting to Cloudflare to the origin web server." The trusted range list should be generated from Cloudflare's published files rather than typed out, because it changes.

# regenerate the trusted proxy list from Cloudflare's own published ranges
{ curl -fsS https://www.cloudflare.com/ips-v4; echo; curl -fsS https://www.cloudflare.com/ips-v6; } \
  | grep -E '^[0-9a-fA-F]' \
  | sed 's|^|set_real_ip_from |; s|$|;|' > /etc/nginx/conf.d/cloudflare-realip.conf
echo 'real_ip_header CF-Connecting-IP;' >> /etc/nginx/conf.d/cloudflare-realip.conf
nginx -t && systemctl reload nginx

Only list addresses you genuinely trust in set_real_ip_from. Anything in that list is allowed to claim to be any client it likes, so a range added carelessly hands an attacker a way to impersonate your office.

The allowlist that works because of one sentence in the nginx docs

Both nginx limiting modules carry the same line, and it is the whole mechanism. "Requests with an empty key value are not accounted." Building the allowlist that way is far more reliable than wrapping limits in conditional blocks.

# /etc/nginx/conf.d/ratelimit.conf

# 1 for addresses that must never be limited, 0 for everyone else
geo $never_limit {
    default 0;
    203.0.113.0/24  1;   # client head office
    198.51.100.7/32 1;   # payment provider callback source
}

# an empty key for the allowlisted, so nginx does not account them at all
map $never_limit $limit_key {
    0 $binary_remote_addr;
    1 "";
}

limit_req_zone $limit_key zone=login:10m rate=5r/m;
limit_conn_zone $limit_key zone=perip:10m;

geo reads the client address, which by this point is the address the real IP module already replaced, so the office is recognised by its own IP rather than by the CDN's. Then apply the zone where it belongs rather than globally, because a site-wide request limit is how you break your own product tour.

location = /customer/account/loginPost/ {
    limit_req zone=login burst=10 nodelay;
    limit_req_status 429;
    limit_req_log_level warn;
    proxy_pass http://app_backend;
}

The burst parameter allows "not more than the specified number of requests above the rate limit before they are rejected", and nodelay is what you want "If delaying of excessive requests while requests are being limited is not desired". limit_req_status defaults to 503, and 429 is both more honest about what happened and far easier to pick out of a log.

Run it in dry run before you run it for real

Both modules ship a dry run mode that accounts the excess without rejecting anything, and skipping this step is how the incident happens.

limit_req_dry_run on;

Leave it on across a full business week, including a month end if the business has one. Then read who would have been blocked.

grep "limiting requests" /var/log/nginx/error.log \
  | grep -oE 'client: [0-9a-fA-F.:]+' | sort | uniq -c | sort -rn | head -20

Every address near the top of that list is either an attacker or somebody you were about to break, and there is no way to tell which from the count alone. Resolve each one by name before the dry run comes off.

Ban the repeat offenders, with the same allowlist repeated

fail2ban already ships a filter for exactly this. The nginx-limit-req filter matches nginx's own limiting messages, so the jail stays short.

# /etc/fail2ban/jail.d/nginx-limit-req.local
[nginx-limit-req]
enabled  = true
port     = http,https
filter   = nginx-limit-req
logpath  = /var/log/nginx/error.log
findtime = 10m
maxretry = 20
bantime  = 1h
ignoreip = 127.0.0.1/8 ::1 203.0.113.0/24 198.51.100.7

The manual page describes ignoreip as a "list of IPs not to ban. They can include a DNS resp. CIDR mask too", and maxretry as the "number of failures that have to occur in the last findtime seconds to ban the IP". The shipped defaults are maxretry = 5, findtime = 10m and bantime = 10m, which are tuned for SSH rather than for a web request limiter, so raise maxretry here.

The allowlist has to exist in both places. nginx declining to limit an address does not stop fail2ban banning it for something else, and fail2ban ignoring an address does not stop nginx returning 429 to it.

For an attacker who keeps coming back, escalating bans cost nothing.

# /etc/fail2ban/jail.d/00-defaults.local
[DEFAULT]
bantime.increment = true
bantime.factor = 1
bantime.maxtime = 1w

Fail2ban documents bantime.increment as the option that "allows to use database for searching of previously banned ip's to increase a default ban time using special formula", and bantime.maxtime as "the max number of seconds using the ban time can reach (doesn't grow further)". The cap matters, because without it a persistent botnet address ends up banned for a length of time nobody intended.

Test the filter against your own log before you trust it

A filter that matches nothing is a jail that looks perfectly healthy and does nothing at all.

fail2ban-regex /var/log/nginx/error.log nginx-limit-req

Then confirm the jail is loaded, and keep the two recovery commands somewhere you can find them at three in the morning.

fail2ban-client status nginx-limit-req
fail2ban-client set nginx-limit-req unbanip 203.0.113.45
fail2ban-client set nginx-limit-req addignoreip 198.51.100.7

addignoreip takes effect immediately but is a runtime change, so add the address to the jail file as well once the phone call is over, otherwise the next restart re-creates the problem.

The addresses that must never be banned

Write this list before you need it, and keep it in the jail file and the geo block together so the two cannot drift apart.

• The client's office ranges, and any VPN or site-to-site tunnel they egress from
• Payment provider callback sources, and any other webhook sender whose retries you cannot afford to drop
• Your own monitoring and uptime checks, which look exactly like a bot from the log's point of view
• Load balancer and CDN health check sources
• The addresses your deploy pipeline connects from

A ban you can explain and undo in thirty seconds is worth more than a clever one you cannot.

Building rules that stop abuse without collateral damage is part of security and compliance, and keeping them current as traffic and providers change belongs to servers management. If the host itself is not hardened yet, start with our guide to hardening a fresh Ubuntu VPS.