How to Drain a Node Without Dropping a Single Request

Draining a node looks like a solved problem. You run one command, Kubernetes evicts the pods politely, the scheduler puts them somewhere else. Then someone checks the logs and finds a spike of connection resets at exactly the minute you drained.

The cause is a race, and once you see it you cannot unsee it.

The race

When a pod is deleted, two things start at the same moment and neither waits for the other.

The kubelet begins termination: it sends SIGTERM to your container. Separately, the endpoints controller notices the pod is going away and starts removing it from the Service endpoints, which then has to propagate to every kube-proxy or ingress controller in the cluster.

The first path is local and fast. The second is distributed and slower. So for a short window your application has already been told to shut down while load balancers are still confidently sending it traffic.

Your application does the correct thing, stops accepting connections, and those in-flight requests become errors.

The fix, in three parts

1. Wait before you shut down.

lifecycle:
  preStop:
    exec:
      command: ["sleep", "10"]

The preStop hook runs to completion before SIGTERM is sent. Sleeping here does nothing except give the endpoint removal time to reach every proxy. Ten seconds is a common starting point; on a large cluster it may need more.

This looks crude. It is also the recommended answer, because there is no event your pod can subscribe to that says "every load balancer has forgotten about me".

2. Give the shutdown enough time.

terminationGracePeriodSeconds: 45

The grace period is the total budget from deletion to SIGKILL, and the preStop hook is spent from it. A 10 second sleep out of a 30 second default leaves 20 seconds for actual draining. If your slowest legitimate request takes 30 seconds, the maths does not work and Kubernetes will kill the container mid-request.

Set it to preStop plus your realistic worst-case request, plus margin.

3. Actually handle SIGTERM.

The grace period only helps if your application uses it. Stop accepting new connections, finish the ones in flight, close the database pool, then exit. An application that ignores SIGTERM gets killed at the end of the grace period no matter how generous the number is.

While you are there, make the readiness probe fail as soon as shutdown begins. Readiness controls endpoint membership, so failing it is the fastest signal you can send.

Then stop the drain taking your last replica

A disruption budget tells the eviction API how much unavailability the application can survive.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

With this in place, kubectl drain blocks rather than evicting the pod that would take you below two. That is the intended behaviour: the drain waits for a replacement elsewhere to become ready.

Two traps come with it.

minAvailable: 1 on a single-replica deployment deadlocks the drain. There is no second pod to become available, so the eviction can never be allowed. Either run two replicas or accept the disruption for that workload.

A budget the cluster can never satisfy blocks maintenance forever. maxUnavailable: 0 reads like caution and behaves like a permanent lock on node upgrades.

Drain, and read what it tells you

kubectl drain node-3 --ignore-daemonsets --delete-emptydir-data --timeout=300s

--ignore-daemonsets is needed because DaemonSet pods are recreated on the same node by design and would otherwise stop the drain. --delete-emptydir-data acknowledges that anything in an emptyDir volume on that node is going away, which is worth reading twice if a workload keeps state there.

If the command sits and waits, it is usually a disruption budget doing its job, and the answer is patience or another replica rather than --force.

Prove it rather than assume it

Put a load generator against the service, drain a node, and count non-200 responses.

hey -z 120s -c 20 https://app.example.com/healthz

If the number is zero, the configuration is right. If it is not, the usual culprit is a preStop delay shorter than your endpoint propagation, or an application that treats SIGTERM as a reason to exit immediately.

Rolling nodes without users noticing is one of the things our Kubernetes management work exists to make routine.