kubectl drain politely evicts your pods and your users still see errors, because a pod that is terminating is not automatically a pod that has stopped receiving traffic. The gap between those two states is where the dropped requests live. This is how to close it with a pre-stop delay, a grace period long enough for real requests, and a disruption budget that stops the node from taking your last replica with it.
Draining a node looks like a solved problem. You run one command, Kubernetes evicts the pods politely, the scheduler puts them somewhere else. Then someone checks the logs and finds a spike of connection resets at exactly the minute you drained.
The cause is a race, and once you see it you cannot unsee it.
The race
When a pod is deleted, two things start at the same moment and neither waits for the other.
The kubelet begins termination: it sends SIGTERM to your container. Separately, the endpoints controller notices the pod is going away and starts removing it from the Service endpoints, which then has to propagate to every kube-proxy or ingress controller in the cluster.
The first path is local and fast. The second is distributed and slower. So for a short window your application has already been told to shut down while load balancers are still confidently sending it traffic.
Your application does the correct thing, stops accepting connections, and those in-flight requests become errors.
The fix, in three parts
1. Wait before you shut down.
lifecycle:
preStop:
exec:
command: ["sleep", "10"]
The preStop hook runs to completion before SIGTERM is sent. Sleeping here does nothing except give the endpoint removal time to reach every proxy. Ten seconds is a common starting point; on a large cluster it may need more.
This looks crude. It is also the recommended answer, because there is no event your pod can subscribe to that says "every load balancer has forgotten about me".
2. Give the shutdown enough time.
terminationGracePeriodSeconds: 45
The grace period is the total budget from deletion to SIGKILL, and the preStop hook is spent from it. A 10 second sleep out of a 30 second default leaves 20 seconds for actual draining. If your slowest legitimate request takes 30 seconds, the maths does not work and Kubernetes will kill the container mid-request.
Set it to preStop plus your realistic worst-case request, plus margin.
3. Actually handle SIGTERM.
The grace period only helps if your application uses it. Stop accepting new connections, finish the ones in flight, close the database pool, then exit. An application that ignores SIGTERM gets killed at the end of the grace period no matter how generous the number is.
While you are there, make the readiness probe fail as soon as shutdown begins. Readiness controls endpoint membership, so failing it is the fastest signal you can send.
Then stop the drain taking your last replica
A disruption budget tells the eviction API how much unavailability the application can survive.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api
spec:
minAvailable: 2
selector:
matchLabels:
app: api
With this in place, kubectl drain blocks rather than evicting the pod that would take you below two. That is the intended behaviour: the drain waits for a replacement elsewhere to become ready.
Two traps come with it.
minAvailable: 1 on a single-replica deployment deadlocks the drain. There is no second pod to become available, so the eviction can never be allowed. Either run two replicas or accept the disruption for that workload.
A budget the cluster can never satisfy blocks maintenance forever. maxUnavailable: 0 reads like caution and behaves like a permanent lock on node upgrades.
Drain, and read what it tells you
kubectl drain node-3 --ignore-daemonsets --delete-emptydir-data --timeout=300s
--ignore-daemonsets is needed because DaemonSet pods are recreated on the same node by design and would otherwise stop the drain. --delete-emptydir-data acknowledges that anything in an emptyDir volume on that node is going away, which is worth reading twice if a workload keeps state there.
If the command sits and waits, it is usually a disruption budget doing its job, and the answer is patience or another replica rather than --force.
Prove it rather than assume it
Put a load generator against the service, drain a node, and count non-200 responses.
hey -z 120s -c 20 https://app.example.com/healthz
If the number is zero, the configuration is right. If it is not, the usual culprit is a preStop delay shorter than your endpoint propagation, or an application that treats SIGTERM as a reason to exit immediately.
Rolling nodes without users noticing is one of the things our Kubernetes management work exists to make routine.
Frequently asked questions
Why does kubectl drain cause dropped requests when pods are evicted gracefully?
Pod deletion starts two processes at once and neither waits for the other. The kubelet sends SIGTERM to the container, which is local and fast, while the endpoints controller removes the pod from the Service endpoints, a change that has to propagate to every kube-proxy and ingress controller. For a short window the application is shutting down while load balancers still send it traffic, and those in-flight requests become errors.
Why add a preStop sleep hook to a Kubernetes pod?
A preStop hook runs to completion before SIGTERM is sent, so a preStop exec that runs sleep 10 gives endpoint removal time to reach every proxy before the application starts shutting down. Ten seconds is a common starting point, and a large cluster may need more. The approach looks crude, yet it is the recommended answer, because no event exists that tells a pod every load balancer has forgotten about it.
What should terminationGracePeriodSeconds be set to in Kubernetes?
Set terminationGracePeriodSeconds to the preStop delay plus the realistic worst-case request, plus a margin. The grace period is the whole budget from deletion to SIGKILL, and the preStop hook is spent from it, so a 10 second sleep inside the 30 second default leaves only 20 seconds for draining. A request that takes 30 seconds would then be killed partway through.
How should an application handle SIGTERM during a Kubernetes node drain?
Stop accepting new connections, finish the requests in flight, close the database pool, then exit. An application that ignores SIGTERM is killed at the end of the grace period no matter how generous the number is. Make the readiness probe fail as soon as shutdown begins too, since readiness controls endpoint membership and failing it is the fastest signal a pod can send.
How does a PodDisruptionBudget affect kubectl drain?
A PodDisruptionBudget tells the eviction API how much unavailability an application can survive. With minAvailable set to 2, kubectl drain refuses to evict the pod that would take the application below two and waits for a replacement elsewhere to become ready. A drain that sits waiting is usually the budget doing its job, and the answer is patience or another replica instead of --force.
Why is kubectl drain stuck forever on a node?
A drain blocks indefinitely when its PodDisruptionBudget can never be satisfied. Setting minAvailable to 1 on a single-replica deployment deadlocks it, because there is no second pod to become available, so the eviction is never allowed. maxUnavailable set to 0 reads like caution and behaves like a permanent lock on node upgrades. Run two replicas, or accept the disruption for that workload.
What do --ignore-daemonsets and --delete-emptydir-data do in kubectl drain?
The --ignore-daemonsets flag is needed because DaemonSet pods are recreated on the same node by design and would otherwise stop the drain. The --delete-emptydir-data flag acknowledges that anything in an emptyDir volume on that node is going away, which deserves a second look if a workload keeps state there. A full command looks like kubectl drain node-3 --ignore-daemonsets --delete-emptydir-data --timeout=300s.
How do I test that draining a Kubernetes node drops no requests?
Put a load generator against the service, such as hey -z 120s -c 20 pointed at a health endpoint, drain a node while it runs, and count the non-200 responses. Zero means the configuration is right. Anything else usually comes from a preStop delay shorter than endpoint propagation, or from an application that treats SIGTERM as a reason to exit immediately.
Read how we handle it in Cloud & Server Management.
Related Articles
How to Run Ephemeral CI Runners on Your Own Hardware
A build that passes because of something left behind by the previous build is not a passing build, it is a coincidence. Ephemeral runners remove that class of problem by giving every job a machine that has never run anything else. GitHub supports this directly through single-use runner registration and just-in-time configuration, so the runner deregisters itself after one job and your automation disposes of the host. This guide covers both approaches, the Kubernetes version, and the one situation where self-hosted runners are the wrong answer.
Server & DevOpsHow to Do Canary Releases Without a Service Mesh
You can send five percent of production traffic at a new version, watch the error rate, and roll back in seconds without installing a service mesh. This walks through replica weighted canaries with a progressive delivery controller, real percentage splitting at the edge, an automated pass or fail check against Prometheus, and the rollback path. It also covers what changed when Kubernetes retired Ingress NGINX in March 2026.
Server & DevOpsZero-Downtime Deployments with K3s and ArgoCD - A Practical Guide
A hands-on guide to achieving zero-downtime deployments using K3s and ArgoCD, covering GitOps workflows, rolling update strategies, health checks, and complete YAML manifests.