How to Set Requests and Limits From Real Usage Instead of Guesses

Two failures show up in almost every cluster that has grown past its first year. Pods get OOM killed for memory they never legitimately needed, and a service is mysteriously slow at a fraction of the CPU it is apparently allowed to use. Both usually come from the same place, a resources block copied from a tutorial into fifty manifests.

Getting this right is worth real money on the node bill and real latency in the p99, and it takes an afternoon.

What the two numbers actually do

A request is a scheduling promise. The kube-scheduler uses it to decide which node a Pod goes on, so requests, not usage, are what fill up a cluster. On a contended node the CPU request also acts as a weighting, and the Kubernetes documentation puts it plainly, workloads with larger CPU requests are allocated more CPU time than workloads with small requests.

A limit is enforced by the kernel, and the enforcement differs by resource in a way that matters more than anything else in this article.

The CPU limit is a hard ceiling on CPU time. During each scheduling interval the kernel checks whether the limit has been exceeded, and if it has, it waits before letting that cgroup run again. Nothing dies, the work just takes longer.

The memory limit is enforced by killing the container. The docs are careful about the timing, saying the kernel may terminate a container that uses more than its memory limit, and that terminations only happen when the kernel detects memory pressure. Exceeding the memory request is a different risk, because if a container is over its request and the node runs short of memory overall, the Pod is likely to be evicted.

That asymmetry is the whole design. Too little CPU is slow, too little memory is dead.

Read what the workload actually uses

Start with the crude view, which needs metrics-server installed.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl top pod -n prod --containers --sort-by=memory

Do not size anything off that alone. The metrics-server project states outright that it is meant only for autoscaling purposes and should not be used as a source of monitoring metrics. It gives you a live reading, not a history, and sizing is a question about history.

For the real numbers, query the cAdvisor metrics that Prometheus already scrapes.

# peak working set per container over the last 7 days
max_over_time(container_memory_working_set_bytes{namespace="prod", container!=""}[7d])

# 95th percentile CPU usage per container over the last 7 days
quantile_over_time(0.95,
  rate(container_cpu_usage_seconds_total{namespace="prod", container!=""}[5m])[7d:5m]
)

container_memory_working_set_bytes is described by cAdvisor as the current working set, and it is the number to size memory against. container_memory_usage_bytes includes memory regardless of when it was accessed, so it reads higher and will talk you into buying page cache.

What throttling looks like

CPU throttling is visible, precisely, in two counters. container_cpu_cfs_periods_total counts elapsed enforcement period intervals and container_cpu_cfs_throttled_periods_total counts the throttled ones, so the ratio is the fraction of periods in which the kernel made the container wait.

sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{namespace="prod"}[5m]))
  /
sum by (pod) (rate(container_cpu_cfs_periods_total{namespace="prod"}[5m]))

The reason this surprises people is the enforcement window. The kubelet's --cpu-cfs-quota-period defaults to 100ms, so a container with a 500m limit gets 50ms of CPU time per 100ms window across all its threads. An eight thread runtime handling a burst can exhaust that in a few milliseconds of wall clock and then sit still for the rest of the window, while the dashboard shows average CPU usage well under the limit. Average usage is low, tail latency is terrible, and the throttling ratio is the only metric that shows it.

Arriving at numbers you can defend

Set the memory request from steady state usage and the memory limit from the observed peak plus headroom for a bad day, say 25 to 50 percent depending on how spiky the workload is. Then look at what that does to QoS.

A Pod is Guaranteed only when every container has a memory limit and request that are both greater than zero and equal, and a CPU limit and request that are both greater than zero and equal. Anything with a request but no matching limit is Burstable, and a Pod with no requests or limits at all is BestEffort. Under node pressure Kubernetes evicts BestEffort first, then Burstable, then Guaranteed, so QoS class is a real reliability lever for the workloads you care about most.

Set the CPU request from the 95th percentile of real usage rather than the peak. Bursts are what unused node capacity is for.

The CPU limit is the genuinely optional one. A container with no CPU limit can use idle CPU when the node is quiet and falls back to its request-weighted share when the node is busy, which is often exactly what you want for a latency sensitive service. Set a CPU limit when you need predictable behaviour for benchmarking or when one tenant must not be able to eat a shared node, and accept that you are trading throughput for predictability.

Memory limits are not optional in the same way. Without one, a leak takes the node's other pods with it.

Check the answer, then keep checking

The Vertical Pod Autoscaler will do the arithmetic continuously without touching your pods. In Off mode the autoscaler never changes Pod resources, but the recommender still writes recommendations into the object, which the docs describe as a dry run.

# k8s/vpa-checkout.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: checkout
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
      - containerName: '*'
        controlledResources: ["cpu", "memory"]

Leave it for a week and read it with kubectl describe vpa checkout-vpa. Compare its recommendation against your own numbers before you change anything.

When a pod does die, confirm the cause rather than assuming. kubectl describe pod shows Reason: OOMKilled with Exit Code: 137 for a container the kernel killed, which is a different problem from a container that exited on its own.

In-place resizing has also become practical. Pod resize reached stable in Kubernetes v1.35, so CPU and memory can be changed on a running Pod with kubectl patch --subresource=resize, subject to real constraints, including that the Pod's original QoS class cannot change.

Right-sizing is where a large share of cloud waste hides, and it is a standing item in our cloud cost optimization work and our ongoing Kubernetes management. If the next question is how these numbers interact with autoscaling, Kubernetes HPA Deep Dive picks up there.