Skip to main content

Kubernetes HPA Deep Dive: Scaling Under Load

8 min read
Tune my Kubernetes autoscaling

Your autoscaler gets custom metrics, scaling policies and stabilization windows tuned to real load.

Configure Kubernetes Horizontal Pod Autoscaler with custom metrics, scaling policies, and behavior tuning for predictable production auto-scaling.

Beyond Basic CPU Autoscaling

The Horizontal Pod Autoscaler (HPA) is one of Kubernetes' most powerful features, yet most deployments use only the simplest configuration - scale when CPU exceeds 50 percent. In production, CPU-based scaling often reacts too slowly or too aggressively, leading to either request queuing or resource waste.

This guide covers advanced HPA configuration including custom metrics, scaling behaviors, and stabilization windows that we implement as part of our Kubernetes management practice.

How HPA Works Internally

The HPA controller runs a control loop every 15 seconds (configurable via --horizontal-pod-autoscaler-sync-period). It queries the metrics API, calculates the desired replica count using the formula:

desiredReplicas = ceil(currentReplicas * (currentMetricValue / desiredMetricValue))

The controller then applies scaling policies, stabilization windows, and tolerance thresholds before issuing a scale command.

The Tolerance Band

By default, HPA has a 10 percent tolerance (--horizontal-pod-autoscaler-tolerance=0.1). If the ratio of current to desired metric value falls within 0.9 to 1.1, no scaling action occurs. This prevents oscillation but can delay response to genuine load changes.

Multi-Metric HPA Configuration

Production services should scale based on multiple signals:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-api
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 75
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "100"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60
        - type: Pods
          value: 5
          periodSeconds: 60
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 10
          periodSeconds: 120
      selectPolicy: Min

Understanding the Behavior Section

Scale-up uses a 30-second stabilization window and two policies: add 50 percent of current replicas or 5 pods, whichever is greater (selectPolicy: Max). This allows aggressive scale-up during traffic spikes.

Scale-down uses a 300-second (5-minute) stabilization window and a conservative policy: remove at most 10 percent of replicas every 2 minutes (selectPolicy: Min). This prevents the cluster from scaling down too quickly after a traffic spike, which would cause problems if traffic returns.

Custom Metrics with Prometheus

The built-in CPU and memory metrics are limited. Real application health is better measured by request latency, queue depth, or error rates.

Setting Up the Metrics Pipeline

Install the Prometheus Adapter to expose custom metrics to the HPA:

# prometheus-adapter-config.yaml
rules:
  - seriesQuery: 'http_request_duration_seconds_count{namespace!="",pod!=""}'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        pod: {resource: "pod"}
    name:
      matches: "^(.*)_count$"
      as: "http_requests_per_second"
    metricsQuery: 'rate(<<.Series>>{<<.LabelMatchers>>}[2m])'

  - seriesQuery: 'rabbitmq_queue_messages{namespace!=""}'
    resources:
      overrides:
        namespace: {resource: "namespace"}
    name:
      as: "queue_depth"
    metricsQuery: 'avg(<<.Series>>{<<.LabelMatchers>>})'

Verifying Custom Metrics

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/production/pods/*/http_requests_per_second" | jq .

Scaling Based on Queue Depth

For worker deployments that process jobs from a message queue, scale based on queue depth rather than CPU:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: worker-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 2
  maxReplicas: 30
  metrics:
    - type: Object
      object:
        describedObject:
          apiVersion: v1
          kind: Service
          name: rabbitmq
        metric:
          name: queue_depth
        target:
          type: Value
          value: "50"

This configuration maintains roughly 50 messages per worker pod. As the queue grows, new workers spawn. As it drains, workers scale down.

Scaling Pitfalls to Avoid

Cold Start Penalty

If your application takes 30 seconds to start, newly scaled pods cannot serve traffic during that window. Combine HPA with proper readiness probes and consider overprovisioning by setting minReplicas high enough to handle average load without scaling.

Resource Requests and Limits

HPA CPU percentage is calculated against resources.requests, not limits. If requests are set too low, the HPA sees artificially high utilization and over-scales:

resources:
  requests:
    cpu: 250m
    memory: 512Mi
  limits:
    cpu: "1"
    memory: 1Gi

Set requests to match your P50 usage and limits to match P99 usage.

Cluster Autoscaler Interaction

HPA adds pods, but those pods need nodes to run on. Ensure the Cluster Autoscaler is configured with matching instance types and can provision nodes faster than your scale-up rate. A node that takes 5 minutes to provision defeats the purpose of a 30-second scaling reaction.

Monitoring HPA Performance

Track these metrics to verify HPA effectiveness:

  • kube_hpa_status_current_replicas vs kube_hpa_spec_max_replicas - if current regularly hits max, increase the ceiling
  • kube_hpa_status_condition{condition="ScalingLimited"} - indicates HPA wants to scale but cannot
  • Request latency P99 during scale-up events - reveals whether scaling is fast enough

Use these metrics to iteratively tune stabilization windows and scaling policies for your specific workload patterns. For hands-on help configuring auto-scaling, explore our server optimization services.

Frequently asked questions

How often does the Kubernetes HPA recalculate the replica count?

The HPA controller runs its control loop every 15 seconds, configurable with --horizontal-pod-autoscaler-sync-period. Each pass computes desiredReplicas = ceil(currentReplicas * (currentMetricValue / desiredMetricValue)), then applies scaling policies, stabilization windows and the tolerance band before issuing a scale command.

Why doesn't the Kubernetes HPA react to small changes in load?

Small deviations fall inside the HPA's default 10 percent tolerance, set with --horizontal-pod-autoscaler-tolerance=0.1. When the ratio of the current to the desired metric value sits between 0.9 and 1.1, no scaling happens. The band prevents oscillation, though it can delay the response to genuine load changes.

How do I make the Kubernetes HPA scale up fast and scale down slowly?

Tune the behavior section of an autoscaling/v2 HPA. For scale-up, a 30-second stabilization window with two policies, 50 percent of current replicas or 5 pods per 60 seconds, and selectPolicy Max adds whichever is greater. For scale-down, a 300-second window with at most 10 percent of replicas removed every 120 seconds and selectPolicy Min keeps the cluster from shrinking too fast after a traffic spike.

How do I autoscale Kubernetes pods on requests per second with Prometheus?

Install the Prometheus Adapter and add a rule that turns http_request_duration_seconds_count into a per-pod http_requests_per_second metric using rate over 2m. Reference it in the HPA as a Pods metric with an AverageValue target, for example 100. Confirm the metric is exposed with kubectl get --raw on /apis/custom.metrics.k8s.io/v1beta1/namespaces/production/pods/*/http_requests_per_second piped to jq.

How do I scale Kubernetes queue workers on RabbitMQ queue depth?

Expose rabbitmq_queue_messages through the Prometheus Adapter as a queue_depth metric, then point the worker HPA at it with an Object metric whose describedObject is the rabbitmq Service. A target value of 50 keeps roughly 50 messages per worker pod, with the worker Deployment ranging from 2 to 30 replicas. Workers spawn as the queue grows and scale down as it drains, which fits job processors better than CPU-based scaling.

Does the Kubernetes HPA measure CPU utilization against requests or limits?

Against requests. HPA CPU percentage is calculated from resources.requests, so requests set too low make utilization look artificially high and the HPA over-scales. Set requests to match P50 usage and limits to match P99, for example requests of 250m CPU and 512Mi memory with limits of 1 CPU and 1Gi.

Why are new pods slow to take traffic when the Kubernetes HPA scales up?

Two delays sit outside the HPA. An application that takes 30 seconds to start cannot serve traffic during that window, so pair the HPA with proper readiness probes and set minReplicas high enough to handle average load without scaling. New pods also need nodes, and a node that takes 5 minutes to provision defeats a 30-second scaling reaction, so configure the Cluster Autoscaler with matching instance types.

How can I tell if the Kubernetes HPA is hitting its limits?

Compare kube_hpa_status_current_replicas with kube_hpa_spec_max_replicas, and raise the ceiling if current replicas regularly reach the maximum. The kube_hpa_status_condition metric with condition ScalingLimited shows when the HPA wants to scale and cannot. P99 request latency during scale-up events reveals whether scaling is fast enough.

Read how we handle it in Cloud & Server Management.