Moving EKS to Production, and the Six Bills Nobody Budgets

The transition is the expensive part, not the technology

You can run an EKS cluster for weeks with one node group, one subnet, a single service, no persistent storage and no logging, and the line it leaves on the invoice will look like a rounding error. Nothing about that cluster is wrong. Almost none of the things that make it production-worthy have been switched on yet. Production means several availability zones, an environment per stage of the pipeline, more services, storage that survives a pod restart, and enough telemetry to answer questions at three in the morning. Each of those is a sound decision, and each one starts a meter that was not running the day before.

The control plane fee is per cluster, and you have more than one

AWS states it plainly on its pricing page. All Amazon EKS clusters have a per cluster per hour fee based on the cluster's Kubernetes version, and you pay separately for the AWS resources you use to run your applications on worker nodes. What catches people is the shape of the charge rather than its size. It attaches to the cluster and to time, not to what you run inside it, so an idle cluster kept warm for a demo bills the same as a busy one.

Now count your clusters. A team that separates development, staging and production is paying that fee three times, and a team that also gives each squad a sandbox is paying it rather more than three times. Very few people model it that way, because the cluster gets created once, by a pipeline, and then nobody looks at it again as a recurring cost with a multiplier in front of it.

The answer is not to collapse production and development into one cluster, since blast radius and IAM boundaries do not stop mattering because of a per-hour fee. What helps is knowing how many clusters exist and what each is for, treating short-lived review environments as namespaces in a shared non-production cluster rather than as clusters of their own, and deleting the experiment from last quarter. Run this across your regions and count what comes back.

for region in eu-west-1 eu-central-1 us-east-1; do
  aws eks list-clusters --region "$region" \
    --query "clusters[]" --output text \
    | tr '\t' '\n' | sed "s|^|$region  |"
done

Multi-AZ resilience makes your own service mesh a line item

Spreading pods across availability zones is the whole point of a resilient deployment. AWS is direct about the consequence in its own architecture guidance, where data transfer charges apply for cross-availability-zone communication between EC2 instances, and data transfer within the same availability zone is free. The moment your pods stop sharing a zone, ordinary internal chatter starts crossing a boundary that has a price on it.

The EKS best practices guide explains why this is the default rather than an edge case. On EKS, kube-proxy distributes traffic across all pods in the cluster regardless of their node or AZ placement, so a service with replicas in three zones will happily send a request from a pod in one zone to a pod in another, on every call. Nothing is misconfigured. A chatty pair of internal services generates metered traffic that scales with request volume, not with anything you would think to check on a capacity dashboard.

There are three levers worth knowing, all documented by AWS. Topology Aware Routing (the annotation service.kubernetes.io/topology-mode: Auto) asks the EndpointSlice controller to hint that endpoints should serve their own zone. The newer trafficDistribution field prefers same-zone endpoints and falls back to any healthy endpoint when none are available, and it is worth checking which spelling your cluster takes. PreferSameZone and PreferSameNode went GA in Kubernetes 1.35, and the original PreferClose, which means exactly the same thing as PreferSameZone, is deprecated in favour of it. On a cluster older than 1.33, PreferClose is the only one of the two that exists. The blunt instrument is internalTrafficPolicy: Local, which restricts traffic to node-local endpoints and, as AWS warns, drops traffic outright when there is no node-local endpoint to serve it.

apiVersion: v1
kind: Service
metadata:
  name: orders-service
spec:
  trafficDistribution: PreferSameZone
  selector:
    app: orders
  type: ClusterIP
  ports:
    - protocol: TCP
      port: 3003
      targetPort: 3003

One more from the same guide, because it is nearly free to fix. With the AWS Load Balancer Controller in instance mode, traffic lands on a NodePort and may take an extra hop to a pod in another zone, which is charged. In ip mode the load balancer proxies straight to the destination pod, and AWS states there are no data transfer charges in that approach.

Twelve services without an Ingress means twelve load balancers

When you create a Kubernetes Service of type LoadBalancer, a real Elastic Load Balancing resource is provisioned in your account. The EKS documentation is unambiguous about the constraint that follows, stating that you cannot share a Network Load Balancer across multiple services. There is no pooling and no cleverness. Twelve services of that type give you twelve load balancers.

Each one bills on its own clock. The Elastic Load Balancing pricing page describes a charge for each hour or partial hour that a load balancer is running, with each partial hour billed as a full hour, plus capacity units consumed per minute. The fleet cost follows how many you have provisioned rather than how much traffic they carry, and a load balancer fronting a service nobody calls is a full-price load balancer.

For HTTP and HTTPS traffic there is a documented alternative. A Kubernetes Ingress provisions an Application Load Balancer, and the alb.ingress.kubernetes.io/group.name annotation puts several Ingress resources into one IngressGroup that the controller merges behind a single ALB. Read the security warning in the AWS documentation before you reach for it, since anyone with RBAC permission to create or modify an Ingress can join your group and overwrite rules with higher-priority ones. Use it inside a trust boundary, not across tenants.

# Every Service that provisions its own load balancer
kubectl get svc -A --field-selector spec.type=LoadBalancer \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,LB:.status.loadBalancer.ingress[0].hostname'

# What actually exists in the account, to compare against the list above
aws elbv2 describe-load-balancers \
  --query 'LoadBalancers[].[LoadBalancerName,Type,Scheme,CreatedTime]' --output table

EBS volumes outlive the workloads that asked for them

The EBS CSI driver manages the lifecycle of the EBS volumes backing your persistent volumes, and the end of that lifecycle is decided by the reclaim policy. In the Kubernetes documentation, a PersistentVolume created dynamically by a StorageClass takes the reclaimPolicy from that class, either Delete or Retain, and a StorageClass created without the field defaults to Delete. The trap is that Retain is exactly what a careful platform engineer picks for anything holding real data, and a retained volume is never cleaned up by the cluster. It sits in the account, unattached, until a person deletes it.

StatefulSets add the second half of the problem. The Kubernetes documentation states that deleting or scaling down a StatefulSet will not delete the volumes associated with it, and that claims created from volumeClaimTemplates are retained until manually deleted unless you set a retention policy. That is a deliberate data-safety choice and a good one. It also means every scaled-down replica and every abandoned proof of concept can leave a full-sized volume behind. AWS flagged this in the EKS release notes for Kubernetes 1.32, where the retention policy is described as helping prevent orphaned PVCs in your cluster.

Billing does not care that nothing is attached. The EBS pricing page charges volume storage by the amount you provision per month until you release the storage, so an unattached volume is a fully priced one. AWS suggests the cost optimization category of Trusted Advisor for spotting them, and the CSI driver tags what it creates if you would rather work from the command line.

# Unattached volumes created by the EBS CSI driver, oldest first
aws ec2 describe-volumes \
  --filters Name=status,Values=available Name=tag-key,Values=CSIVolumeName \
  --query 'sort_by(Volumes,&CreateTime)[].{ID:VolumeId,GiB:Size,Type:VolumeType,Created:CreateTime}' \
  --output table

# Which of your storage classes retain, and which delete
kubectl get storageclass \
  -o custom-columns='NAME:.metadata.name,PROVISIONER:.provisioner,RECLAIM:.reclaimPolicy'

Observability gets switched on during hardening and never switched off

Control plane logging is off until you ask for it. By default, cluster control plane logs are not sent to CloudWatch Logs, and you enable each type individually from api, audit, authenticator, controllerManager and scheduler. Production hardening is exactly when somebody enables all five, usually with one CLI call and for a good reason. AWS is explicit that you are then charged the standard CloudWatch Logs data ingestion and storage costs, and that enabled log types are sent at verbosity level 2.

The audit log is the loud one, because it records every action taken against the API server, including everything your controllers and operators do on a loop. AWS's own cost guidance suggests enabling specific types selectively in non-production clusters and turning them off after the analysis, while keeping the full set in production where you cannot reproduce the event you are investigating. It also notes that EKS logs are classified as Vended Logs in CloudWatch, which have their own pricing treatment.

Retention compounds. CloudWatch's default retention policy is to keep logs indefinitely and never expire, so a log group created during a hardening sprint accrues storage for as long as the account exists unless somebody sets a period. It is configurable per log group from one day to ten years, and streaming older logs to S3 through a subscription filter is the documented way to keep them without keeping them in CloudWatch.

Container Insights deserves its own look. It automatically creates a log group for its performance log events, and the billing model depends on which version you run. With the original version, metrics collected and logs ingested are charged as custom metrics; with enhanced observability for EKS, they are charged per observation instead. Neither is a bad deal for what it gives you, and both are worth understanding before you enable it on every cluster in the account.

# Which control plane log types are enabled
aws eks describe-cluster --name my-cluster \
  --query 'cluster.logging.clusterLogging' --output json

# Retention per log group. A None in the second column means it never expires
aws logs describe-log-groups \
  --query 'logGroups[].[logGroupName,retentionInDays]' --output table

Standing still on an old Kubernetes version bills by default

This one is worth knowing because it needs no action from you at all. A Kubernetes minor version gets 14 months of standard support on EKS, then 12 months of extended support at an additional cost per cluster hour. Extended support is enabled by default, for new clusters and existing ones, through an upgrade policy whose supportType is EXTENDED unless you specified otherwise. Billing starts at the beginning of the day the version reaches end of standard support, and once a cluster has entered extended support you cannot disable it. To change the setting, the cluster has to be back on a version in standard support.

The alternative is the STANDARD policy, under which AWS automatically upgrades the cluster at the end of standard support instead of charging for extended support. Neither setting is right for everyone, but one of them is chosen for you if you never look, and the one chosen for you is the one that bills. At the end of extended support the control plane is auto-upgraded regardless, and AWS states this can happen at any time after that date without notice, while managed and self-managed node groups stay where they are.

aws eks describe-cluster --name my-cluster \
  --query 'cluster.upgradePolicy.supportType' --output text

What to look at first

Run the checks above against production first, since that is where the multipliers live. Counting clusters and load balancers pays back fastest because those are inventory questions with unambiguous answers. Unattached volumes and log group retention come next. Keeping traffic inside a zone is last, being real architectural work that needs measurement rather than a checklist, and AWS publishes guidance on getting per-pod visibility into cross-AZ bytes before you change any routing behaviour.

None of these are reasons to run a less resilient cluster. Multi-AZ, separate environments, retained volumes and audit logging are all things you want. They are just the things that were free on the cluster you built to learn on, and are not free on the cluster that serves customers. Whether you should be on Kubernetes at all is the other half of this question, and we have argued that most teams running EKS would have been fine on ECS. Once the answer is yes, working the list above is ordinary cloud cost optimization.