How to Run Multi AZ So It Actually Survives an AZ Failure

Most AWS accounts are already multi AZ on paper. Subnets in three zones, an Auto Scaling group spanning them, a database with Multi-AZ ticked. Then a zone has a bad day and the site is down anyway. Spreading a deployment across zones and keeping it serving when one disappears are two different properties, and the second costs more thought than the first.

Capacity is the part people skip

An Auto Scaling group across two zones running near full utilization has no spare capacity. When one zone goes, half your instances go with it, and the replacements have to launch into a Region where a lot of other customers are asking for the same instance types at the same moment. The design that survives is statically stable, which means it already holds the capacity it needs in the remaining zones before the failure, so recovery does not depend on a launch succeeding. Three zones is the cheaper shape for that, since losing one of three costs a third of the fleet rather than half.

What the managed services actually do

Failover behaviour is where the surprises live, and it differs per service.

An RDS Multi-AZ DB instance keeps a synchronous standby replica in another Availability Zone. That standby is not readable, and it exists only to take over. AWS states that failover times are typically 60 to 120 seconds, and that large transactions or a lengthy recovery process can increase that. The mechanism is DNS. The failover automatically changes the DNS record of the DB instance to point to the standby, and as a result you need to re-establish any existing connections, so every open connection in your pool breaks. If your application runs on a JVM, AWS specifically recommends configuring the JVM with a DNS TTL of no more than 60 seconds, because some Java configurations cache a name lookup until the process restarts.

An RDS Multi-AZ DB cluster is a different product that shares part of the name. It runs a writer and two reader DB instances in three separate Availability Zones, replication is semisynchronous and requires acknowledgment from at least one reader for a change to commit, and the readers serve read traffic as well as acting as automatic failover targets. Failover time depends on replica lag, because the reader has to apply its unapplied transactions before promotion, which is why ReplicaLag is a metric to alarm on rather than a curiosity.

ElastiCache with Multi-AZ enabled promotes the replica with the least replication lag, and AWS describes the promotion as typically just a few seconds. It propagates the DNS name of the promoted replica, so an application writing to the primary endpoint needs no endpoint change. Two details matter more than the timing. Replication is asynchronous, so a small amount of data can be lost to replication lag. And a customer-initiated reboot of a primary does not trigger automatic failover, which means rebooting the primary is not a failover test.

Recovery has its own catch. If an entire Availability Zone fails, the replacement replica for the failed primary is created only when that Availability Zone is back up and available. You run with reduced redundancy until the zone returns, not until AWS notices the failure.

The single points that quietly remain

A healthy load balancer does not imply a healthy zone. By default a target group is considered healthy as long as it has at least one healthy target, which is nowhere near enough for a large fleet. Set explicit thresholds so a degraded zone is taken out of DNS instead of being kept in on the strength of one surviving instance.

aws elbv2 modify-target-group-attributes \
  --target-group-arn TARGET_GROUP_ARN \
  --attributes \
    Key=target_group_health.dns_failover.minimum_healthy_targets.percentage,Value=50 \
    Key=target_group_health.unhealthy_state_routing.minimum_healthy_targets.percentage,Value=50

Even with that set, DNS failover removes the unhealthy zone's addresses from the load balancer hostname while the local client DNS cache can still hold them until the record TTL expires, which is 60 seconds.

The NAT gateway is the classic hidden single point. Each NAT gateway is created in a specific Availability Zone and implemented with redundancy in that zone only. AWS puts it plainly. If you have resources in multiple Availability Zones sharing one NAT gateway and that gateway's Availability Zone is down, resources in the other zones lose internet access. The fix is a NAT gateway in each zone with routing that keeps traffic in its own zone, which most Terraform modules do by default and most hand-built VPCs do not.

There is now a second option. A regional NAT gateway detects a network interface in a new Availability Zone and automatically expands to that zone, contracting again from zones with no active workloads, so one NAT ID serves every zone in your route tables. Expansion into a new zone can take up to 60 minutes after a resource is instantiated there, and until it completes that traffic is processed across zones. Regional NAT gateways do not offer private connectivity, so private NAT use cases stay on zonal gateways.

Everything else that is zone-scoped deserves the same look. An EBS volume lives in one Availability Zone and so does the instance attached to it, and anything you run yourself on a single instance, a message broker, a scheduler, a cache with no replica, is a zonal dependency no matter how the subnets around it are arranged.

Rehearse it before the zone does

Each of these can be forced on purpose, which is the only way to learn whether your connection pool, your retries and your alarms actually cope. Both of the commands below cause a real interruption, so run them in a lower environment first and in a maintenance window after that.

# forces an RDS Multi-AZ failover on the DB instance
aws rds reboot-db-instance \
  --db-instance-identifier my-db \
  --force-failover

# forces an ElastiCache failover on one shard
aws elasticache test-failover \
  --replication-group-id my-cluster \
  --node-group-id 0001

For the zone itself, Amazon Application Recovery Controller offers a zonal shift, which moves traffic for a supported resource away from one Availability Zone to the healthy zones in the same Region. Supported resources are Amazon EC2 Auto Scaling groups, Amazon EKS, Application Load Balancers and Network Load Balancers, with cross-zone load balancing either enabled or disabled. Every zonal shift is temporary. You set an initial expiration from one minute up to three days and can extend it if you need to.

aws arc-zonal-shift start-zonal-shift \
  --resource-identifier LOAD_BALANCER_ARN \
  --away-from use1-az2 \
  --expires-in 30m \
  --comment "zonal failure rehearsal"

Two conditions come with it. AWS is explicit that you prescale your application before starting a shift, since the shift moves traffic and does not create capacity. And when a load balancer is in a fail open state a zonal shift has no effect, because it cannot force a zone to be unhealthy while the load balancer is already failing open.

The rehearsal is the whole point. A multi AZ design nobody has ever failed over is the same category of artefact as a backup nobody has ever restored, which is the kind of gap SRE services exist to close, and closing it usually starts with an honest look in architecture and planning.