Skip to main content

Disaster Recovery Plans for Cloud Infrastructure

9 min read
Plan my disaster recovery

We set your RPO and RTO, build multi-region failover and write the runbooks to recover.

Design and implement disaster recovery strategies for cloud infrastructure with RPO/RTO planning, multi-region failover, and automated recovery runbooks.

Why DR Planning Cannot Wait

Every organization believes a major outage will not happen to them - until it does. AWS region outages, database corruption, ransomware attacks, and human error can take down production systems for hours or days. The cost of downtime varies by industry, but for most SaaS companies, a one-hour outage costs between $10,000 and $500,000 in lost revenue, SLA penalties, and customer trust.

Disaster recovery planning is a core part of architecture planning that transforms reactive panic into structured recovery procedures with predictable outcomes.

Understanding RPO and RTO

Two metrics define every DR strategy:

Recovery Point Objective (RPO): The maximum acceptable data loss measured in time. An RPO of 1 hour means you can lose at most 1 hour of data.

Recovery Time Objective (RTO): The maximum acceptable downtime. An RTO of 4 hours means your systems must be operational within 4 hours of an incident.

DR Tier Framework

TierRPORTOStrategyMonthly Cost (Example)
Tier 1Near-zero< 15 minMulti-region active-active$15,000+
Tier 2< 1 hour< 1 hourWarm standby$5,000-10,000
Tier 3< 4 hours< 4 hoursPilot light$1,000-3,000
Tier 4< 24 hours< 24 hoursBackup and restore$200-500

Most organizations need different tiers for different systems. Customer-facing APIs might need Tier 1, while internal reporting can tolerate Tier 4.

Backup Strategy

Database Backups

#!/bin/bash
# Automated database backup script
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
DB_NAME="production"
BACKUP_DIR="/backups/mysql"
S3_BUCKET="s3://company-backups/mysql"

# Full backup with point-in-time recovery support
mysqldump --single-transaction --routines --triggers \
  --master-data=2 --flush-logs \
  -u backup_user -p"$DB_PASS" "$DB_NAME" | \
  gzip > "${BACKUP_DIR}/${DB_NAME}_${TIMESTAMP}.sql.gz"

# Upload to S3 with server-side encryption
aws s3 cp "${BACKUP_DIR}/${DB_NAME}_${TIMESTAMP}.sql.gz" \
  "${S3_BUCKET}/${DB_NAME}_${TIMESTAMP}.sql.gz" \
  --storage-class STANDARD_IA \
  --sse aws:kms

# Cross-region replication handles copying to DR region
# Verify backup integrity
gunzip -t "${BACKUP_DIR}/${DB_NAME}_${TIMESTAMP}.sql.gz"
echo "Backup completed: ${DB_NAME}_${TIMESTAMP}.sql.gz"

# Cleanup local backups older than 7 days
find "$BACKUP_DIR" -name "*.sql.gz" -mtime +7 -delete

Backup Verification

A backup nobody has restored is an assumption. Automate a restore test. The testing schedule below sets monthly as the minimum, and once the script exists, running it weekly costs nothing extra:

#!/bin/bash
# Automated backup restore verification (monthly at minimum)
LATEST_BACKUP=$(aws s3 ls s3://company-backups/mysql/ | sort | tail -1 | awk '{print $4}')

# Download to restore test server
aws s3 cp "s3://company-backups/mysql/${LATEST_BACKUP}" /tmp/

# Restore to test database
gunzip -c "/tmp/${LATEST_BACKUP}" | mysql -u root -p"$TEST_DB_PASS" test_restore

# Run integrity checks
EXPECTED_USERS=$(mysql -u root -p"$PROD_DB_PASS" -h prod-db -e "SELECT COUNT(*) FROM users" -sN)
RESTORED_USERS=$(mysql -u root -p"$TEST_DB_PASS" -e "SELECT COUNT(*) FROM test_restore.users" -sN)

if [ "$EXPECTED_USERS" -eq "$RESTORED_USERS" ]; then
  echo "PASS: Backup integrity verified ($RESTORED_USERS users)"
else
  echo "FAIL: Count mismatch (expected $EXPECTED_USERS, got $RESTORED_USERS)"
  # Send alert
fi

# Cleanup
mysql -u root -p"$TEST_DB_PASS" -e "DROP DATABASE test_restore"
rm -f "/tmp/${LATEST_BACKUP}"

Multi-Region Architecture

Warm Standby with AWS

# Terraform configuration for DR region
resource "aws_rds_cluster" "dr" {
  provider = aws.dr_region

  cluster_identifier = "production-dr"
  engine            = "aurora-mysql"
  engine_version    = "8.0.mysql_aurora.3.05.2"

  # Read replica of production cluster
  replication_source_identifier = aws_rds_cluster.production.arn

  # Smaller instance in DR (scale up during failover)
  instance_class = "db.r6g.large"

  tags = {
    Purpose = "disaster-recovery"
  }
}

resource "aws_autoscaling_group" "dr_app" {
  provider = aws.dr_region

  # Keep minimal capacity in DR region
  min_size         = 1
  max_size         = 20
  desired_capacity = 1  # Scale up during failover

  launch_template {
    id      = aws_launch_template.app_dr.id
    version = "$Latest"
  }
}

DNS Failover with Route 53

resource "aws_route53_health_check" "primary" {
  fqdn              = "api-primary.example.com"
  port               = 443
  type               = "HTTPS"
  resource_path      = "/healthz"
  failure_threshold  = 3
  request_interval   = 10
}

resource "aws_route53_record" "api" {
  zone_id = aws_route53_zone.main.id
  name    = "api.example.com"
  type    = "A"

  failover_routing_policy {
    type = "PRIMARY"
  }

  alias {
    name                   = aws_lb.primary.dns_name
    zone_id                = aws_lb.primary.zone_id
    evaluate_target_health = true
  }

  health_check_id = aws_route53_health_check.primary.id
  set_identifier  = "primary"
}

resource "aws_route53_record" "api_dr" {
  zone_id = aws_route53_zone.main.id
  name    = "api.example.com"
  type    = "A"

  failover_routing_policy {
    type = "SECONDARY"
  }

  alias {
    name                   = aws_lb.dr.dns_name
    zone_id                = aws_lb.dr.zone_id
    evaluate_target_health = true
  }

  set_identifier = "secondary"
}

Recovery Runbooks

Automated Failover Runbook

#!/bin/bash
# disaster-recovery-failover.sh
set -euo pipefail

echo "=== DISASTER RECOVERY FAILOVER INITIATED ==="
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "Operator: $USER"

# Step 1: Verify primary is truly down
echo "[Step 1] Verifying primary region failure..."
for i in {1..5}; do
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 \
    https://api-primary.example.com/healthz || echo "000")
  if [ "$STATUS" == "200" ]; then
    echo "Primary appears healthy. Aborting failover."
    exit 1
  fi
  sleep 2
done
echo "Primary confirmed down after 5 checks."

# Step 2: Promote DR database
echo "[Step 2] Promoting DR database to primary..."
aws rds promote-read-replica-db-cluster \
  --db-cluster-identifier production-dr \
  --region us-west-2

echo "Waiting for DB promotion..."
aws rds wait db-cluster-available \
  --db-cluster-identifier production-dr \
  --region us-west-2

# Step 3: Scale up DR application tier
echo "[Step 3] Scaling DR application servers..."
aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name dr-app-asg \
  --desired-capacity 6 \
  --region us-west-2

# Step 4: Update DNS (if not using Route 53 health checks)
echo "[Step 4] DNS failover in progress via Route 53 health checks..."

# Step 5: Verify DR environment
echo "[Step 5] Running smoke tests against DR..."
sleep 30
DR_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  https://api-dr.example.com/healthz)
if [ "$DR_STATUS" == "200" ]; then
  echo "DR environment is healthy."
else
  echo "WARNING: DR health check returned $DR_STATUS"
fi

echo "=== FAILOVER COMPLETE ==="
echo "Action items:"
echo "1. Monitor DR environment for 30 minutes"
echo "2. Notify customers of incident"
echo "3. Begin root cause analysis on primary"
echo "4. Plan failback procedure"

DR Testing Schedule

DR plans must be tested regularly to remain reliable:

  • Monthly: Backup restore verification (automated)
  • Quarterly: Tabletop exercise with the engineering team
  • Semi-annually: Full failover test to DR region during maintenance window
  • Annually: Chaos engineering exercise (simulate unexpected failure)

Communication Plan

During a disaster, clear communication is as important as technical recovery:

  1. Internal: Incident commander announces failover in dedicated Slack channel
  2. Status page: Update status.example.com within 5 minutes of incident detection
  3. Customer email: Send notification within 30 minutes for prolonged outages
  4. Post-mortem: Publish incident report within 5 business days

Cost Optimization

DR infrastructure does not need to match production capacity during normal operations. Use the pilot light model:

  • Keep minimal compute running in the DR region (1-2 instances)
  • Use database read replicas that can be promoted
  • Store AMIs and container images in the DR region
  • Script the scale-up process for rapid capacity expansion during failover

For comprehensive disaster recovery planning tailored to your infrastructure, our team conducts DR assessments that map business requirements to technical recovery strategies.

Frequently asked questions

What is the difference between RPO and RTO in disaster recovery?

RPO (Recovery Point Objective) is the maximum data loss a business can accept, measured in time, and RTO (Recovery Time Objective) is the maximum acceptable downtime. An RPO of 1 hour means losing at most one hour of data. An RTO of 4 hours means systems must be operational again no later than 4 hours after the incident. Together the two numbers decide which disaster recovery strategy a system needs.

How much does a cloud disaster recovery setup cost per month?

Example monthly costs run from $200-500 for Tier 4 backup and restore, with RPO and RTO under 24 hours, up to $15,000+ for Tier 1 multi-region active-active with near-zero RPO and an RTO under 15 minutes. Warm standby (Tier 2, under 1 hour) sits at $5,000-10,000 and pilot light (Tier 3, under 4 hours) at $1,000-3,000. Most organizations mix tiers, for example Tier 1 for customer-facing APIs and Tier 4 for internal reporting.

How do I reduce the cost of a disaster recovery region on AWS?

Run the DR region on the pilot light model, so it stays well below production capacity during normal operations. Keep minimal compute there (1-2 instances), use database read replicas that can be promoted, and store AMIs and container images in the DR region. Script the scale-up so capacity can grow quickly during a failover. In the example Terraform setup, the DR Auto Scaling group runs at a desired capacity of 1 with a maximum of 20, and the Aurora replica uses a smaller db.r6g.large instance.

How do I test that a MySQL backup can actually be restored?

Restore the newest backup into a separate test database and compare it with production. The example script downloads the latest dump from S3, loads it into a test_restore database with gunzip and mysql, then compares the row count of the users table with the production count and reports PASS or FAIL. Afterwards it drops the test database and deletes the downloaded file. The backup job itself also checks each compressed dump with gunzip -t, but a backup that has never been restored is still only an assumption.

How does Route 53 DNS failover to a disaster recovery region work?

Route 53 pairs a health check on the primary endpoint with two failover records for the same name. In the example, the health check calls /healthz over HTTPS on port 443 every 10 seconds and marks the primary unhealthy after 3 failures. The PRIMARY record points at the main load balancer and carries that health check, while the SECONDARY record points at the DR load balancer. When the primary fails its checks, DNS answers switch to the DR region.

What steps should an automated disaster recovery failover script run?

First the script confirms the primary is really down, for example by checking its /healthz endpoint 5 times and aborting if any check returns 200. Next it promotes the DR database with aws rds promote-read-replica-db-cluster, waits until the cluster is available, scales the DR Auto Scaling group (to 6 instances in the example) and runs smoke tests against DR. Route 53 health checks handle the DNS switch. Follow-up items are monitoring DR for 30 minutes, notifying customers, root cause analysis and planning failback.

How often should a disaster recovery plan be tested?

Test at four intervals, with a different exercise at each. Automated backup restore verification runs monthly, the engineering team holds a tabletop exercise every quarter, a full failover to the DR region happens twice a year during a maintenance window, and a chaos engineering exercise once a year simulates an unexpected failure. A DR plan stays reliable only when it is exercised regularly.

How quickly should customers be told about an outage during disaster recovery?

Update the status page within 5 minutes of detecting the incident, and email customers within 30 minutes when the outage is prolonged. Inside the company, the incident commander announces the failover in a dedicated Slack channel. A post-mortem incident report follows within 5 business days. During a disaster, clear communication matters as much as the technical recovery.

Read how we handle it in Infrastructure Setup.