Design and implement disaster recovery strategies for cloud infrastructure with RPO/RTO planning, multi-region failover, and automated recovery runbooks.
Why DR Planning Cannot Wait
Every organization believes a major outage will not happen to them - until it does. AWS region outages, database corruption, ransomware attacks, and human error can take down production systems for hours or days. The cost of downtime varies by industry, but for most SaaS companies, a one-hour outage costs between $10,000 and $500,000 in lost revenue, SLA penalties, and customer trust.
Disaster recovery planning is a core part of architecture planning that transforms reactive panic into structured recovery procedures with predictable outcomes.
Understanding RPO and RTO
Two metrics define every DR strategy:
Recovery Point Objective (RPO): The maximum acceptable data loss measured in time. An RPO of 1 hour means you can lose at most 1 hour of data.
Recovery Time Objective (RTO): The maximum acceptable downtime. An RTO of 4 hours means your systems must be operational within 4 hours of an incident.
DR Tier Framework
| Tier | RPO | RTO | Strategy | Monthly Cost (Example) |
|---|---|---|---|---|
| Tier 1 | Near-zero | < 15 min | Multi-region active-active | $15,000+ |
| Tier 2 | < 1 hour | < 1 hour | Warm standby | $5,000-10,000 |
| Tier 3 | < 4 hours | < 4 hours | Pilot light | $1,000-3,000 |
| Tier 4 | < 24 hours | < 24 hours | Backup and restore | $200-500 |
Most organizations need different tiers for different systems. Customer-facing APIs might need Tier 1, while internal reporting can tolerate Tier 4.
Backup Strategy
Database Backups
#!/bin/bash
# Automated database backup script
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
DB_NAME="production"
BACKUP_DIR="/backups/mysql"
S3_BUCKET="s3://company-backups/mysql"
# Full backup with point-in-time recovery support
mysqldump --single-transaction --routines --triggers \
--master-data=2 --flush-logs \
-u backup_user -p"$DB_PASS" "$DB_NAME" | \
gzip > "${BACKUP_DIR}/${DB_NAME}_${TIMESTAMP}.sql.gz"
# Upload to S3 with server-side encryption
aws s3 cp "${BACKUP_DIR}/${DB_NAME}_${TIMESTAMP}.sql.gz" \
"${S3_BUCKET}/${DB_NAME}_${TIMESTAMP}.sql.gz" \
--storage-class STANDARD_IA \
--sse aws:kms
# Cross-region replication handles copying to DR region
# Verify backup integrity
gunzip -t "${BACKUP_DIR}/${DB_NAME}_${TIMESTAMP}.sql.gz"
echo "Backup completed: ${DB_NAME}_${TIMESTAMP}.sql.gz"
# Cleanup local backups older than 7 days
find "$BACKUP_DIR" -name "*.sql.gz" -mtime +7 -delete
Backup Verification
A backup nobody has restored is an assumption. Automate a restore test. The testing schedule below sets monthly as the minimum, and once the script exists, running it weekly costs nothing extra:
#!/bin/bash
# Automated backup restore verification (monthly at minimum)
LATEST_BACKUP=$(aws s3 ls s3://company-backups/mysql/ | sort | tail -1 | awk '{print $4}')
# Download to restore test server
aws s3 cp "s3://company-backups/mysql/${LATEST_BACKUP}" /tmp/
# Restore to test database
gunzip -c "/tmp/${LATEST_BACKUP}" | mysql -u root -p"$TEST_DB_PASS" test_restore
# Run integrity checks
EXPECTED_USERS=$(mysql -u root -p"$PROD_DB_PASS" -h prod-db -e "SELECT COUNT(*) FROM users" -sN)
RESTORED_USERS=$(mysql -u root -p"$TEST_DB_PASS" -e "SELECT COUNT(*) FROM test_restore.users" -sN)
if [ "$EXPECTED_USERS" -eq "$RESTORED_USERS" ]; then
echo "PASS: Backup integrity verified ($RESTORED_USERS users)"
else
echo "FAIL: Count mismatch (expected $EXPECTED_USERS, got $RESTORED_USERS)"
# Send alert
fi
# Cleanup
mysql -u root -p"$TEST_DB_PASS" -e "DROP DATABASE test_restore"
rm -f "/tmp/${LATEST_BACKUP}"
Multi-Region Architecture
Warm Standby with AWS
# Terraform configuration for DR region
resource "aws_rds_cluster" "dr" {
provider = aws.dr_region
cluster_identifier = "production-dr"
engine = "aurora-mysql"
engine_version = "8.0.mysql_aurora.3.05.2"
# Read replica of production cluster
replication_source_identifier = aws_rds_cluster.production.arn
# Smaller instance in DR (scale up during failover)
instance_class = "db.r6g.large"
tags = {
Purpose = "disaster-recovery"
}
}
resource "aws_autoscaling_group" "dr_app" {
provider = aws.dr_region
# Keep minimal capacity in DR region
min_size = 1
max_size = 20
desired_capacity = 1 # Scale up during failover
launch_template {
id = aws_launch_template.app_dr.id
version = "$Latest"
}
}
DNS Failover with Route 53
resource "aws_route53_health_check" "primary" {
fqdn = "api-primary.example.com"
port = 443
type = "HTTPS"
resource_path = "/healthz"
failure_threshold = 3
request_interval = 10
}
resource "aws_route53_record" "api" {
zone_id = aws_route53_zone.main.id
name = "api.example.com"
type = "A"
failover_routing_policy {
type = "PRIMARY"
}
alias {
name = aws_lb.primary.dns_name
zone_id = aws_lb.primary.zone_id
evaluate_target_health = true
}
health_check_id = aws_route53_health_check.primary.id
set_identifier = "primary"
}
resource "aws_route53_record" "api_dr" {
zone_id = aws_route53_zone.main.id
name = "api.example.com"
type = "A"
failover_routing_policy {
type = "SECONDARY"
}
alias {
name = aws_lb.dr.dns_name
zone_id = aws_lb.dr.zone_id
evaluate_target_health = true
}
set_identifier = "secondary"
}
Recovery Runbooks
Automated Failover Runbook
#!/bin/bash
# disaster-recovery-failover.sh
set -euo pipefail
echo "=== DISASTER RECOVERY FAILOVER INITIATED ==="
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "Operator: $USER"
# Step 1: Verify primary is truly down
echo "[Step 1] Verifying primary region failure..."
for i in {1..5}; do
STATUS=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 \
https://api-primary.example.com/healthz || echo "000")
if [ "$STATUS" == "200" ]; then
echo "Primary appears healthy. Aborting failover."
exit 1
fi
sleep 2
done
echo "Primary confirmed down after 5 checks."
# Step 2: Promote DR database
echo "[Step 2] Promoting DR database to primary..."
aws rds promote-read-replica-db-cluster \
--db-cluster-identifier production-dr \
--region us-west-2
echo "Waiting for DB promotion..."
aws rds wait db-cluster-available \
--db-cluster-identifier production-dr \
--region us-west-2
# Step 3: Scale up DR application tier
echo "[Step 3] Scaling DR application servers..."
aws autoscaling update-auto-scaling-group \
--auto-scaling-group-name dr-app-asg \
--desired-capacity 6 \
--region us-west-2
# Step 4: Update DNS (if not using Route 53 health checks)
echo "[Step 4] DNS failover in progress via Route 53 health checks..."
# Step 5: Verify DR environment
echo "[Step 5] Running smoke tests against DR..."
sleep 30
DR_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
https://api-dr.example.com/healthz)
if [ "$DR_STATUS" == "200" ]; then
echo "DR environment is healthy."
else
echo "WARNING: DR health check returned $DR_STATUS"
fi
echo "=== FAILOVER COMPLETE ==="
echo "Action items:"
echo "1. Monitor DR environment for 30 minutes"
echo "2. Notify customers of incident"
echo "3. Begin root cause analysis on primary"
echo "4. Plan failback procedure"
DR Testing Schedule
DR plans must be tested regularly to remain reliable:
- Monthly: Backup restore verification (automated)
- Quarterly: Tabletop exercise with the engineering team
- Semi-annually: Full failover test to DR region during maintenance window
- Annually: Chaos engineering exercise (simulate unexpected failure)
Communication Plan
During a disaster, clear communication is as important as technical recovery:
- Internal: Incident commander announces failover in dedicated Slack channel
- Status page: Update status.example.com within 5 minutes of incident detection
- Customer email: Send notification within 30 minutes for prolonged outages
- Post-mortem: Publish incident report within 5 business days
Cost Optimization
DR infrastructure does not need to match production capacity during normal operations. Use the pilot light model:
- Keep minimal compute running in the DR region (1-2 instances)
- Use database read replicas that can be promoted
- Store AMIs and container images in the DR region
- Script the scale-up process for rapid capacity expansion during failover
For comprehensive disaster recovery planning tailored to your infrastructure, our team conducts DR assessments that map business requirements to technical recovery strategies.
Frequently asked questions
What is the difference between RPO and RTO in disaster recovery?
RPO (Recovery Point Objective) is the maximum data loss a business can accept, measured in time, and RTO (Recovery Time Objective) is the maximum acceptable downtime. An RPO of 1 hour means losing at most one hour of data. An RTO of 4 hours means systems must be operational again no later than 4 hours after the incident. Together the two numbers decide which disaster recovery strategy a system needs.
How much does a cloud disaster recovery setup cost per month?
Example monthly costs run from $200-500 for Tier 4 backup and restore, with RPO and RTO under 24 hours, up to $15,000+ for Tier 1 multi-region active-active with near-zero RPO and an RTO under 15 minutes. Warm standby (Tier 2, under 1 hour) sits at $5,000-10,000 and pilot light (Tier 3, under 4 hours) at $1,000-3,000. Most organizations mix tiers, for example Tier 1 for customer-facing APIs and Tier 4 for internal reporting.
How do I reduce the cost of a disaster recovery region on AWS?
Run the DR region on the pilot light model, so it stays well below production capacity during normal operations. Keep minimal compute there (1-2 instances), use database read replicas that can be promoted, and store AMIs and container images in the DR region. Script the scale-up so capacity can grow quickly during a failover. In the example Terraform setup, the DR Auto Scaling group runs at a desired capacity of 1 with a maximum of 20, and the Aurora replica uses a smaller db.r6g.large instance.
How do I test that a MySQL backup can actually be restored?
Restore the newest backup into a separate test database and compare it with production. The example script downloads the latest dump from S3, loads it into a test_restore database with gunzip and mysql, then compares the row count of the users table with the production count and reports PASS or FAIL. Afterwards it drops the test database and deletes the downloaded file. The backup job itself also checks each compressed dump with gunzip -t, but a backup that has never been restored is still only an assumption.
How does Route 53 DNS failover to a disaster recovery region work?
Route 53 pairs a health check on the primary endpoint with two failover records for the same name. In the example, the health check calls /healthz over HTTPS on port 443 every 10 seconds and marks the primary unhealthy after 3 failures. The PRIMARY record points at the main load balancer and carries that health check, while the SECONDARY record points at the DR load balancer. When the primary fails its checks, DNS answers switch to the DR region.
What steps should an automated disaster recovery failover script run?
First the script confirms the primary is really down, for example by checking its /healthz endpoint 5 times and aborting if any check returns 200. Next it promotes the DR database with aws rds promote-read-replica-db-cluster, waits until the cluster is available, scales the DR Auto Scaling group (to 6 instances in the example) and runs smoke tests against DR. Route 53 health checks handle the DNS switch. Follow-up items are monitoring DR for 30 minutes, notifying customers, root cause analysis and planning failback.
How often should a disaster recovery plan be tested?
Test at four intervals, with a different exercise at each. Automated backup restore verification runs monthly, the engineering team holds a tabletop exercise every quarter, a full failover to the DR region happens twice a year during a maintenance window, and a chaos engineering exercise once a year simulates an unexpected failure. A DR plan stays reliable only when it is exercised regularly.
How quickly should customers be told about an outage during disaster recovery?
Update the status page within 5 minutes of detecting the incident, and email customers within 30 minutes when the outage is prolonged. Inside the company, the incident commander announces the failover in a dedicated Slack channel. A post-mortem incident report follows within 5 business days. During a disaster, clear communication matters as much as the technical recovery.
Read how we handle it in Infrastructure Setup.
Related Articles
What a Real Disaster Recovery Drill Looks Like on a Tuesday
A recovery plan gets tested one of two ways, on a Tuesday morning with a scenario and a stopwatch, or at three in the morning with a customer on the phone. This is the first version, run in working hours and announced in advance, because the failures worth finding are missing documents and expired credentials rather than whether anyone can be woken up. What to scope, the four numbers to measure, what counts as a pass, and the failures a drill turns up nearly every time.
StrategySysOps or DevOps? Understanding the Core Differences
A practical comparison of SysOps and DevOps operational models, covering their philosophies, responsibilities, tooling, and guidance on choosing the right approach for your organization.
StrategyThe Real Cost of Server Downtime - And How to Calculate Yours
A practical guide to calculating the true cost of server downtime, including revenue loss formulas, SLA penalties, brand damage, recovery expenses, and the ROI of prevention.