What a Real Disaster Recovery Drill Looks Like on a Tuesday

A recovery plan gets tested one of two ways. Either somebody sits down on a Tuesday morning with a scenario and a stopwatch, or the test arrives at three in the morning with a customer already on the phone. The first costs two hours and produces a list of fixes. The second costs whatever an outage costs you.

A drill is not a fire alarm. Nobody is being ambushed, nothing in production is being broken, and the point is not to prove the team can perform under pressure. The point is to walk the recovery path in daylight and write down every place where it does not work.

Give it a scope small enough to finish

Pick one scenario and one system. "The primary database is gone" is a drill. "The region is gone" is a workshop that ends in an argument about DNS while nobody restores anything.

Rotate the scenario so that over a year you have walked the paths that matter, the database, the object storage, a full application host, and the credentials themselves. Write the scenario in two sentences before the day, naming the system presumed lost, the moment it was lost, and what is still assumed to be working. Half of a bad drill is people arguing about what is allowed to still be up.

Run it in working hours and announce it

Out of hours drills test whether people can be woken up, which you already know. Daylight tests the documentation, the access paths and the assumptions, and those are the parts that actually fail.

Announce it, put it in the calendar, and let people prepare in the sense of reading the runbook. If a recovery only works when nobody sees it coming, the thing you have discovered is not a scheduling preference.

Two roles, and the operator should not be the author

The person running the recovery should be someone who did not write the runbook. That single choice finds more problems than any other part of the exercise, because the author's private knowledge is exactly what is missing from the document.

The second role holds the clock and writes. Every time the operator pauses, asks a question, opens a chat window, guesses, or takes a step that is not in the document, that is a line in the log. Those lines are the output of the drill. The recovery itself is only the vehicle for producing them.

Measure four things

• Time from declaration to the first useful action. Not the first message in a channel, the first command that moves recovery forward. Ten minutes spent finding the runbook is ten minutes you will spend again.
• Time to a service that answers real requests, set against the recovery time objective written on paper.
• Data actually lost, set against the recovery point objective. Compare the newest row in the recovered copy with the newest row that existed, in a table that changes every minute.
• The number of stops. This is the number that predicts how the real incident goes.

A recovery that hits its target because one person happened to remember a detail is not a recovery the company owns. It belongs to that person, and at some point they will be on a plane.

Decide what a pass is before you start

Write the pass condition down while nobody is under pressure. A useful one has the recovered system serving real requests, a second person verifying the data with a query they choose themselves, and no step that was missing from the runbook. "We could see the data" is not a pass. "The data was there and the application connected to it and answered" is.

Make the verification query something a human recognises rather than a row count on a table that only grows. Recent orders, yesterday's invoices, the last hour of events.

The failures a drill finds, nearly every time

• The runbook names a person instead of a role, and the person left in March.
• Credentials on the recovery path have expired, because nobody rotates a secret nobody uses.
• The DNS record everybody plans to change has a time to live longer than the recovery time objective.
• The restore works and nobody had ever timed it, so the documented objective was a hope rather than a measurement.
• The recovery environment has drifted from production. AWS puts this in its own disaster recovery guidance as "manage configuration drift at the DR Region. Ensure that your infrastructure, data, and configuration are as needed at the DR Region."
• The monitoring that would confirm recovery runs on the thing that is presumed dead.

Rarely used paths fail because they are rarely used, and the same AWS guidance says it about as plainly as it can be said. "Our experience has shown that the only error recovery that works is the path you test frequently."

Write the outcome the same day

One page, while it is fresh. The scenario, the start time, the time the service came back, the targets against the observed numbers, the list of stops, and against each stop an owner and a date. A finding without an owner is a wish with better formatting.

Then book the next run of the same scenario. Comparable numbers are the whole reason to repeat one, and a shorter list of stops on the second run is the evidence that the fixes were real rather than filed.

Where the exercise gets wasted

Treating it as an exam is the common one. If the drill touches anyone's performance review, the environment gets quietly prepared beforehand and you have bought theatre.

Drilling against a copy that is not the real backup is the second. A restore from a hand-made snapshot taken that morning proves nothing about the nightly job, which is the thing you will actually reach for.

Skipping the write-up because it went well is the third, and it is the most expensive. Six months later, a clean run with no document is indistinguishable from no run at all.

If you want the drill designed, run and written up by people who do it regularly, that sits inside our SRE services and our disaster recovery and backup work. The restore job that a drill assumes is already working is covered in how to build a backup you have actually restored.