DISASTER RECOVERY · RESTORE DRILL
Disaster recovery: test that a backup can restore
Take scheduled backups, inject a hidden defect, lose primary storage and find out which backup actually restores.
Event trace
Computed data
Final metrics
Fictional schedule. A detected bad backup raises an alert and a corrected backup is taken one hour later if that is before the next scheduled backup and the failure. Restoring a block-corrupted backup fails only after a full read; a backup with a lost key fails after five minutes at decryption. Log replay, partial restores, cross-region copies and human response time are not modeled.
A backup that was never restored
Backups run every 6 h, the last 4 are kept, and nothing is verified. The hidden defect: backups after hour 24 use a lost encryption key. Primary storage fails at hour 40. The hour 36 and hour 30 backups each fail after 5 minutes at decryption, then hour 24 restores 500 GB at 5 GB/min in 100 minutes. Total recovery is 110 minutes, inside the 120 minute RTO, but 16 hours of data are lost against a 6 hour RPO: RECOVERED, TARGETS MISSED.
Verify by restoring
Set Verification after each backup to Test restore into a scratch instance. The hour 30 backup fails its test, an alert fires and a corrected backup is taken at hour 31; hour 36 then passes. After the failure, hour 36 restores in 100 minutes with 4 hours lost: RECOVERED WITHIN TARGETS. Choose Checksum of stored bytes instead and the result is the same miss as no verification, 16 hours lost: the stored bytes are intact, they just cannot be decrypted.
When a checksum does help
Set Hidden defect to Newest backup has a corrupted block. Without verification the hour 36 restore reads all 500 GB before failing, then hour 30 succeeds: 200 minutes and 10 hours lost, missing both targets. With checksums the corrupted block is caught at write time, a corrected backup is taken at hour 37, and recovery takes 100 minutes with 3 hours lost. With no defect the default run recovers from hour 36 within targets.
Retention, RPO and RTO
Back on the lost key, set Backups retained to 1: only hour 36 is kept, it fails, and the result is NO RESTORABLE BACKUP. With test restores even 1 retained backup is enough. With no defect, Restore throughput at 4 GB/min takes 125 minutes and misses the RTO; Data size at 1,000 GB takes 200 minutes. Set Backup interval to 12 h and Primary failure at hour to 47: 11 hours are lost, missing a 6 hour RPO; set Recovery point objective to 12 h and the same run meets it. Recovery time objective is the minute budget for the whole restore sequence.
Transport and limits
Play, Pause, Next event, Restart, Speed and Inspection time step through each backup, the failure and each restore attempt; the verdict and metrics settle at 60 s, and the preview shows it immediately. A detected bad backup is retaken one hour later only if that fits before the next backup and the failure. Log replay, partial restores, cross-region copies and human response time are not modeled.