dr-drill-runner

Per the Google Cloud DR planning guide, DR planning requires "end-to-end recovery design addressing backup, restoration, and cleanup procedures." Drills test that the procedure works AND that the team can run it. Both surface different failures.

When to use

• Quarterly DR drill (mandatory in compliance-heavy industries: banking, healthcare, defense).
• After a region-failover incident: rerun the drill with the lessons learned.
• New service onboarding: every new tier-1 service ships with its drill defined.

Step 1 - Define RTO + RPO per service tier

Per the Google Cloud DR planning guide:

Document per service in a service catalog; drills enforce the contract.

Step 2 - DR-pattern tier per service

Per the Google Cloud DR planning guide:

• Cold: Minimal preparation; recovery requires external intervention and extended downtime.
• Warm: Basic readiness with resources available; recovery stops normal ops temporarily.
• Hot: Continuous operation with built-in redundancy; minimal interruption.

Drill expectations differ:

• Cold: Test bring-up from backup (restore-time tests - references/restore-time.md).
• Warm: Test failover automation + warm-up time.
• Hot: Test traffic redirection + sticky-session impact.

Step 3 - Pre-drill checklist

## Pre-Drill Checklist - `<service>` `<date>`

- [ ] Drill window scheduled (low-traffic; aligned with
      customer-comm window)
- [ ] Drill scope decided (region, single service, full app)
- [ ] Replication lag confirmed within RPO at T-30 min
- [ ] Monitoring alerts SILENCED for expected failure indicators
      (alert routing redirected to drill channel)
- [ ] On-call notified (avoid duplicate paging during drill)
- [ ] Customer comms sent if customer-impacting drill
- [ ] Rollback path documented (what triggers abort?)
- [ ] Drill commander assigned (owns go/no-go calls)
- [ ] Postmortem time scheduled (within 48hr of drill end)

Skipping the pre-drill = drills become incidents.

Step 4 - Drill workflow

## Drill Workflow

### T-0: Announce
- Post in #drill-channel; confirm all participants ready.
- Drill commander gives "GO" - record T-0 timestamp.

### T+0..N: Fail-over
- Execute the runbook step-by-step (everyone follows the doc; no
  improvisation).
- Capture timestamp of each step.

### Verify
- Run the verification suite (smoke + customer-impact + data integrity).
- Compare actual vs expected RTO; if RTO breached, decide:
  abort + rollback, or continue + capture learning.

### Fail-back
- If hot/warm: redirect traffic back to primary.
- If cold: tear down DR environment + restore primary.
- Verify primary is healthy before claiming drill complete.

### Cleanup
- Re-enable alerts (Step 3).
- Send "all clear" customer comms.
- Reconcile any drill-introduced data divergence.

Step 5 - Post-drill report

## Drill Report - `<service>` `<date>`

**Drill objective:** Verify warm standby fails over within RTO 4hr.

**Timeline:**
- T-30 min: Replication lag verified (52s - within RPO 1hr) ✓
- T-0: Announced, on-call silenced
- T+12m: Failover initiated
- T+47m: Standby took traffic
- T+1h22m: Verified service healthy on standby
- T+2h11m: Failback to primary
- T+3h05m: Drill complete

**RTO observed:** 1h22m (target: 4hr) ✓

**Issues found:**
1. CRITICAL: DNS TTL was 24hr in standby DNS records; users
   couldn't reach service for 23min after failover. Fix: lower
   TTL to 60s in standby zone before next drill.
2. MAJOR: Secret-manager copy step was undocumented; commander
   improvised. Fix: add Step 3.4 to runbook.
3. MINOR: One alert wasn't silenced in advance; on-call was paged.

**Action items (with owners + dates):**
- DNS TTL fix → @platform-team - 2026-05-20
- Runbook Step 3.4 → @sre - 2026-05-13
- Alert routing audit → @sre - 2026-05-13

**Next drill:** 2026-08-06 (quarterly cadence).

Step 6 - Cold-tier-specific drill pattern

Cold drills = bring up from backup. Verifies:

• Backup is current within RPO (references/backup-verification.md).
• Restore time is within RTO (references/restore-time.md).
• Infrastructure-as-code provisioning works (Terraform / CloudFormation / Bicep in DR account).
• Permissions + secrets are in place (per the Google Cloud DR planning guide, "Permission and access validation in DR environments" + "Security synchronization").

Step 7 - Hot-tier-specific drill pattern

Hot drills = redirect traffic between active replicas. Verifies:

• Health check propagation (load balancer detects standby is healthy).
• Sticky-session handling (do connections drain or break?).
• Cache warmup not required (or warmup time is within RTO).
• Cross-region replication lag stays within RPO during the drill.

Step 8 - Cadence

Per the Google Cloud DR planning guide: "test it regularly, noting any issues." Without cadence, runbooks rot.

Run protocol (executing the drill end to end)

Nobody delegates failover execution; a human drill commander runs the five stages below against this protocol. This is the rehearsed DR path - for injecting unrehearsed failures see the chaos-drill-protocol skill in this plugin.

Refuse to start when:

• No declared RTO + RPO. Per the Google Cloud DR planning guide, RTO is "the maximum acceptable length of time that your application can be offline" and RPO bounds acceptable data loss - without them there is no pass/fail criterion.
• The DR environment identifier matches prod / production.
• Any CRITICAL pre-drill item fails (backup integrity, key recovery, alert silencing) - halt and emit the blocking checklist instead of proceeding.

Stage 1 - pre-drill. Run the Step 3 checklist. Verify backup SHA-256 integrity, replication lag within RPO at T-30 min, and encryption-key recoverability in the DR region per references/backup-verification.md. Verify DR-environment configuration drift is within bounds - per the AWS DR testing whitepaper, "Manage configuration drift at the DR Region. Ensure that your infrastructure, data, and configuration are as needed at the DR Region."

Stage 2 - failover. Record T-0; execute the Step 4 runbook step by step, capturing a timestamp per step. Per the AWS DR testing whitepaper, "Our experience has shown that the only error recovery that works is the path you test frequently" - a runbook step that requires improvisation is logged as a MAJOR finding immediately, never silently adapted.

Stage 3 - RTO/RPO monitor. While failover is active, sample on a fixed interval (60 s is a workable default): replication lag at the DR side, smoke-suite pass rate, and data-row spot checks. Measure time-to-functional for the Restore + Verification segments per references/restore-time.md and compare against the per-segment RTO budget. If observed TTF exceeds the RTO - abort the failover, record the breach metrics, and skip directly to the report. Record the peak RPO gap (replication lag at fail-over time); flag if it exceeds the declared RPO.

Stage 4 - fail-back. Per Step 4: redirect traffic back (warm/hot) or tear down + restore primary (cold), re-enable silenced alerts, send the all clear, verify primary health before recording fail-back complete, reconcile drill-introduced data divergence.

Stage 5 - report. Emit the Step 5 post-drill report, including the observed TTF, total RTO, and peak RPO gap versus their declared targets. Schedule the postmortem within 48 hours; every finding gets an owner + due date before the drill closes.

Anti-patterns

Limitations

• DR drills don't replace chaos engineering (this plugin's chaos skills) - they test rehearsed paths; chaos tests unrehearsed ones.
• Cloud-managed services may have built-in regional failover that bypasses your runbook; document boundaries.
• Some compliance regimes (FFIEC for banks) prescribe specific drill frequencies + scopes - verify per regulation.

References

• Google Cloud DR planning guide - RTO / RPO / cold-warm-hot tiers / testing requirements
• AWS DR testing whitepaper - configuration-drift management, "test the path you execute" principle
• references/backup-verification.md, references/restore-time.md - drill prerequisites: backup integrity + RTO measurement
• error-budget-tests - incident metrics fed by drills (MTTR/MTBF schema in its references)