A useful recovery runbook gives each observable symptom a bounded decision path, names the authority and preconditions for every action, and ends with explicit evidence that normal service behavior returned.
Connect symptoms to the first safe decision
Organize the runbook around observable symptoms such as elevated errors, stalled workflows, missing output, incorrect data, duplicate effects, queue backlog, or a failed dependency. Each entry should name the dashboard or query, threshold, alert owner, affected service versions, and initial evidence to preserve. The first action may be to stop new work, reduce a feature, roll back a deploy, isolate a queue, or hold automated replay while uncertain effects are reconciled.
Google's incident-management guidance starts with restoring operations while preserving evidence for root-cause work. The runbook should identify the incident commander or technical owner, communication location, customer-impact recorder, and handoff method. A service can be technically recovering while coordination fails, so the document needs live status, decisions, owners, and timestamps alongside commands.
Give commands preconditions and stopping rules
For every recovery action, state the required role, current system state, expected effect, command or interface, verification query, rollback or compensating action, and point where escalation replaces further attempts. A command copied without its environment and version can make the incident worse. Use named variables and examples that cannot run against production by accident, and keep credentials outside the document.
Test the runbook with a staging failure or tabletop scenario. AWS reliability guidance calls for playbooks, post-incident analysis, resilience tests, game days, and tested disaster-recovery implementation. Record the date, service and runbook versions, participants, injected failure, actions, outcome, and corrections. An unexercised step remains a proposal, even when its syntax looks plausible.
Verify recovery and convert the incident into controls
Recovery evidence should include health checks, customer-path tests, queue and state reconciliation, error-rate return, effect counts, and the decision that ended the incident. Keep degraded or held behavior visible. The post-incident record should distinguish confirmed cause, contributing conditions, and unresolved questions, then assign corrective actions that change instrumentation, tests, alerts, permissions, deployment, or the runbook itself.
Reality Contact, LLC prepares and tests this runbook for Incident Recovery Record. The buyer retains incident command, production credentials, communication authority, and approval for every action. The runbook covers the named service and scenarios; it does not create a continuing response obligation or promise recovery from failures outside the tested paths.
Where the service stops
Reality Contact, LLC installs bounded telemetry and recovery controls but does not provide forensic determinations, certify security or compliance, decide lawful retention, approve production access, or supply continuing incident response. The buyer approves the telemetry fields, redaction and retention rules, alert owners, recovery authority, scenarios, production credentials, and final deployment. This is technical implementation and incident-record preparation; it does not replace the buyer's legal, privacy, security, compliance, or forensic review. We do not promise complete telemetry, a single root cause, error-free recovery, continuous availability, or visibility into systems that do not expose records.
Sources: Google SRE guidance for managing incidents; AWS Well-Architected reliability pillar.