Most disaster recovery writing is aimed at organizations that can afford to run two full production regions in parallel. The rest of us - teams running real workloads on AWS without that budget - are left with whitepapers, vague checklists, and a hope that the primary region keeps working. This post lays out what low-cost DR on AWS actually looks like in practice: what to pre-stage continuously, what to rebuild mid-incident, and the specific design decisions that make the difference between a runbook that works and one that doesn’t.
Why we need DR
On December 7, 2021, us-east-1 went down for roughly seven hours. Slack, Disney+, Robinhood, Coinbase, Netflix, Ring, and large parts of the AWS Console itself were unreachable or degraded. It wasn’t a one-off. us-east-1 had another partial outage two weeks later. There were further significant incidents in 2023 and 2024 - DynamoDB, Lambda, IAM control-plane events that rippled across services. The pattern is consistent: AWS regions fail, and when they do, the workloads sitting inside them go with them.
The thing engineers underestimate is not the probability - most teams accept that regions fail - but the operational state they’ll be in when it happens. You will not be at your desk. The on-call engineer may not have full AWS access. Half your tooling may be co-located in the failed region. Decisions made under that pressure are uniformly bad. The point of DR is not just to have a second region; it is to make those mid-incident decisions cheap and few.
SOC 2 requires a documented, tested business continuity and disaster recovery process. Auditors don’t accept “we have backups” - they want evidence of a defined RTO/RPO, a written recovery procedure, and a dated record of it being executed. If you are pursuing SOC 2, DR stops being optional regardless of how stable your primary region is.
So the question is not whether to invest in DR. It’s how cheaply you can do it without it being theatre.
The five hard parts
Most of the cost and most of the failure modes in DR live in five places. Naming them is half the battle.
1. State migration. Data is the one thing you cannot rebuild from a template. It splits into three classes with three different recovery models. Durable state (RDS, Aurora, anything with a primary copy) is recovered via point-in-time recovery or cross-region snapshot copies - AWS does the heavy lifting if you’ve enabled the right setting. Object state (S3) is recovered via cross-region replication, which is per-bucket and easy to forget on new buckets. Transient state (SQS, Redis, in-flight jobs) is generally not recovered at all - accept the loss, design idempotent consumers, and document what gets replayed on failover.
2. Configuration as code, replicated continuously. Parameter Store, Secrets Manager, environment configuration - these are small, change frequently, and are catastrophic to recreate from memory mid-incident. Set up a continuous replicator (a Lambda triggered by SSM events works fine) and forget about it. The subtle bug here: a small subset of parameters are region-bound (queue URLs, region-specific endpoints) and need to be rewritten on failover, not blindly copied. Tag them in the parameter name and handle them explicitly.
3. DNS as the cutover mechanism. The actual moment of failover is a DNS change. Two things determine whether it works. First, TTL: if your production records have a 24-hour TTL, your effective RTO is 24 hours regardless of how fast everything else is. Drop production TTLs to 60 seconds well before you need them. Second, if you use Cloudflare (or any proxy CDN), proxy-mode caching means flipping the DNS record alone won’t always route traffic - you may need to grey-cloud during validation and re-orange after. Test this, do not assume.
4. IAM in a multi-region world. Half the resources in AWS have region-scoped ARNs - ECR repositories, KMS keys, ACM certificates, Secrets Manager entries. Your CI/CD policies, your deploy roles, your application execution roles all reference these. The dangerous pattern is keeping a local copy of the policy JSON and re-uploading it on changes - this silently strips any ARN the local copy doesn’t include. The right pattern is fetch-merge: pull the current policy from AWS, append the secondary-region ARNs derived from the existing primary-region ones, upload the result. The merge is one jq expression. Hardcoded local copies are the single most common cause of a broken deploy after a DR-related policy update.
5. External integrations. Stripe webhooks, GitHub Apps, OAuth callbacks, Slack apps, Linear integrations - each holds a webhook URL pinned to your primary region. Some support multiple URLs; many do not. This is the most error-prone phase of any failover by a wide margin, because it lives outside AWS and outside your runbook’s automation. Maintain a list of every external integration, document the failover procedure for each, and accept that this section of the runbook will always be the longest.
The runbook as an executable artifact
The single biggest distinction between teams whose DR works and teams whose DR doesn’t is how they treat the runbook.
The runbook is not documentation. It is an executable artifact. Treat it like code.
Version it. Review changes. Refactor when phases stop making sense. Every command is copy-pasteable; every phase is idempotent; nothing assumes the previous step’s output is still on screen. There are no instructions of the form “the team should…” - only “run this.” If a step requires judgment, the runbook lists the judgment criteria explicitly, not by implication.
The anti-pattern is a Confluence page last edited two years ago describing what should happen in the abstract. That is a story, not a plan. If it has not been executed in six months, treat it as deprecated.
A common counter-argument is that incidents should be fully automated - push a button, region failover happens. This is wrong for most teams, and the reason is brittleness. Automation is reliable in steady state, which is exactly when you don’t need it. In the conditions that produce real incidents - control-plane degradation, partial service availability, unexpected resource state - the automation fails in ways that are hard to debug under pressure. A well-written runbook executed by a calm engineer is faster and more reliable than a half-working automation script.
Infrastructure as Code is the right tool for the rebuild steps - the ECS services, ALBs, target groups, security groups, ElastiCache clusters. Those are the parts of recovery that are mechanical, well-defined, and benefit from being declarative. The runbook calls IaC. What the runbook does not delegate is the sequencing, the verification gates, and the decision points - those stay human. IaC for the rebuild, runbook for the orchestration. Treating them as opposed misses that they solve different parts of the problem.
Drills as routine validation
The objection to drills is always cost: you don’t want to bring up a full second region every quarter. The good news is you don’t have to. The non-destructive drill pattern lets you validate the entire secondary stack without touching production DNS. Bring up the secondary region, then validate it directly:
curl --resolve api.example.com:443:<secondary-alb-ip> <https://api.example.com/health>
-resolveoverrides DNS for this one request. The request hits your secondary ALB with the correct Host header, exercises the full TLS and application stack, and proves the region works - all without a single production user being routed there. Repeat for every critical endpoint.
Two-tier cadence works well for most teams: a dev-tier drill quarterly (full stand-up of the secondary region against dev data, end-to-end validation, full teardown) and a prod-tier drill annually (full rehearsal with real data, no actual cutover). The dev-tier drill catches runbook drift, broken IAM, stale snapshots, and integration gaps. The prod-tier drill catches everything the dev-tier missed and forces the team to actually do it on production-shaped data.
The kicker is the cost. An end-to-end dev-tier drill costs roughly $2 in AWS charges - a few hours of ECS, ALB, and RDS uptime in the secondary region, then teardown.
Anti-patterns to design around
Six patterns we see repeatedly, with the design that prevents each.
Hardcoded region-specific resources in runbook commands. Prevention: fetch-merge for any cross-region resource list. Never keep a “local copy” of an IAM policy.
IAM managed policy version exhaustion. AWS caps managed policies at five versions. Prevention: auto-prune the oldest non-default version as part of every policy update.
Silent gaps in state coverage, particularly S3. Buckets created after the DR plan was written often miss cross-region replication. Prevention: maintain an explicit state-coverage matrix in the runbook header and re-audit quarterly.
Implicit region-bound configuration. Parameters that look generic but contain a region-specific value. Prevention: tag every region-bound parameter at creation; the failover step rewrites only the tagged set.
Skipping context verification before destructive commands. Prevention:
aws sts get-caller-identityand an explicit region check as the first commands of any DR session. The cost is ten seconds; the alternative is running prod commands against dev.Runbook drift. Documentation and reality diverge silently. Prevention: drills are CI for the runbook itself. If a drill fails, the runbook is wrong, not the cloud.
Cost as a design constraint
The economic argument for DR has shifted entirely in the last few years, and most teams haven’t updated their priors.
A pre-staged DR setup - configuration replicated continuously, secondary-region IAM and IaC in place, no running compute - costs in the range of a few dollars to a few tens of dollars per month in steady state. The exact number depends on snapshot retention, parameter count, and KMS keys, but it is small. A drill costs around $2. A real failover costs full secondary-region pricing for the duration of the incident, which is bounded by hours.
Compare that to the cost of a multi-hour outage, the cost of a SOC 2 finding, or the cost of an engineer reconstructing infrastructure from memory at 3 a.m. Cost has stopped being the limiting factor for DR. The limiting factor is now design discipline - whether the team is willing to invest the days of architectural work required to do it correctly.
Design, then drill
Two principles worth carrying away.
DR is a design problem, not a procurement problem. No amount of AWS spend compensates for an undesigned failover process. A team with a well-designed pre-staged setup and a tested runbook will recover faster than a team with a continuously running warm standby that has never been exercised.
A runbook that hasn’t been drilled is a story, not a plan. If your DR plan lives in Confluence and hasn’t been executed in six months, treat that as a P1.
The work is not small, but it is bounded, finite, and - at the scale most teams operate - cheap. The hard part is starting.


