👋 Hey {{first_name|there}},
Two weeks on what you depend on and what happens when it fails. This one's about the decisions you made deliberately, which is a harder conversation to have.
Why this matters
Every resilience model has exceptions in it.
The service that has to sit next to the database because the round trip won't fit in the budget. The cluster that can't span zones without consensus going slow. Something pinned to a particular rack or instance family because that's where the latency profile works out. Someone made each of those calls, usually for a decent reason, usually after a fair bit of argument.
The exceptions themselves aren't the problem. Resilience rules applied without judgement give you systems that are slower and cost more and aren't meaningfully more available than what they replaced.
What goes wrong is quieter.
The exception gets made, and justified, and then it stops being visible. Six months on, the diagram says multi-AZ across the board. The runbook assumes failover. Nobody wrote down that one component sits outside all of it, and the two people who could explain why have moved to other teams.
So when the zone goes, what happens isn't a failover. It's four engineers on a call working out, live, what recovery even looks like for something nobody expected to recover by hand.
Somebody will ask which parts weren't covered by the failover. Most teams answer that one slowly.
🧭 The shift
From: "We're multi-AZ."
To: "We're multi-AZ apart from these four things, and here's when each was last exercised."
Plenty of architecture diagrams make the first claim. Very few teams can produce the second on request.