👋 Hey {{first_name|there}},
Last week was about the generators that were never told to start. This week I want to go back a few days earlier, to the state the site was in when the cable failed. It’s a less dramatic story than the generators, but I think it’s the one more of us will recognise.
Google’s London data centre was built with two independent power feeds. When the outage began, it had been running on just one of them for several days. Nobody had done anything wrong to get it into that position. The site was partway through fixing a problem that one of its own routine tests had turned up.
Why this matters
Every bit of redundancy you design spends part of its life not being redundant. A database replica gets rebuilt. One of your two HSMs goes back to the vendor. A broker is pulled out mid-migration, the second acquirer connection gets paused for recertification, or a network link is down for an upgrade. The diagram still shows two of everything. For a while, though, you’ve really got one.
None of this is unusual. You can’t maintain or repair anything without taking it out of service, so these windows are always going to happen. What matters is whether anyone is keeping track of them.
Quite often nobody is. The team doing the work knows, and the change ticket probably mentions it somewhere. On-call often doesn’t, though, and neither does the incident commander. The people in risk who signed off on your resilience testing almost certainly don’t. “Temporary” also has a way of stretching. A rebuild meant to take a day ends up taking a week because a part arrived late.
In the meantime, every failure scenario you’ve tested assumed the full set was in place. While you’re down to one, a single fault you’d normally barely notice can take you down. If you think in UK operational resilience terms, planned maintenance plus one unplanned failure is about as severe but plausible as scenarios get.
There’s also an awkward overlap with last week. Systems behave differently in maintenance states, and those are usually the least tested states of all. Google’s generator logic only got confused because a breaker happened to be open for repairs.
🧭 The shift
From: “We’re fully redundant.”
To: “We’re designed to be fully redundant, and we know where we aren’t right now, since when, and which single failure would hurt us until it’s sorted.”
🔍 The lens: the reduced-redundancy register
Get an hour with the people who run the infrastructure under your payment path, and whoever covers on-call. You’re building one list, kept somewhere people will actually look.
Where is redundancy meant to be? Write down everywhere your critical path is supposed to have more than one of something. That includes database replicas, availability zones, brokers, HSMs, network links, acquirers or processors, DNS providers, and the number of people on call. If a diagram or an audit document says “redundant”, it goes on the list.
Is each one at full strength right now? Go through them one at a time. Where something’s down, note since when, why, and when it’s due back. Look hard at anything called “temporary” that’s more than a week old. The entries I’ve found most interesting over the years are the ones where someone in the room says “oh, that’s been like that for a while”.