👋 Hey {{first_name|there}},
New arc today. It’s seven issues, and it’s about backups.
The first arc asked who you depend on, and the second asked how far your own changes can reach. This one asks something a bit more uncomfortable. When the thing you built to rescue you finally gets called on, does it actually do anything? Redundancy almost always looks fine on the diagram and in the audit pack. What matters is how it behaves on a bad morning when half the conditions it was designed around aren’t there.
Most of the arc hangs off Google Cloud’s London outage on 20 October. You might have felt it yourself. Revolut put its disruption that day down to it, and people were reporting problems with Lloyds and HSBC too. Google put out its full report on 4 November, and it’s one of the more useful postmortems I’ve read this year.
We’ll start with the generators, because that’s where it started.
Why this matters
Underneath every automatic failover there’s one small decision: has the primary died? Database clusters make it, and so do load balancers, region routing, payment-route switches, and HSM pools. They make it all day long, mostly with nobody watching.
And that decision always comes down to a signal of some kind. It might be a health check, a heartbeat, or a voltage reading. Something gets measured somewhere, and a rule says that if it looks bad, switch.
We put a lot of thought into what happens after the switch. Is the standby warm? Is the data replicated? Is there enough capacity? Far less thought goes into the signal. Where is it measured, and could it look perfectly healthy while the thing it’s meant to protect is gone?
Get the signal wrong and nothing downstream of it matters. Your carefully replicated standby sits there, ready and idle, while everything around it goes dark, because nothing ever told it to start.
There’s a regulatory angle too. Google Cloud EMEA has been a designated Critical Third Party in the UK since 13 July, and this was the first time one of those providers properly fell over. The Treasury Committee wrote to the Bank of England and the FCA the same day asking what they plan to do about it. I’d expect questions about failover to start arriving from the board as well as from engineers.
🧭 The shift
From: “We have automatic failover.”
To: “We know what signal starts each failover, where it’s measured, and whether it could say ‘healthy’ while the primary is dead.”
🔍 The lens: the failover trigger audit
Take your most important payment path and get hold of whoever owns each bit of it. A first pass usually takes an hour or two.
List every automatic failover on the path. Anything that switches from one thing to another without a person deciding counts. That means database HA, load balancer pools, zone or region routing, leader election in your message broker, the switch between payment processors or acquirers, and DNS failover. You’ll probably find a few you’d forgotten, tucked inside managed services behind a checkbox someone ticked years ago.
Name the exact signal for each one. “There’s a health check” doesn’t count. Which endpoint is it? Where is it measured from, and what does it check? An endpoint that returns 200 because the web process is running tells you nothing about the database behind it. A heartbeat on the management network tells you nothing about the path customers’ requests take.
Could that signal look healthy while the thing it protects is dead? Look hard at two things. First, is the measurement taken upstream of something that can be switched off, like a breaker, a disabled pool, a feature flag or a maintenance mode? Second, is it checking a side path instead of the one customers actually use?