👋 Hey {{first_name|there}},
This is the third issue with Google's London outage, and probably the one that's most likely to apply to your own systems this week.
The first two were about the building: generators that never started, and a site quietly running on one feed. This one is about software, and a promise most of us have made at some point. You keep three copies of the data in three zones, so if a zone goes, the other two carry on. Google's Apigee X was built that way. On 20 October, it still returned errors for more than seven hours. Two of its three copies were sitting in the zone that lost power.
Why this matters
A lot of what sits on a payment path depends on a quorum. That includes databases with a primary and replicas, Kafka with its in-sync replicas, etcd underneath Kubernetes, and Redis with Sentinel. Most managed "highly available" services work the same way too. You keep three copies and need two of them to agree. If each copy lives in a different zone, losing any one zone still leaves you two.
That only holds if the copies really are in three zones, and stay there.
On the day a cluster's built, they usually are. After that, things move. A node gets replaced. A zone has a bad half-hour, and pods get rescheduled wherever there's room. A failover moves the primary somewhere handy, or an upgrade drains a node, and the scheduler makes its best guess about where to put things back. You can't fault any one of those moves. But after a few months, two of the three copies can end up in the same zone without anyone ever deciding that.
The design doc still says three zones, and so does the Terraform. The dashboard shows three healthy replicas, which is accurate, since all three are healthy. It just doesn't tell you where they are.
You won't notice on an ordinary day either. A skewed cluster works fine right up to the moment the zone holding two copies disappears, and then the quorum goes with it.
🧭 The shift
From: "It's replicated across three zones."
To: "It's replicated across three zones today, we've checked, and something stops it from drifting into two."
🔍 The lens: three questions and a tally
The lens: the quorum placement check
You can mostly do this from a desk. You'll need someone who can run commands against your clusters, plus someone who knows which managed services you rely on. An afternoon should cover a first pass.
List the replicated things on your payment path. That means databases, message brokers, coordination stores like etcd or ZooKeeper, and caches running in HA mode, plus the managed versions of all of them. If it has replicas and something has to agree before a write goes through, it's on the list.
Write the numbers down. How many copies are there, and how many need to be up for it to keep working? For Kafka, that's your replication factor against min.insync.replicas. For a three-node database cluster, it's usually two out of three. Put it in writing, because the next questions lean on it.
Find out where the copies are right now. Ask the running system, not the diagram or the infrastructure code. On Kubernetes, check which node each pod is on and which zone each node sits in. On Kafka, check which broker holds each replica and which rack each broker is in. For managed services, see what the console or API tells you. If it won't tell you, that goes in the notes too.