👋 Hey {{first_name|there}},
A new arc starts today, six lessons long.
The last one spent seven weeks on failures that begin somewhere else. A cloud zone overheats, a managed service quietly stops doing its job, a payment processor loses power, and somehow it ends up as your incident. Lesson #72 pulled all of that onto one page.
This time the failure is ours. Someone on the team ships a change, and the damage spreads the same way it did when AWS was the cause. The difference is that there's no provider to point at.
First up, a bank, and a deploy that broke nothing anyone was looking at.
Why this matters
Think about what usually happens after a deploy. Someone watches the error rate and the latency graph for a while, and maybe a synthetic transaction runs. If nothing moves for twenty minutes, the change is done, and people log off.
For plenty of bad changes, that's enough. A crash shows up straight away. So does a timeout, or a config that stops the service from starting.
What it won't catch is a change that keeps answering, fast and without complaint, with somebody else's data.
Every one of those signals asks whether a response came back. None of them asks who the response was meant for. So a page full of another customer's transactions looks exactly like a good page: right status code, 180 milliseconds, nothing to see.
It doesn't behave like an outage afterwards either. You can measure an outage off a graph, in minutes and failed requests. An exposure keeps growing for as long as the defect is live, one customer at a time, and none of it can be rolled back. By the time you find out, all that's left is counting and writing letters.
🧭 The shift
From: "Clean deploy. No errors, latency flat."
To: "Clean deploy, and we checked that every customer still sees only their own data."