👋 Hey {{first_name|there}},
Last week's change was a file nobody deployed. This week's was deployed on purpose, by people trying to protect their customers, with the clock running. It's an awkward one to criticise, and I'll try to be fair about it.
Yes, it's Cloudflare again, three weeks after the outage in #75. I did look for another company, just for variety. But this write-up is so clear, and it follows on from the last one so directly, that it would have been silly to skip it.
Why this matters
Every team has a quick way into production. It might be an emergency change board that can meet in ten minutes, or a feature flag console, a global config system, or a kill switch. Sometimes it's just a hotfix branch that jumps the queue. They're there for good reasons. When a critical vulnerability is being exploited, or a fraud ring is working through your cardholders, nobody wants to wait for next Tuesday's release.
What the quick route leaves out is usually some mix of staging, canaries, and gradual rollout, and occasionally the second reviewer. That's the deal you make. And when the change is the urgent fix itself, people tend to be extra careful to make up for it.
Where they're less careful is everything around the fix. An emergency is hardly ever one change. There's the fix, and then there's whatever you have to do to get it out. Maybe you switch off a check that's in the way, bump a limit, or disable a tool that keeps complaining. All of those feel tiny and obviously safe, so they go out however quickly, and nobody gives them a second look.
In financial services, there's a bit of extra pressure on top. Regulators ask how quickly you patch critical vulnerabilities and how fast you react to fraud. I've never heard of anyone being asked why they patched too quickly.
🧭 The shift
From: "The emergency fix went out carefully."
To: "The emergency fix went out carefully, and so did everything we changed to get it out."
🔍 The lens: the fast-path check
Grab your platform lead and whoever owns incident response for an hour. If you can, bring someone who was actually on call for the last couple of emergencies. They'll remember what really went on, which isn't always what the runbook says.
What are all the quick ways into production? Start listing. You'll probably have the emergency change process, flag and config systems, and kill switches. Add rule consoles for fraud or the WAF, hotfix branches, break-glass access, and the odd direct database update. For each one, write down who's allowed to use it and how long a change takes to reach everything. Some take minutes. Some take seconds, and with those there's no time to notice anything.
What does each one skip? Put it next to your normal pipeline. Is it staged anywhere first? Does it roll out a bit at a time? Is there a health check that can stop it halfway? Is there a tested way to undo it quickly? Skipping some of these is often the right call. The worry is when nobody in the room can say which ones get skipped.