👋 Hey {{first_name|there}},
In #74 we watched a routine Coinbase config change knock over a shared gateway. I mentioned then that the postmortem had a second half, about how they got back. This is that half.
There isn’t much drama in it, to be honest. Everyone knew what the fix was within minutes. The trouble was getting to where the fix could be applied.
Why this matters
Ask a team how they’d undo a bad deploy and the answer comes straight back. Roll it back through the pipeline, flip the flag, redeploy the last version. It’s step one in nearly every runbook I’ve read, and most of the time it’s correct.
What almost nobody writes down is what the rollback itself needs to work. An engineer has to get from their laptop to the tool that does the rolling back. On the way, they go through the VPN, then single sign-on. Maybe there’s a bastion host or an ingress gateway. Then they reach the deploy tool, its runners and the place the images come from, and finally whatever API actually applies the change. Every one of those has to be up at exactly the moment you need it, which is also the moment something has just gone wrong.
Usually that’s fine, because whatever broke is somewhere else. The ugly case is when the change breaks one of those hops. Then the bad change and the means of undoing it go down together. You know precisely what to do, and you can’t reach the place to do it.
There’s normally a plan B, some break-glass route with privileged access handed out just in time. In financial services, that route is meant to be awkward. Auditors and regulators want privileged access to production locked down, approved, and logged, and they’re right to. The side effect is that the backup route is slow on purpose. Very few teams know how slow.
🧭 The shift
From: “If it goes wrong, we’ll roll it back.”
To: “If it goes wrong, we know what the rollback needs, which bits of that the change could break, and how long the other route takes.”