The Change Nobody Deployed

The last two issues were about changes somebody shipped: an app update at Lloyds and a config change at Coinbase. Each one had a ticket, a reviewer, and a pipeline, even if the pipeline missed something.

This week's change never touched a pipeline. It was a file that a job rebuilt every five minutes, and as far as anyone was concerned it wasn't a change at all. On 18 November 2025, it took a big chunk of Cloudflare down for most of the morning.

Ask a team what changed in production yesterday, and someone will open the deploy log. Reasonable; that's what it's there for.

But production also reads plenty of things nobody deploys. A job runs, builds a file or a table out of some data and pushes it out, or the servers fetch it themselves every so often. In fintech, the list grows quickly once you start writing it down. Fraud rules get rebuilt overnight from a table the analysts maintain, and sanctions lists refresh from a provider. Card routing tables, FX rates, fee schedules and feature flags all change on their own schedules. Somewhere there's usually an allowlist that a script updates every hour, and nobody quite remembers who wrote the script.

All of these change how production behaves, sometimes more than a code release does. And almost none of them go through change review. There are decent reasons for that. They have to be fresh, they change far too often for anyone to sit and review them, and anyway, it's just data.

They also move quicker than code. Releases tend to go out in stages, with a canary and somebody watching the graphs. A generated file usually gets built once and sent everywhere at the same moment, because freshness is the whole point. And whatever reads it tends to trust it completely. A colleague wrote it, after all, not some stranger on the internet.

Then there are the inputs, which usually belong to someone else. Your job reads a table whose schema and permissions belong to another team. Their change process has no idea your file depends on any of it.

From: "Every change to production goes through review."
To: "We know what production reads that nobody deploys, and what happens when one of those is wrong."

Get an hour or two with someone from each team that owns a service reading generated data. You're after one page.

What does production read that nobody deploys? List everything a job builds that production consumes: files, tables, caches, flags. Put the external feeds on the list too, like the sanctions provider, because their refresh is a change as well. I've never seen this list come out shorter than people expected.

How does it get out? For each item, write down how often it's rebuilt, and whether it arrives everywhere at once or in stages. Then ask an awkward follow-up: would anyone notice a sudden change in its size or shape before production does?

What does the reader do with a bad copy? If you only get through one question in the session, make it this one. Does the service that reads the file check it before using it? Size, row counts, and a quick comparison with yesterday's version would all be cheap. And when the check fails, what happens? Does it keep the last good copy, refuse to start, fall over, or carry on with defaults? For a fraud score or a sanctions list, that choice really doesn't belong to engineering alone. Carrying on with an empty sanctions list lands you in a compliance conversation. Declining every card payment because the fraud rules didn't load is an outage. Somebody senior should pick which of those they'd rather explain. In my experience, nobody has.