The Reversibility Gamble
Two things I keep saying out loud at work, to different rooms.
On a product feature: we should prioritise small changes going live over a big change that’s “right”. Nobody knows what’s going to work until it’s in front of users.
On a database upgrade: I could have done it in place, in an afternoon. Instead I did what was “right”: a parallel cluster, backups and restores, the coordination layer, a shadow of production traffic, fixing all the issues beforehand — and cut over days later. It cost real money in duplicated infrastructure, and I signed off on that.
Read together, those sound like two different engineers. One is impatient with planning. The other built a parallel copy of production rather than run one command.
On the surface, it may sound like the cautious one simply cared less about the product work. But in reality both changes mattered — and mattering more would only ever have explained the caution, never the impatience.
What explains both is reversibility — an undo button.
You have it already: CTRL+Z, and the last thing you did is gone. It’s why you’ll delete a paragraph you spent an hour writing. And why nobody convenes a review to rename a file. We read that casualness as confidence. In hindsight, it’s mostly the button.
Kent Beck put the same idea in the most quotable form I know: the difference between a bad haircut and a bad tattoo is that the haircut grows out.
Most of the time, when we call a decision risky, what we mean is that we cannot picture the way back. Hold on to that phrase: “the way back”.
The button#
Software’s undo button was manufactured, deliberately, over years, and what strikes me is how consistent the shape is across the whole toolkit. A feature flag turns a redesign into something you can switch off from a phone on a train. Blue-green gives you a router flip where you used to have a restore. Expand and contract takes one schema change nobody can walk back and turns it into three that anyone can.
John Allspaw made the underlying case in 2010: being able to recover quickly from failure is more important than having failures less often. Seven years later he went back and said the statistical framing hadn’t held up — averaging incidents is not arithmetic anyone should have accepted, and that he’d have dropped the word mean from the title.
That mindset — recover quickly rather than fail less often — is the default now, and I think it’s right. When something looks irreversible, the productive response is to go hard after the irreversibility.
Why the button beats arguing#
Product work is the best case for the whole idea, because there the button ships in the box. You put the feature out, you watch it for a week, and if it’s bad you switch it off and almost nobody remembers.
And there’s a study that shows why this cheap reversibility is worth so much. Microsoft’s experimentation team once ran a challenge inside the company. They took eight finished A/B tests, turned each one into a three-way question — A won, B won, or no statistically significant difference between them — and offered a nice polo shirt to anyone who could get six of the eight right. Over two hundred people answered.
They handed out no shirts. Six people got five right, nobody got six, and the average was 2.3. With three options a question, blind guessing scores 2.67. The write-up’s own verdict on that is one word: “Humbling!”
That result is not about product builders being wrong. It measures how much we can know about a change before it meets real users. Being closest to something is not the same as being able to call it. So when the choice is between the whole flow, designed properly, a month of work, and a thin version in a week, arguing it out means trying to settle which version users would prefer — and that is the one question the shirt survey says none of us can answer. What can be settled in a room is not which version wins, but what it costs to be wrong about it.
A meticulous design costs a month and leaves you no more certain than a test that runs on Thursday. Where reversal is cheap, it beats the expensive argument every time it’s on offer.
The gamble#
So the industry agrees that what a risky change needs is a way back, and the industry is right. But this wisdom has three gotchas:
- short tests mislead about anything slow
- the undo patterns stop at your data
- we assume version upgrades are backwards compatible
The first gotcha is well lit, because the ship-small people drew it themselves. Google built machinery specifically to measure what users learn over months rather than days — their standard studies run about ninety days — on the stated grounds that the short-term effect is not always predictive of the long-term one. That work drove a sequence of launches which cut the ad load on mobile search by half. The paper calls it “a substantially short-term revenue negative change.” Long-run revenue came out neutral and the user-experience metrics moved sharply — but no two-week test gets you to that decision on its own.
So the honest claim is narrower than the slogan I keep saying in rooms. Shipping small tells you which of the things you already thought of is better. It won’t tell you that you were choosing from the wrong set, and it will mislead you cheerfully about anything whose effect accrues slowly.
The second gotcha took me years to notice, for an ordinary reason: the undo toolkit — flags, blue-green, expand and contract — kept working, and nobody rereads the fine print of something that works. That’s where this gotcha lives. Read any of the patterns’ write-ups to the end and they get polite the moment they reach your data. Fowler’s own blue-green post names the database as the awkward part, particularly when the schema has to change, and hands the problem back to you: make the schema work for both versions, a different technique carrying a different bill. That’s not a failure of nerve. The undo button is a solved problem for anything you can throw away and rebuild, and data is the part you cannot. So data gets its own playbook, the careful one everybody agrees on: run both, compare, cut over.
The third gotcha is different in kind. It isn’t a place where the wisdom stops working — it’s an assumption we make without noticing: that version upgrades are backwards compatible, that the version you came from can still run against whatever the new one leaves behind. You can see the assumption in how differently we treat two moves of the same data. Moving between systems gets the full run-both treatment, but moving between major versions of the same engine gets a backup, a maintenance window and a rolling restart. Nobody argues for the exemption out loud — it rests entirely on the assumption holding.
My own upgrade — the ClickHouse one from before — is the perfect example of the third gotcha. The upgrade documentation tells you to roll through the replicas and never let every replica of a shard be offline at the same time; the rest is convention. And on rollback, the same page says you may be able to downgrade, “if you have not started to use any of the new features.” May — conditional on something that is hard to know and easy to get wrong.
Then I read a filed issue about a coordination layer that doesn’t come back after a major-version jump. It can’t read its own log on restart, records are lost, and recovery is manual: clear the coordination state, then restore the replicas table by table. The part that keeps the replicas agreeing about what has happened — the part a rollback runs on — is the part that fails. One filed issue doesn’t tell you how often that happens; it tells you the failure and the escape route can share a component. In place, with a backup stops being the cheap option and becomes a wager that the button will still be there when I reach for it — a button I have never once pressed on this cluster, with this data already written to disk.
That’s the reversibility gamble, and the whole industry takes it: every flag shipped and never switched off in anger, every restore that has never actually been run, every migration called reversible because the runbook said so. We price the cost of being wrong carefully, but we price the condition of the way back almost never. Mostly we win anyway, and cheap reversal remains the best deal available in this industry; I’d take it every time. But it is a bet, some changes move its odds, and the ones that do are the ones whose failure goes after the way back.
Swap the question#
If the gamble has a fix, it isn’t more caution. It’s a swap of questions at the moment of deciding: stop leading with how important is this, and lead with is my way back conditional, and on what exactly?
I know what the first question is worth, because I’ve watched it be confidently wrong. I once had a cache tier in a monolith I was fairly sure was load-bearing. It was old and thinly documented, and nobody was keen to be the person who touched it — the kind of component that acquires respect simply by having survived. Taking it out felt like a decision that deserved a quarter, a rollback plan and somebody senior nodding, and all the fancy meetings.
But when we asked how much work it was actually absorbing, the answer turned out to be about 0.6% of cluster CPU. We killed it that week and nothing happened. Cases like this are classic examples of how bad our biases are.
The question asked bare: how important is this, with nothing to measure against, returns very for everything, because everyone believes their work matters and I’m no exception. It only became answerable when someone gave it a reference: 0.6% of cluster CPU is an importance you can act on. The reversibility question needs no reference at all. Asked of any change, it comes back with a different answer, which is the least you should demand of a question.
Put the reversibility question to my two statements from the top and the department turns out to be a coincidence. What actually differed was whether the button was there. The feature had one in the box. The upgrade had a may in the manual.
The second cluster is what acting on the answer looks like. Seen that way it’s procurement: blue-green for a thing that doesn’t come with blue-green, a router flip instead of a restore, bought at the price of running two of everything for a few days.
The shadow traffic was the same purchase. A load test uses the traffic you imagined, and a shadow uses the traffic you have, with its real query shapes, its skew, its one row in ten million nobody would have thought to write down. Shadowing is aimed at the distribution rather than the logic.
The cutover was uneventful and the parallel cluster found nothing that was a proper issue.
A rehearsal buys less than it feels like it buys. Complex systems fail in combinations nobody enumerated, and a rehearsal only covers the failures you were able to imagine well enough to stage. And I’d be wrong about all of it if reversal turns out to have been solid all along. If the downgrade path holds under real conditions, if the restore is fast and verified and routine, then in place is correct and the second cluster was theatre.
I’d spend the money again, because I was buying an option rather than information. You buy an option before you know whether you’ll need it, and the good outcome is the one where you never use it. Nobody calls a year without a claim a wasted insurance premium.
None of this has made me more careful, only more specific about what I check, which is cheaper. How important is this, asked bare, has never told me anything I could act on — it said the cache tier deserved a quarter, and a measurement said otherwise. So I’ve stopped leading with it, and started leading with the other one.
Is my way back conditional, and on what exactly?