The demo went perfectly. Confidence: 94%. Everyone in the room left convinced. Three months later, on real deal data, the same model was wrong four times out of ten. The model hadn't changed. We'd just never designed the part where it's wrong.
The room that left impressed
In 2020 I sat in on a machine-learning team presenting a new model to a private-equity client. It was a clean demo — curated inputs, a confident number, a tidy answer. The client was sold. So was I, for about a week. Then the model met deal data it had never seen, and the 94% became a coin-flip with delusions of certainty. The failure wasn't dramatic. It was quiet, and the people relying on it didn't find out until they'd already acted on a few of the wrong answers.
What stuck with me wasn't that the model was wrong sometimes — every model is. It was that the interface had no language for being wrong. It had been designed entirely around the demo: the happy path, the case where the data is clean and the answer is obvious. The 40% where the model is uncertain, the inputs are missing, or the world has drifted away from the training set — none of that had a screen. And that 40% is exactly where the client's trust either held or evaporated.
Designing the failure state isn't error handling. It's trust recovery. When the system stumbles, the user needs to know why and what to do next — and getting that moment right is what lets them keep using the system at all.
So now I start at the bottom
After that project I changed the order I design in. I no longer start with the case where the model is right. I start with the question almost nobody asks until production forces it: what does this look like when it's wrong? Low confidence, missing data, a user whose situation the model has never seen — I design those screens first, because they're the ones that decide whether anyone trusts the good answers later.
The reflex everywhere else is to treat the unhappy path as a rounding error. "Edge cases are too rare to be worth it." They are rare, right up until the day production hands the system something it's never seen — and by then the trust that took months to build is gone in an afternoon. The happy path is maybe 30% of the real work. The other 70% is the part nobody demos, and it's the part that decides whether a business will run on what you built.
What I believe now
Every intelligent system I design now opens with one question, asked before a single happy-path screen: "What happens when this fails?" Not as a polish pass at the end. As the first design problem, because it's the one the user will actually judge you on.
Want the playbook, not the story? The failure-state patterns I design from — prevention, transparency, recovery — with the do's, don'ts, and worked examples.
See the pattern reference: AI Failure State Patterns →Continue exploring