The demo went perfectly. Everyone in the room left convinced. Three months later, on real deal data, the answers stopped holding. The model hadn’t changed. We’d just never designed the part where it’s wrong.
The room that left impressed
In 2020 I sat in on a machine-learning team presenting a new model to a private-equity client. It was a clean demo — curated inputs, a confident number, a tidy answer. The client was sold. So was I, for about a week. Then the model met deal data it had never seen, and that confident number became a coin-flip with delusions of certainty. The failure wasn’t dramatic.
What stuck with me wasn’t that the model was wrong sometimes — every model is. It was that the interface had no language for being wrong. It had been designed entirely around the demo: the happy path, the case where the data is clean and the answer is obvious. The cases where the model is uncertain, the inputs are missing, or the world has drifted away from the training set (what a data scientist would call distribution shift) — none of that had a screen. And that is exactly where the client’s trust either held or evaporated.
Designing the failure state isn’t error handling. It’s trust recovery. When the system stumbles, the user needs to know why and what to do next. Get that moment wrong and the good answers stop mattering..
So now I start at the bottom
After that project I changed the order I design in. I no longer start with the case where the model is right. I start with the question almost nobody asks until production forces it: what does this look like when it’s wrong? Low confidence, missing data, a user whose situation the model has never seen — I design those screens first, because they’re the ones that decide whether anyone trusts the good answers later. The patterns library names this the capability contract: what the system is for, where it taps out, and what it hands back to a human. I draw those edges before any happy path.
The reflex everywhere else is to treat the unhappy path as a rounding error. “Edge cases are too rare to be worth it.” They are rare, right up until the day production hands the system something it’s never seen — and by then the trust that took months to build is gone in an afternoon. The happy path is the smaller half of the real work. The rest is the part nobody demos, and it’s the part that decides whether a business will run on what you built.
What I believe now
Every intelligent system I design now opens with one question, asked before a single happy-path screen: “What happens when this fails?” Not as a polish pass at the end. As the first design problem, because it’s the one the user will actually judge you on.
Want the playbook, not the story? The failure-state patterns I design from — prevention, transparency, recovery — with the do’s, don’ts, and worked examples.
See the pattern reference: AI Failure State Patterns →Continue exploring