A recommendation you can trace back to evidence is one you’ll stand behind. One you can’t is one you’ll override — and then you rebuild your own process beside it. These patterns show a person why the machine decided. Not the full math, not the model internals. Just enough of the reasoning to act on it without a stats degree.
Explainability
Turn a black-box output into an auditable argument.
-
Drivers
- Payment history
- weight +0.42
- Account age
- weight +0.27
- Recent inquiries
- weight -0.15
-
Why this output
Explanation drawer
Payment history · Account age · Recent inquiries
-
Output
724
risk score
traced back from the score
A recommendation you can trace, you’ll defend. One you can’t, you’ll quietly rebuild around.
Try it — the pattern, live
The same live demo that runs in the book edition’s field guide. Synthetic data; nothing leaves the page.
1. Feature Importance (What Mattered Most)
After the example, add: 'In production this was the reasoning printed on the plan itself — which audiences, which environments, what trade-off against the brief.' The person reads three reasons, not a black box.
Strengths: Cheap to build. The user can check the work. Repeated exposure trains their intuition.
Weaknesses: It hides feature interactions. When features are correlated, the ranking lies.
2. Counterfactual Reasoning (“What If”)
Show the one input that flips the call. “The system rated this lead high-value. Change [one thing] and it drops to medium-value.” Or: “You’re 3 steps away from the next tier. Here’s what closes the gap.”
A counterfactual lands because the person reads it as a causal story, not a verdict. The decision stops feeling fixed. They know exactly what lever to pull.
Choose it when the user can act: sales (close a deal), education (lift a score), recommendations (steer toward better picks).
3. Comparative Explanation (“How It Compares”)
Set the pick against its rivals. “I recommended Option A. It beats Option B on [dimension 1] and loses to it on [dimension 2].”
A person reads a comparison faster than an absolute score. The trade-off is right there: Option A is riskier with more upside, Option B is the safe call.
4. Reasoning Path (“Step by Step”)
Walk the chain in order. “To predict conversion, I checked industry (SaaS), then company size (50–200), then engagement (high), then landed on 73%.”
It reads the way a human analyst would talk through the case. The person follows the branch, step by step.
Works for: tree-based models, decision trees, rule-based systems — anything with a real path to trace. Breaks for: neural networks. There’s no path to show, and inventing one is hand-waving with a UI.
5. Confidence + Uncertainty Bands
Ship the prediction with its range. “Predicted revenue: $45K (±$15K). Past errors put the real number between $30K–$60K.”
A band is an honest surface. It says: the model’s best guess is $45K, it isn’t 100% sure, and here’s the spread it usually misses by. This one doesn’t explain why — it tells the person how hard to lean.
“Experts don’t have time to read explanations.” Correct — which is why the pattern is three reasons on the card, not a stats lecture behind a link. The depth is the design decision: enough to argue with, never enough to drown in. The users who “had no time” for reasons were the same ones spending hours re-deriving scores the interface refused to explain.
Ship it: the checklist.
- Pick the explanation type from the domain — feature importance, counterfactual, or comparative.
- Compute it: tree models give you importance for free; neural networks need LIME/SHAP; rule-based systems trace the rules.
- Say it in plain words: “Feature X has importance 0.34” becomes “Job title was the strongest signal — 34% of the decision.”
- Watch a real user read it: can they say why the model decided? Do they lean on it more after?
- Don’t fake it: if you can’t follow the model’s logic, don’t write a story that pretends you can.
- Progressive disclosure: top 3 features by default, the rest one click away.
Trade-offs: Accuracy vs. Understandability
| Method | Accuracy | Understandability | Best For |
|---|---|---|---|
| Feature Importance | Partial — hides interactions | Highest | Expert users |
| Counterfactual | Good | High | Action-oriented users |
| Comparative | High | High | Choice / trade-off scenarios |
| Reasoning Path | Exact for tree models | Low | Compliance / critical decisions |
| Full network details | Exact | Near zero | Required by law / research only |
*If tree-based.
Where this pattern earned its keep
- Deal-Screening AI That Cites Its Sources: the surface that let analysts pull a score apart and see why the model ranked a deal.
- An Ad Agency Became the Market’s Aggregator: the reasoning printed on the plan itself — which audiences, which environments, what trade-off against the brief.
One pattern a month. The tradeoffs I paid for, plus the templates I use.
Reasons the user can’t verify or act on become noise — by week three, unread explanation chips are pixels. Rotate to showing reasons only when they’d change the decision (low confidence, high stakes, disagreement), not on every card. Explaining everything equally is the same as explaining nothing.