An override is the user telling you what the training data missed. Treat it as noise and you throw the lesson away. Treat it as signal and the next version arrives smarter. Every pattern here keeps the person in command when the stakes outgrow the model’s confidence — and logs what they do about it.
An override is not defiance. It is the training data the next version needs.
Try it — the pattern, live
The same live demo that runs in the book edition’s field guide. Synthetic data; nothing leaves the page.
1. The Approve / Reject / Flag Interface
The model proposes. The user picks one of three: approve and execute, reject and discard, or flag for later. This is the simplest human-in-the-loop flow there is.
The model carries the load — pulling the data, running the inference. The person carries the judgment: read the recommendation, then bless it or question it.
Choose it for high-volume, low-friction calls — content moderation, fraud detection, segmentation. Where it bit me: people click through without reading. On anything high-stakes, put a confirmation between the click and the consequence.
2. The Editable Recommendation
The model hands over a draft — an email, a document, a decision. The person edits it before it ships. “AI suggested this email subject. You modified it. Predicted open rate increased from 22% to 25%. Ship it?”
A draft you can edit respects the expertise you brought to the screen. You verify the model’s thinking while you improve it. After a few rounds you know its blind spots, and you edit for them on sight.
Choose it for personalised content like email, drafting like documents and code, or creative output like design suggestions.
3. The Expert Override
The model recommends action X at high confidence. The user overrides with action Y. The system logs the swap and asks why. “You chose Y instead of X. Why?” → the user explains → the model has a new lesson.
An override is not a failure. It is the user pointing at something the model couldn’t see. Track the overrides and the patterns surface: “90% of the time users override Prediction B, they cite Domain Concern X. Add that to the model.”
Choose it when you want the model to get better in production. Log the override, name the missing feature, retrain, repeat.
4. The Escalation Ladder
Clear the top confidence threshold and the system auto-approves. Fall below it and the call goes to a human. Fall below a second, lower line and it goes to a senior one. Risk earns review; certainty earns speed.
| Confidence | Action | Speed |
|---|---|---|
| 90%+ | Auto-approve | Instant |
| 70–89% | Queue for analyst review | 1–4 hours |
| 50–69% | Queue for senior review | 4–24 hours |
| <50% | Reject or request more data | N/A |
It scales without straining: confident calls clear instantly, doubtful ones get human eyes in proportion to what they could cost.
5. The Feedback Loop (Learn From Corrections)
Every correction a person makes — a reject, an edit, a line of explanation — is a small lesson the model hasn’t learned yet. Catch it the moment it happens, and once a month let the month’s worth of lessons retrain the thing.
“In May, you corrected 120 recommendations. Top theme: you rejected deals in healthcare when industry was <2 years old. We’ve added 'company age’ as a feature. Accuracy improved from 81% to 85% in June.”
The loop pays off: the model sharpens, the user trusts it more, the trust buys more feedback, and the feedback sharpens it again.
“Letting users overrule the model undermines it.” That was the room’s position at the UK’s largest out-of-home advertiser, and it dissolved the first time a buyer watched their own override sharpen the following week’s recommendations. An unchallengeable system doesn’t read as authoritative — it reads as unaccountable, and experts decline to stake their name on the unaccountable.
Ship it: the checklist.
- Name the human’s role: verifier who approves or rejects, editor who modifies, or expert who overrides with a reason?
- Draw the decision boundary: what does the model decide alone, what needs a review, what needs an expert?
- Log every decision: the recommendation, the human’s call, the confidence, the reasoning.
- Read the divergence: weekly, see where humans override; monthly, find the patterns; quarterly, ship the fixes and retrain.
- Close the loop out loud: tell users “Your feedback improved accuracy by X%.”
- Guard against approval fatigue: don’t make a person rubber-stamp 100 low-risk items a day.
Trade-offs: Automation vs. Control
| Approach | Speed | Trust | Best For |
|---|---|---|---|
| Full automation | Instant | Low | Low-stakes, internal |
| Approve/Reject | Fast | High | High-volume, moderate-stakes |
| Editable | Medium | Very high | Creative, personalisation |
| Expert review | Slow | Very high | Critical decisions |
Where this pattern earned its keep
- Deal-Screening AI That Cites Its Sources: analysts read the model’s scores and override them with domain expertise.
- An Ad Agency Became the Market’s Aggregator: media buyers approve or reject the algorithm’s calls, and the rejections train the next ones.
- OrgOS: peer review and feedback loops running at the scale of a whole company.
One pattern a month, with the tradeoffs I paid for and the templates I use.
Watch the override rate: near zero means the loop is theater and the human is a rubber stamp — retire the review step or raise its stakes. Under volume pressure, review queues decay into approve-all; if you can’t staff the loop honestly, narrow what enters it instead of pretending.