Last week we wrote about evaluation — how you measure whether an AI system is actually good at the thing you've asked it to do. Evaluation answers a question about the system in aggregate: across a hundred decisions, how many were right? That's a different question from the one that shows up the first time something goes wrong in a specific instance: who approved this?
If the honest answer is "no one, it ran automatically," that's not necessarily a problem. It depends entirely on what "this" was. The mistake most businesses make isn't having too few approval gates or too many — it's not having a clear way to decide which actions need one.
The two failure modes
Put a human in front of every AI decision and you've built a bottleneck. Someone has to read and approve each output before it goes anywhere, which means the system is only as fast as that person's calendar. It also tends to degrade quietly: when the approval queue is long and the system has been right for weeks, people start approving without really reading. The gate still exists on paper. It's stopped doing anything.
Put a human in front of nothing and you've built the other problem — a system taking actions in the world with no one accountable for a specific one of them until a customer, a regulator, or a credit card company flags it after the fact. By then the cost of being wrong has usually already landed.
Neither failure mode is really about how much oversight you have. It's about whether the oversight is sitting in the right place for the kind of decision being made.
Two different jobs for a human to do
There are two distinct things a person can do in relation to an AI decision, and they're not interchangeable.
Sit in the path. The action doesn't happen until a person has looked at it and said yes. This is the only option when the action is hard or expensive to undo — issuing a refund above a threshold, sending a contract out for signature, publishing something under the company's name that a customer or journalist might screenshot.
Review after the fact. The action happens, and a person checks a sample of what happened — or the full set, depending on volume — on some cadence. This works when the action is reversible, cheap to undo, or low-stakes enough that catching a problem a day or a week later costs you about the same as catching it immediately.
The question that decides which of these applies isn't "how good is the AI at this task." A system can be highly accurate and still need a human in the path, if being wrong occasionally is expensive enough. And a system that's merely decent can be safe to run with after-the-fact review, if being wrong is cheap and quick to fix.
What actually determines the answer
A few questions, asked about the specific action rather than the system in general, tend to sort this out:
Can it be undone, and at what cost? An email that goes to the wrong customer is annoying. A payment that goes to the wrong account is a different category of problem. The second one needs a human in the path; the first probably doesn't.
Who finds out, and when? If a mistake surfaces immediately — a customer replies confused, a dashboard shows an anomaly — after-the-fact review can catch it fast enough that the cost stays low. If a mistake can sit undetected for weeks because nothing downstream flags it, you've effectively removed oversight altogether, even if you told yourself you were reviewing it later.
How legible is the decision to someone checking it afterwards? Some actions are easy to audit after the fact — the input, the output, and the reasoning are all sitting there in a log, and a reviewer can tell in seconds whether it was reasonable. Others are harder to reconstruct, especially if the context that justified the decision at the time isn't captured anywhere. If a decision isn't legible after the fact, after-the-fact review isn't actually available to you as an option, whatever the logs claim.
What's the volume? A gate that works fine at ten decisions a day can become the bottleneck at a thousand. Volume doesn't change whether a gate is needed — it changes whether a person-in-the-path gate is operationally sustainable, which is a separate problem that usually gets solved by narrowing what triggers the gate rather than removing it.
Where this breaks down in practice
Two things tend to go wrong even when the framework above is applied properly.
The first is that the gate assignment goes stale. A system that needed a human in the path six months ago, when its error rate was higher, might not need one now — but nobody revisits the decision, because revisiting a working gate feels riskier than leaving it alone. The gate stays in place not because it's still earning its keep, but because removing it is a decision someone has to own.
The second is that pre-decision gates quietly become rubber stamps. This is worth watching for specifically, because it's the failure mode that looks, from the outside, exactly like oversight still working. If the person approving has stopped reading — because the queue is long, because the system has been right for a long run, because the approval screen doesn't show enough context to actually evaluate the decision in the time available — the gate is providing the appearance of a human in the path without the substance of one. At that point it's worth being honest about what's actually happening: either redesign the gate so it's reviewable in the time people actually spend on it, or move that class of decision to after-the-fact review and build a better audit trail instead.
What to do with this
Map the actions your AI systems take by two things: how costly a mistake is, and how quickly a mistake surfaces on its own. Actions that are expensive to undo and slow to surface need a human in the path. Actions that are cheap to undo or fast to surface are usually better served by good logging and a reviewer checking in afterwards — which is less work for everyone and, done properly, catches problems just as reliably.
Then revisit the map. The right place for a gate shifts as the system's error rate changes and as volume grows, and the businesses that get this right tend to treat the gate assignment as something to check periodically, not something to decide once and leave alone.
If you're building out AI systems that take real actions and you're not sure yet where the gates should sit, that's a reasonable thing to work through before you ship rather than after. Get in touch — we're happy to talk through what we've seen work.