TL;DR
- Oversight fails when the approver lacks the information or the standing to refuse. Both are design decisions, not training problems.
- Four patterns: in-the-loop, on-the-loop, over-the-loop, and out-of-band. They differ in when the human acts and what they can still change.
- Approval fatigue is a security control failure, not a user failure. Approving everything trains people to approve everything.
- Gate on irreversibility, not on risk score. What cannot be undone needs a human; what can be undone needs a good log.
- The evidence an assessor wants is refusals. An approval queue with a 100% approval rate is evidence that oversight is not operating.
Why most human oversight is theatre
The typical implementation puts a confirmation dialog in front of a consequential action. It satisfies a policy statement and produces a timestamped record. It rarely produces oversight, for three reasons that are structural rather than cultural.
- The approver cannot evaluate the request. "Agent X requests permission to update 1,240 records. Approve?" contains no basis for a decision. Which records, why, what happens if it is wrong, what does correct look like.
- Refusal has a cost and approval does not. Approving continues the workflow; refusing stops something a colleague is waiting on, with no defined path to resolution. Rational people approve.
- Volume destroys attention. A person approving forty requests a day is not reading the fortieth. Every low-stakes gate you add reduces the effectiveness of the high-stakes ones.
The consequence matters beyond the immediate risk: an organisation with approval theatre has worse evidence than one with none, because it has a record showing a human approved the action that went wrong.
Four patterns
| Pattern | When the human acts | Best for | Failure mode |
|---|---|---|---|
| In-the-loop | Before the action; the agent blocks | Irreversible actions: payments, deletions, external communications, production changes | Fatigue, and latency that pushes teams to widen the auto-approve list |
| On-the-loop | During execution, with the ability to intervene | Long-running or batch operations where progress is observable | Nobody is watching; requires an actual attentive operator to mean anything |
| Over-the-loop | After the fact, by review of a sample or an exception queue | High-volume reversible actions | Review never happens, or happens without authority to reverse |
| Out-of-band | Before the action, through a separate channel | Anything where the agent's own channel may be compromised or manipulated | Approval requests become another notification stream to dismiss |
Out-of-band is the pattern most under-used and the most relevant where prompt injection is a concern, because it takes the approval decision off the channel the attacker controls. CIBA provides a clean mechanism: the authorization server contacts the user directly to approve a specific operation, and the agent never handles the user's credentials.
Choosing where the human goes
The common approach is to gate by risk score, which produces a long list of medium-risk actions requiring approval and therefore produces fatigue. A better primary axis is reversibility.
- Irreversible and externally visible: money moves, data leaves the organisation, a message is sent to a customer, a record is destroyed. In-the-loop, always, and keep this list short enough that it stays meaningful.
- Irreversible and internal: a production configuration change, a schema migration. In-the-loop or on-the-loop depending on whether a rollback exists in practice rather than in principle.
- Reversible and consequential: bulk updates with an undo path. Over-the-loop with a real exception queue and someone with authority to reverse.
- Reversible and routine: no gate. Log the delegation chain and move on.
Reversibility is checkable from your own systems, unlike a risk score, which is a judgement that drifts. It also produces a shorter approval list, which is the point.
Designing an approval that can actually be refused
An approval request should contain enough for a competent person to say no. In practice:
- What will change, concretely. Not a count. A sample of the actual records, the actual message text, the actual amount and recipient.
- Why the agent believes this is correct, in the form of the request that led here and the delegation chain behind it.
- What happens on refusal. If refusing leaves a colleague stuck with no path forward, you have designed a control nobody will use.
- A deadline and a default. The default must be refuse. Timing out into approval converts an unattended queue into automatic authorisation, which is the worst possible failure and a common one.
- An identified approver with the standing to refuse. Routing approvals to whoever is on shift, rather than to someone accountable for the outcome, guarantees rubber-stamping.
Evidencing oversight, which is not the same as evidencing approval
Both the EU AI Act's oversight obligations and an ISO/IEC 42001 management system ask whether oversight was exercised, not whether it was designed. Four artefacts answer that:
- The approval record with approver identity, timestamp, and what was presented to them, not merely that a dialog appeared.
- Refusals. This is the one that matters most. An approval queue with a 100% approval rate over a meaningful period is evidence that the control is not operating, and an assessor will read it that way. Track and report the refusal rate.
- Timeout handling. Records showing what happened to unattended requests, demonstrating the default was refusal.
- Escalation records where an approver was uncertain, which demonstrate the mechanism is used rather than routed around.
Report the refusal rate to the same forum that receives your other agent metrics. It is the cheapest available signal that oversight is real, and its absence is the cheapest available signal that it is not.
Frequently asked questions
What is the difference between human-in-the-loop and human-on-the-loop?
In-the-loop means the agent blocks and cannot proceed until a person approves, which suits irreversible actions such as payments, deletions and external communications. On-the-loop means execution proceeds while a person observes with the ability to intervene, which suits long-running or batch operations where progress is visible. On-the-loop only provides oversight if someone is genuinely watching, which is its main failure mode.
Why does approval fatigue break human oversight?
Because approving is cheap and refusing is expensive. Approval continues the workflow while refusal stops something a colleague is waiting on, often with no defined resolution path, so rational people approve. Volume compounds it: a person handling forty requests a day is not reading the fortieth. Every low-stakes gate added reduces the effectiveness of the high-stakes ones, which is why the approval list should be short.
What should trigger human approval for an agent action?
Irreversibility rather than a risk score. Actions that cannot be undone and are externally visible, such as moving money, sending customer communications, exporting data or destroying records, warrant in-the-loop approval. Reversible actions with a real rollback path are better served by after-the-fact review with an exception queue. Reversibility is checkable from your own systems, whereas a risk score is a judgement that drifts.
What is the most common failure in approval workflow design?
Timing out into approval. When an unattended request is automatically approved after a deadline, the control converts into automatic authorisation precisely when nobody was paying attention. The default on timeout must be refusal. The second most common is presenting a count rather than the actual change, which leaves the approver with no basis to say no.
How do you evidence that human oversight is actually operating?
Through refusals. An approval queue with a 100% approval rate over a meaningful period is evidence that the control is not operating, and an assessor will read it that way. Alongside the refusal rate, keep approval records showing what was presented to the approver rather than merely that a dialog appeared, records of how unattended requests were handled, and escalation records showing the mechanism is used rather than routed around.
How does out-of-band approval help against prompt injection?
It takes the approval decision off the channel the attacker may control. If an agent's input has been manipulated, an approval prompt rendered by that same agent is also suspect. Contacting the user through a separate authenticated channel, as CIBA does, means the approval decision does not depend on the integrity of the agent's own context, and the agent never handles the user's credentials.
Need agent oversight that would survive an assessor?
Human oversight that exists in design and not in evidence is the most common finding we see. HumanAudit tests oversight and risk classification against your real deployments, not your policy documents.