Exception-Only Escalation: When to Worry

Q: Is exception-only escalation human oversight?

It is oversight of the exceptions. Whether that constitutes oversight of the process depends on whether the exception definition is still selecting the right items, which is a separate question that the arrangement does not ask on its own schedule.

Q: How often should an exception definition be revisited?

I have not found a published standard, and the honest answer is that the right cadence depends on how fast the underlying population changes. The more useful reframe is to attach the review to population change rather than to the calendar: an acquisition, a system migration, a new product, or a change in how a field is populated are all events that should trigger a look, and generally do not.

Q: Does changing a threshold require a control change?

Formally, usually not, which is the point raised in the first piece of this series. A threshold adjustment moves the coverage ratio and can move the arrangement from something close to approval before action toward something close to sampling without altering a word of the control description.

Q: Can you get a base rate without sampling the cleared population?

Not a real one. Proxies exist — composition of the exception queue over time, downstream correction rates, aging patterns — and each is weaker and cheaper. They can tell you something has changed; they cannot tell you how much you are missing.

Q: How can Prove AI help with exception-only escalation?

We are interested in what happens to the items nobody looked at, and in whether anything about how a decision got made survives into a system of record where a person can act on it later. Prove AI is trying to make AI work reviewable, and most of what we learn comes from talking to people running these processes.

In the past few entries in our AI in finance series, we’ve argued that there are at least four different arrangements fitting under the label of “human-in-the-loop”:

approval before action;
exception-only escalation;
spot-check sampling;
post-hoc confirmation.

Last time, we spent a fair bit of time on the approval-before-action arrangement. Here, we’ll explore exception-only escalation, in which the coverage ratio is readable, movable, and difficult to interpret on its own. The basic idea is that the system clears whatever matches a set of approval rules or a schema, and routes anything that doesn’t to a human; on a setup of this kind, the only thing the analyst sees is the exceptions. That fraction is the coverage ratio in its most legible form, but knowing it tells you what proportion of items a person reviewed without helping you understand whether that was the right proportion. Of the four human-in-the-loop arrangements, this is the one where the label is most nominally true and arguably the least operative.

Reconciliation — comparing subledger to general ledger, bank statement to cash account, custodian position to internal record, and ensuring everything is in agreement — is a routine, high-volume part of financial workflows, and it’s a place where the exception-only escalation arrangement slots in naturally. It’s also a pattern that predates AI by decades, offering a chance for us to readily glimpse the challenges facing finance teams being pressured to use AI to do more with less. It’s far from straightforward to bolt an AI agent onto a longstanding process, especially if it’s being supported by legacy infrastructure or built to accommodate nuances and edge cases not easily handled by messing around with a prompt.

Below, we’ll examine the ins and outs of exception-only escalation, offering the context you need to figure out whether this is an approach that makes sense for your particular tasks.

Where did the judgment go?

Into the exception definition, which is usually specified once at implementation and only rarely revisited by anyone.

This is the substantive difference separating exception-only escalation from approval before action. In the latter, the analyst’s expertise is applied to every item and the question is how deeply they ultimately look. In the former, the analyst’s expertise is applied only to items the definition selected, and the definition decides which items get expertise applied to them at all. This doesn’t mean human judgment disappeared, it means it moved upstream, into the initial configuration.

The people who wrote that configuration were solving a specific problem at a specific moment: making the exception queue tractable for the staff available at go-live. Tolerance bands, confidence thresholds, match-rule hierarchies, and the treatment of aging items were all set against that constraint.

Those are perfectly reasonable choices and I am not suggesting otherwise, but there’s a lingering issue with respect to what happens to them afterward (which is usually nothing). The definition has no owner in the way a policy has an owner, no review cadence in the way a model has a review cadence under SR 11-7, and no natural trigger that says it is time to look again.

Cost follows the same shape. It is front-loaded into the configuration work and near-invisible thereafter, which makes the arrangement look cheap on a run-rate basis. The recurring cost that would keep it honest and effective is the one nobody budgets (a fact we will discuss more shortly).

Why can a falling escalation rate look like improvement?

Because the population drifts while the definition holds still, and both a genuine improvement and drift can push the escalation rate down.

For a bevy of different reasons, the population of interest is almost certainly not stable. Consider situations like:

New counterparties arriving;
An acquisition adding entities with their own idiosyncratic conventions;
A payment rail changing how a reference field is populated;
An ERP migration altering the format of a description string;
The close calendar shifting and a cutoff that used to catch timing differences no longer doing so;
A product launch introducing a transaction type nobody wrote a rule for.

Each of these changes the shape of the items flowing through the system, but none of them changes the rule that decides which items are anomalous (unless someone notices and takes the initiative to update the original specification).

What you observe when this happens looks exactly like what you observe when the model gets better at matching: the escalation rate falls, the exception queue clears faster, the team feels less pressure at close. Every operational signal points the same direction, and the direction is the one everybody was hoping for, but that doesn’t make it correct. The following table shows different ways in which signals can correspond to changes in the underlying process:

What you observe	If the model improved	If the definition no longer suffices	What would separate them
Escalation rate falls	Fewer genuine anomalies reaching the queue	Genuine anomalies clearing silently	A worked sample of cleared items
Auto-match rate rises	Better matching on the same population	Same matching on a changed population	Tracking population composition over time
Exception queue clears faster	Fewer hard items	Hard items no longer being identified	Difficulty mix of what does escalate
Downstream adjustments flat	Nothing is escaping	Escapes have not surfaced yet	Time, and a detection path that exists

The fourth row is the one that most often gets treated as confirmation, and it is the weakest of the four. A flat rate of downstream corrections is consistent with nothing escaping, sure, but it’s equally consistent with escapes that have not yet been found — and reconciliation breaks in particular have long and irregular detection lags.

Firms, by and large, are not instrumented to notice problems of this sort: KPMG’s 2026 finance survey of just over a thousand senior finance leaders found that only 29% formally track where AI adoption fails. A firm that does not track its failures has no mechanism that would surface the second reading of the table above (“errors are making it through undetected”), and will take the first one (“there are fewer errors”) by default.

What would it take to know the escalation rate is right?

A base rate, which means deliberately working a sample of the items the system cleared.

The escalation rate is a numerator without a denominator. It tells you how many items were routed to a person; it cannot tell you how many items should have been. The only way to get the second number is to pull a sample of the cleared population — items the system disposed of without escalation — and have an experienced person work them as if they had escalated, then count how many turn out to have warranted attention.

As is often the case, this is easier said than done. It requires you to lean on the scarcest people you have, since a junior reviewer working a cleared item that looks clean will probably give it a once-over and conclude it’s clean. It produces no throughput; the work has already been done once. Its expected yield is close to zero, which is the point and also the reason it is impossible to defend in a budget cycle against anything with a return. And when it does find something, the finding is that the configuration has been wrong for some period, which nobody is looking forward to. Every incentive around this exercise pushes against performing it.

There is a partial substitute worth noting, which is watching the composition of what escalates rather than the raw count. If the exception queue is increasingly made up of the same few rule triggers while the business has visibly changed, that is a signal the definition is catching a narrowing slice of what it was meant to catch. It is weaker than a worked sample and it costs almost nothing, which is a trade some firms will take.

What evidence survives a non-escalation?

Very little, and the record is biased toward the fraction that was reviewed.

An escalated item leaves a good artifact: a person looked at it, formed a view, wrote something down, and closed it, with a name and a timestamp attached. A cleared item leaves a match record and a disposition and no reasoning, because no reasoning occurred. This means the documented portion of the process is precisely the portion where a human was involved, and the undocumented portion is everything else, which is nearly all of it.

The consequence for anyone reviewing the arrangement after the fact is that the available evidence is unrepresentative by construction. An examiner or an internal auditor pulling a sample of exceptions is sampling the best-documented and least representative slice of the population, and will generally find it in good order, because it is. The question the evidence cannot settle is what was in the other ninety-something percent.

Set this against the previous piece, and the pair is instructive. Under approval-before-action, the record establishes by whom and when an approval is offered, and is silent on the depth of analysis that underwrites that approval. Under exception-only escalation, the record is complete for the items a person saw and silent about the population they didn’t. Both arrangements produce evidence that is entirely truthful and answers a different question than the one being asked.

So here is what I would want to know from anyone running reconciliation this way. When your escalation rate last fell, did anyone establish which of the two readings applied — and if the answer is that nobody checked, what would it have cost to check? Tell me in the M.A.S.E. Discord, or get in touch directly. We are researching how these arrangements behave in production, and the answers shape what we write next.