Spot-Check Sampling: Sized for Real Errors?

Q: Is spot-check sampling adequate oversight for accounts payable?

For a population of low-value, high-volume items it’s often the only arrangement that’s economically possible, and it can be perfectly defensible. Whether it’s adequate depends on whether the sample is drawn along an axis that has some relationship to where errors concentrate, and on whether anyone has stated the error rate the sample is sized to miss.

Q: How should a sample size be set?

Working backward from a tolerable error rate rather than forward from available capacity is the version that produces a statistical claim. Capacity will still constrain the answer in practice, but doing it in that order makes the gap visible, and a documented gap between what you can afford to review and what you’d need to review is considerably more useful than no number at all.

Q: Does stratifying the sample fix the correlated-error problem?

Partly, and only if you stratify on something that predicts error. Stratifying by invoice value is standard and doesn’t help much, since model error tracks layout, vendor, page count, and scan quality instead. Stratifying by vendor or document template is less conventional and closer to the grain of the problem.

Q: What should happen when an unsampled invoice turns out to be wrong?

The instinct is to enlarge the sample. The more informative first move is to establish whether the failure was isolated or a member of a family — pull every invoice sharing that vendor and layout and check them all. That answers a question a larger random sample would take a long time to answer and might never answer cleanly.

If you’ve been following along in this series, you’ll know that we’ve been distinguishing between at least four distinct arrangements commonly subsumed beneath the “human in the loop” label:

approval before action;
exception-only escalation;
spot-check sampling;
post-hoc confirmation.

Last time we worked through exception-only escalation; there, human judgment is exercised in the initial definition of an “exception,” but problems can arise later when the underlying population changes and no one is systematically checking that the definition remains appropriate in the face of evolving needs. Here we’ll take up spot-check sampling, in which an AI agent processes all incoming tasks or tickets, and a person inspects (“spot checks”) some fraction of them. This arrangement has the virtue of stating its own limits outright, since nobody running a 2% sample really believes they are reviewing everything.

To make this more concrete, we’ll illustrate with the example of automating parts of accounts payable workflows. There are different ways this can work, but here’s a sketch of a basic process:

Invoices arrive in a staggering variety of different formats;
Extraction pulls the fields;
A three-way match against purchase order and receipt clears the ones that agree, and what’s left goes to exception handling.

Ignoring the specifics, the important facts are that the volumes are high, the individual items are low-value on their own, and undertaking a comprehensive review was never a viable option.

Below we’ll look at what the arrangement costs, why a clean sample and a clean population are two different things, what the coverage ratio can and can’t tell you when it’s an average, and what the record supports when something clears unsampled.

What does spot-check sampling actually cost to run?

Its cost is statistical, and it is very frequently underspecified.

Two numbers determine whether this arrangement is serving the purpose for which it was intended. The first is the sample size, and the second is the error rate the firm is willing to miss. In most accounts payable (AP) functions, the first number exists, is known to everyone, and was determined by review capacity (which is itself likely calculated from the product of “how many invoices a person can work in a week” and “the number of people we have available”). The second number often doesn’t exist in written form at all, and where a version of it does exist, it is often owned by a different function, written for a different purpose, on a different time horizon.

Without the second number, there’s no way to say whether the first one is adequate, generous, or completely off-base.

This is worth separating from the cost of verification itself. Working a sampled invoice properly is not exactly cheap, as it requires you to re-derive the extraction against the source document, the PO, and the receipt. But observe that the distinctive cost of this arrangement isn’t the per-item work, it’s that the arrangement is generating a statistical claim about a population, and generating that claim correctly requires information that nobody is tracking carefully.

Why can a clean sample come from a population that isn’t clean?

Because sampling arithmetic assumes errors are scattered independently, which is sometimes the case, but it’s also possible for errors to arrive in families because they share an underlying cause.

Classical sampling gives you a rate: draw n items, count the bad ones, project this fraction to all the items you didn’t look at. That inference is sound when each item’s chance of being wrong is roughly independent of every other item’s. Human keying error behaves approximately that way — there’s some rate at which a tired clerk mixes up digits as they type, and these show up more or less at random across the day’s work.

Model error doesn’t necessarily behave that way. For example:

An extraction step that misreads the tax field on one vendor’s layout probably misreads it on every invoice from that vendor;
A prompt or a template that mishandles multi-page invoices likely mishandles all of them;
A model update changes behavior on a date, so errors cluster in time after that date.

In every case, the errors stem from a root cause, which means they arrive together, in the same corner of the population.

Suppose you have 40,000 invoices in a quarter, you sample 200 of them, and it turns out there are three bad invoices among the 200. What should you conclude about everything you didn’t look at? Well, there’s a 1.5% error rate among the invoices sampled, and if you assume uniformity, roughly the same proportion will hold for the rest. But if all three came from a single vendor layout, the error rate might well approach 0% across most of the population and something near 100% within this one cluster.

Sampling a handful of entries and finding nothing, in other words, is weaker evidence than it may appear at first.

What does the coverage ratio tell you here, and what can it hide?

It tells you exactly what it says — the proportion of items a person reviewed — and under this arrangement it’s an average over a population that may or may not be uniform.

The definition doesn’t change. What changes is how much a single figure can carry. An aggregate coverage ratio of 0.5% is consistent with 0.5% everywhere; it’s equally consistent with 20% coverage of invoices above $50,000 and something closer to 0.05% across the long tail of small non-PO invoices, which is where extraction tends to be weakest and where vendor layouts are most kaleidoscopically varied. While the same number may appear in both headlines, they are referring to very different approaches to quality control.

Read stratum by stratum, the same ratio gets considerably more useful, and it starts to answer the question the headline can’t: how much coverage is received by the part of the population where errors actually live?

That reading also surfaces a drift this arrangement can otherwise conceal. Suppose volume grows 30% in the small-vendor tail after a procurement change, and the sample stays at 200 because capacity stayed at 200. The aggregate coverage ratio barely moves, but coverage of the stratum that grew falls by nearly a third. Nothing in the arrangement announces this, and the number everyone watches is the one that didn’t change.

What evidence survives an unsampled item?

Nothing, and — unlike the other three arrangements — that was disclosed from the beginning.

The sampled items leave a good record: a named reviewer, a timestamp, a finding, usually a note. The unsampled items leave a processing record and no human reasoning, because no human reasoning occurred. There’s no arithmetic gap between what someone signed and what someone reviewed, which is a real advantage over the attestation problem we raised earlier in the series; here the signature is explicitly on the sample, not on the population.

So, at least as far as the extent of coverage is concerned, this is the most honest of the four arrangements. It’s also, for exactly that reason, the one where the distance between what’s documented and what’s understood tends to be widest. The coverage number is published, visible, and reported. Almost nobody has read it against a stated tolerable error rate, because that second number usually isn’t written anywhere — so an organization can look directly at an accurate disclosure of its own coverage every month and draw no conclusion from it at all.

The consequence shows up when an unsampled invoice turns out to be wrong — a duplicate payment, a misdirected remittance, a tax field that was wrong for eleven months. The arrangement’s correct answer is that this item was never going to be caught, and that the arrangement is working as designed. That answer is true and nearly impossible to deliver in the room where it’s needed, which is why it tends to get replaced with a promise to increase the sample. Increasing the sample is the right response to a rate that’s too high. It’s close to the wrong response to a cluster, which a bigger random sample finds only a little more reliably and a targeted review of one vendor layout finds immediately.

So here’s what I’d ask anyone running AP this way. What error rate is your sample sized to miss — and if that number isn’t written down anywhere, what would your sample look like if you wrote it down first and sized second? Tell me in the M.A.S.E. Discord, or get in touch directly. We’re researching how these arrangements behave in production, and what comes back shapes what we write next.