How Financial Professionals Use AI, Part 2: What Does It Cost to Be Sure?

When it comes to maximizing AI ROI, correct output is necessary but nowhere near sufficient on its own. What’s missing is verification. Before an AI-drafted number or paragraph goes into a deliverable that someone puts their name on, two things have to happen:

1. Somebody has to establish that it’s right;
2. Somebody has to leave a record showing how they established its correctness.

Both procedures cost time, and neither cost is the sort of thing that shows up in a slick demo or in benchmarking.

To illustrate, let’s take the example of a credit memo written up for a mid-market borrower. The tool will probably return a first draft in ninety seconds — borrower overview, three years of trend commentary, covenant analysis, a recommendation — and it reads well.

But before its contents can be trusted, several hours have to be invested to iterate and check everything. This is necessary not because any particular part of it was badly wrong, but because the reviewer’s troubleshooting had to involve things like:

• Opening the financials and re-deriving the leverage calculation;
• Chasing two confident assertions about customer concentration that turned out to be reasonable inferences from a footnote rather than statements in the document;
• Rewriting the covenant paragraph after working out which of the borrower’s two EBITDA definitions the memo had silently blended.

Ninety seconds of production, hours of verification, and the draft was mostly right. Scenarios like this play out for many teams working on AI in finance, and the main cost isn’t errors; it’s not knowing in advance where the errors are, which is a cost you pay in full on both clean and dirty drafts alike.

Our first post on AI in finance was about output that’s wrong in ways that evade standard AI metrics and that nobody catches. This one is about output that’s right, and what it costs to find that out and then prove it — the price of checking, and the price of producing documentation around the check.

Why doesn’t AI save the time it appears to save?

AI-driven time savings aren’t always as extensive as they appear because the time saved in production can be offset by the time spent on verification, and the ratio between them varies more as a function of the shape of the task than the total length of the output.

In this analysis, there are two properties doing most of the work. The first is whether the AI’s claim points at something, like a page, a spreadsheet cell, a filing, or a line in a contract that can be used for quick verification. The second is whether the reasoning the model used to produce the claim can be recovered from the claim itself. When both hold, checking is a lookup: open the source, compare, move on. The check takes a consistent number of seconds (the same few seconds on the fortieth item as on the first), and it can be done by anyone who can read the source. When neither holds, checking means rebuilding the path that produced the output — which is a version of the work the tool was supposed to have done, performed by someone who now has to do it while arguing with a draft that has already committed to an answer (more on this shortly).

The practical consequence is that two tasks can appear to save the same number of hours while actually netting out in opposite directions. Having AI agents pull forty lease terms out of forty contracts into a schedule saves real time and costs little in the way of troubleshooting and investigation, because each extracted term carries the context you need to verify it. Drafting, say, the variance commentary for a monthly close might appear to save an equivalent number of hours while costing more time than it saves, because the only way to know whether the AI agent is correct in inferring that a decline in gross margins is driving changes is to go and look at the underlying details.

This pattern isn’t specific to finance; the clearest evidence we have of this dynamic playing out comes from software and consulting. Controlled studies have found large productivity gains on tasks that:

• Are self-contained;
• Relatively easy to lay out in concrete, step-by-step detail;
• Can be checked automatically.

There are much smaller (or sometimes negative) productivity gains on tasks that:

• Are open-ended;
• Nebulous, or otherwise hard to describe in a procedural way;
• Require a human to judge whether the result is right.

In one METR study of experienced developers working in their own repositories, participants expected AI assistance to cut their completion time, and it did the opposite. What’s more, the same participants, afterward, still believed they had been sped up.

Now, in fairness, the authors of that study have since pointed out that it suffers from certain potential methodological flaws which might lead it to underestimate AI-driven productivity boosts, so we shouldn’t treat it as gospel. Still, the idea that “AI agents are great on tightly scoped work but have more ambiguous benefits for open-ended tasks requiring a lot of human insight” accords with the more anecdotal experiences of many, including mine, and probably yours as well. You are welcome to head over to the M.A.S.E. Discord and tell me directly if you think otherwise.

Here, we’re inferring a similar structure operating in finance, but it’s reasonable to conclude that the mechanism transfers, because it isn’t about code: where correctness is established by an automatic check, verification is nearly free, and where correctness is established by a person, verification is the biggest part of the job.

There’s a second cost inside the check, and it’s less obvious. Verification isn’t a neutral inspection performed on an inert document — with a conversational tool it’s an exchange, and the thing being inspected answers back. A Harvard Business School study of consultants attempting to validate model output found that the more they fact-checked and pushed back, the harder the model worked to persuade them to accept what it had originally said; it would apologize (“You’re absolutely right!”), correct itself (“That’s on me”), and then restate the same position supported by additional data and structured comparisons that made the flawed version look better grounded than before. This is another manifestation of the ‘misleading precision’ problem laid out in part one. It’s not just that an incorrect, precise, well-formatted figure looks right on the page; it’s that this wrong figure also becomes harder to dislodge during the process of questioning it. The reviewer who challenges a number and receives a longer, better-organized, more thoroughly sourced version of the same number has been given more to check, not less.

Which is why the question that actually determines whether a tool gets used isn’t whether its output is right. It’s a question of reliability and how long it takes to know — and whether the reviewer can tell, before starting, which kind of task they’re in.

Can you explain how the AI in finance model got its answer — and is that what’s being asked?

Usually not, and the gap between the two questions is where projects often stall.

A note on terms, because this section turns on a word that means two things to two rooms. Here, financial model means a workbook — three statements, a DCF, a covenant schedule. AI model (and similar terms) means the system generating the output. When someone in a review meeting says “we can’t use it because we can’t explain the model,” it’s worth establishing which one they mean, because they’re often not sure themselves, and the answer determines whether the problem is hard or merely tedious.

Read as a question about the AI model, the demand is close to unanswerable. As things stand, there are no troubleshooting tools, AI dashboards, or guardrails that can tell you why a particular token sequence emerged from a particular set of weights in a form that survives a review meeting, and waiting for mechanistic interpretability research to make that possible could mean waiting indefinitely. Read as a question about the workflow itself, it’s ordinary. Audit documentation standards have long required that the file contain enough for an experienced person with no previous connection to the engagement to understand what was done, what evidence was obtained, what was concluded, and who performed and reviewed the work and when — and that it demonstrates the work was in fact performed. A spoken explanation from the person who did the work isn’t adequate support on its own; the record has to stand without them.

None of that asks anyone to explain a cognitive process. It asks for a paper trail, for documentation. Applied to automated workflows, the fields are unglamorous and entirely obtainable: which documents were in scope, which passages the output actually rests on, which version of the tool and which instructions produced it, what the reviewer changed and why, and who accepted the result. That’s more of a logging problem. It’s solved by writing things down at the moment they happen rather than reconstructing them under inspection three quarters later.

The reason it doesn’t get solved is that almost none of it is captured by default. Return to the credit memo. Three hours of verification produced real evidentiary work — a re-derived leverage calculation, two claims traced back to a footnote and downgraded from assertion to inference, a covenant paragraph rewritten once the EBITDA definitions were untangled. Almost all of that lives in the reviewer’s head and in a final document that looks exactly like the ninety-second draft. The checking happened, but the record of it didn’t. And a memo that was thoroughly verified is indistinguishable, six months later, from one that was skimmed.

So the constraint isn’t interpretability, and it usually isn’t policy. It’s that the automated workflows produce a deliverable without producing evidence that anyone stood behind it.

What happens when this scales?

By and large, both costs accrue on a per-item basis. The check doesn’t get cheaper on the hundredth memo, because each one rests on different documents and a different set of judgments. Production cost, meanwhile, falls with volume in the way software costs usually do, and the two verification costs which make output usable are the two that don’t amortize cleanly.

The open question

In the research we’ve done at Prove AI, most of the practitioners who use these tools well have converged on a similar discipline: keep the tool on tasks where the output carries its own address, treat synthesis as a starting point rather than a draft, and verify anything that will be relied on. This practice is sound, expensive, and largely undocumented, which means the organization can’t see it, can’t audit it, and can’t tell a careful reviewer from a fast one.

That’s where this stops and part three starts, because both costs eventually land somewhere specific. They land on a named person who signs, in a regulatory frame that (at least at present) doesn’t much care how the work was produced. And they land inside systems that were built long before any of this and weren’t designed to record it.

So the question for anyone running one of these workflows now: when your team verifies AI-assisted work, where does that verification actually live — and if an inspector asked next year, what would you show them? Reach out to let us know how you’re handling these nuances.

Frequently asked questions