From the Prove AI Team
Engineering depth, product thinking, and field notes from building the debugging layer for GenAI pipelines.

Nine out of ten AI agents in a recent study found their way around a spending limit meant to keep a human in charge. An approval step isn’t the same as control.
Latest
AI agents are getting more responsibility at work. But what happens when they leave problems behind that nobody sees coming?
A wrong number usually looks wrong. A made-up one ties, fits its neighbors, and passes the check you run by eye. So what is that check telling you now?
Reviewing AI-assisted edits to a financial model usually means checking the edit. The changes most likely to slip through are the ones nobody asked for, sitting where nobody thinks to look.
Checking a financial model is something people were trained to do, long before AI arrived. The checks still hold up. What changed is something the discipline once took for granted.
An AI model can return dozens of edits to a financial model in a single pass, and every one of them shows up at the same size. What runs out first in review is attention, and a flat list of changes gives you no help spending it.
AI models can build a working financial model in the time it used to take to set up the tabs, and the hours saved are real. But they don’t seem to have come back, and I think it’s worth asking where they went.
Most AI performance numbers in finance are measurements of agreement between a model and a reviewer, presented as measurements of correctness. Four questions separate the two, and they apply as much to your own internal reporting as to anything arriving from outside.
In this series, every approach to AI in finance assumes somebody could still do the work manually, without the model, if the need should arise. That assumption is about an underlying capability, and finance already has a discipline for maintaining and testing exactly this kind of capability — it just isn’t one most firms have started applying to AI.
AI agents don’t just make mistakes — they can waste time, money, and context getting there.
Post-hoc confirmation is the AI-based, human-in-the-loop arrangement whose name is most misleading: by the time a human confirms the output of a finance workflow, it has already been posted, aggregated, and reported, and the person confirming it knows what changing it would cost. The check is real, and it happens at the moment it is most expensive to fail.
AI agents are moving into production faster than teams can understand what happens when they go wrong.
Spot-check sampling is the only one of the four human-in-the-loop arrangements that says outright it isn’t looking at everything, which makes it easy to reason about and difficult to defend after the fact. The challenge isn’t that the sample is too small — it’s that sampling math doesn’t always fit well with the kind of error that models make.
The most dangerous AI failures aren’t always hallucinations. Sometimes the agent understands the task perfectly and still executes it incorrectly.
The system clears whatever matches the rules, and a person sees only the exceptions. The judgment didn’t disappear — it moved upstream into a definition nobody owns, which is why a falling escalation rate looks like improvement whether the model got better or anomalies started clearing silently.
The challenge of agentic AI isn’t simply controlling what agents can do — it’s understanding how their decisions interact to produce an outcome.
A person handles every item, so coverage is always 1.0 — which is exactly why that number tells you almost nothing. The arrangement that’s easiest to defend to an examiner is also the easiest to hollow out without a single document changing.
The more autonomous AI becomes, the less we can rely on the system itself to tell us what happened.
One compliance officer reads every alert; another reads a summary of what closed without her. Both arrangements are documented as human-in-the-loop, and the difference between them is an important question.
The biggest takeaway from Anthropic’s latest disclosure wasn’t what the model did. It was how long it went unnoticed.
The tool produces, a person attests, and the systems remember — or they don’t. Part 3 closes the arc: why the signature, not the model, sets the pace for AI in finance.
AI is getting better at producing answers. The next challenge is helping enterprises understand where those answers came from, and what to do when they’re wrong.
Ninety seconds to produce, hours to verify, and no record that the checking ever happened. Part 2 maps the two costs — checking and documenting the check — that decide what AI actually saves finance teams.
Prove AI’s new research and development team is tackling the problems that sit just beyond current tooling: decision provenance, runtime control for multi-agent systems, and human judgment in the loop.
Financial teams don’t just need AI to produce answers. They need to understand how those answers — and every action behind them — came to be.
AI already earns its keep in finance on extraction, summarization, and first drafts. But the two dominant failure modes both produce output that looks correct — which is why everyone still checks every number.
Two very different stories this week — a security audit and a debugging technique — reveal the same shift.
Meta’s slowdown says more about production than intelligence.
Two mechanisms are driving enterprise LLM bills sky-high: unbounded usage (the loud one finance sees) and context bloat (the quiet one nobody’s measuring). One audit attributes 62% of agentic AI spend to re-sent context — and the model never even needs most of it.
The myth that proprietary code is inherently more secure than open source falls apart on inspection. Hidden blueprints aren’t a security posture — they’re a deferred bill, payable when the vendor changes terms, ships late, or leaves you with no path to a patch.
When agents in a multi-agent workflow quietly duplicate work or produce contradictory outputs — and nothing throws an exception — you’re paying the coordination tax.
Reliability multiplies instead of adding — so a chain of individually solid steps can still fail most of the time in production. The decay is a function of architecture, not model quality, and the right composition patterns bend it back.
Traditional observability catches execution failures — exceptions, timeouts, 5xxs. But an agent can execute flawlessly and still return the wrong answer, and that’s the gap standard tracing never sees.































