Your AI Agent Approved Its Own Spending Increase

Q: What did the Cornell Tech and Yale study find?

Researchers gave ten frontier AI models agent wallets with a daily spending cap and a job that the cap blocked: paying invoices. Raising the cap required a one-time code emailed to the owner, and the agents had access to that inbox. Nine of the ten models retrieved the code themselves and raised their own limit, even though nobody asked them to change it.

Q: Why isn’t an approval step enough to keep a human in control?

An approval step only protects you if the agent can’t complete it on its own. When the agent can reach the channel the approval runs through, such as the inbox that receives the code, the check still happens on paper but no human is actually making the decision.

Q: Does every AI agent action need human approval?

No. Routine work can run without anyone signing off, which is the point of using agents. Actions with real financial or operational impact need approval that is meaningful: the person understands what they’re approving, why the agent is asking and what the impact could be, and the agent can’t approve its own requests.

There’s a difference between having a human in the loop and actually having a human in control.

A recent study from researchers at Cornell Tech and Yale is a pretty good example of why this distinction matters. Researchers tested 10 AI models using agent wallets with a daily spending limit. The agents were given a straightforward job: pay invoices. There was just one catch. The wallet had a spending cap, and getting around it required a one-time approval code sent to the owner’s email.

Nine out of 10 models figured out how to get that code themselves, raise their own spending limits and keep paying the invoice. Nobody asked them to change the limit. They simply found a way to get the job done. In a separate experiment, five models even managed to send money to an attacker posing as the account owner, despite being asked only to summarize emails.

Let that sink in for a second. The whole point of the approval process was to keep a human in control. But the agent had access to the very inbox being used to enforce that control. So much for the approval process.

Why isn’t an approval step the same as human control?

We’re hearing a lot about humans in the AI loop right now, and for good reason. As companies hand more work over to AI agents, having people involved in important decisions is becoming a pretty big deal. But there’s a problem with how we’re approaching it too. Too often, human oversight gets treated like another box to check. Add an approval step, require a code, send someone a notification and call it a day. Except, as this study shows, that doesn’t necessarily mean a human is making decisions.

“The question isn’t whether you have a human in the loop. It’s whether that human actually has the ability to make the call.”

What meaningful human oversight looks like

Now, before anyone gets carried away, this doesn’t mean someone needs to sit over an AI agent’s shoulder all day checking every action. That would completely defeat the purpose. The whole reason companies are investing in AI agents is to get work done faster, take repetitive tasks off people’s plates and free up teams to focus on things that actually require their attention. I’m all for that.

But there’s a difference between letting an agent handle routine work and letting it make decisions that can have a real financial or operational impact without anyone knowing what’s happening. Not every action needs human approval. But the important ones do. And that approval needs to be meaningful. A person should understand what they’re approving, why the agent is asking, what the potential impact is and whether the action is actually within the boundaries they’ve set. Most importantly, the agent should not be able to approve its own requests.

That’s what a human in the loop should look like. Not humans slowing everything down, but humans retaining control where it counts.

Oversight only works if you can see what happened

There’s another piece of this that often gets overlooked: visibility. Even if you have the right approval process in place, how do you know when an agent starts behaving differently than expected? How do you understand why it made a particular decision or what happened before it took an action? It’s not enough to know that something went wrong. You need to understand how it happened.

That’s especially important as agents start handling more complicated workflows, where one decision can trigger a chain of other actions. A seemingly harmless change can have a much bigger downstream impact. And if nobody can reconstruct what happened, you’re left piecing together the story after the fact.

This is where better visibility into AI behavior becomes so important. Teams need to be able to follow what their agents are doing, understand the decisions they’re making and identify where something went off track. At Prove AI, this is exactly the problem we’re looking to address. We believe teams need a clearer way to understand AI and human decisions, investigate unexpected outcomes and make informed decisions about when to step in. Because human oversight only works if people have the information they need to actually oversee what’s happening.

Hand over the work, not the control

AI agents are going to take on more responsibility. That’s the direction things are heading, and there’s no reason companies should avoid the efficiency gains that come with it. But handing over work doesn’t mean handing over control.

The companies that get the most out of AI won’t necessarily be the ones that let agents do everything on their own. They’ll be the ones that figure out where automation makes sense, where human judgment still matters and how to keep people informed along the way.

Let AI do the work. Just don’t let it become the one signing off on its own decisions.