AI Agents Can Be Right and Still Break Production

Q: What incidents did Adversa catalogue?

Nine publicly documented cases of AI coding agents deleting production data, wiping files, or taking down live infrastructure — including Replit wiping SaaStr’s production database, Amazon’s Kiro taking Cost Explorer offline for roughly 13 hours, and Claude Code emptying an entire Supabase database. In many of them the agent’s intent wasn’t obviously wrong; the failure happened during execution.

Q: How can an AI agent be “right” and still cause damage?

The objective can be correct and the command can look right — but a path expands differently than expected, a parameter points at the wrong environment, or the agent misreads a tool’s response as success. These are human-class mistakes (shell quoting, tilde expansion, exit-code parsing), made at machine speed, with the agent continuing to operate before anyone notices.

Q: Why does this matter especially for financial workflows?

Because a polished explanation can sit on top of a wrong number: a calculation applied to the wrong range, a duplicated source, a tool call that changed data unexpectedly. Evaluating the final output isn’t enough — teams need to see what data the agent accessed, which tools it called, what changed, what came back, and what the agent did next.

Q: What should teams log to make agent failures fixable?

The commands, not just the output. A record that says “analysis completed successfully” doesn’t help when the analysis is wrong — the useful artifact is the chain: input, decision, tool call, action, result, and what happened next. That chain is what turns an investigation into a fix instead of a reconstruction.

AI agents are getting very good at following instructions… but that might actually be the problem.

A recent analysis from Adversa catalogued nine incidents involving AI coding agents that deleted production data, wiped files, or took down live infrastructure. The incidents included Replit wiping SaaStr’s production database, Amazon’s Kiro taking Cost Explorer offline for roughly 13 hours, and Claude Code emptying an entire Supabase database. In many cases, the agent’s intent wasn’t obviously wrong. The failure happened during execution; in shell commands, parameters, environment assumptions, or how the agent interpreted a tool’s response.

That distinction matters. We tend to talk about AI failures as a reasoning problem: the model misunderstood the request, hallucinated an answer, or made a bad decision. But an agent can understand the assignment and still produce the wrong outcome. The reasoning can look perfectly reasonable while the execution quietly goes off the rails.

The agent can be right and still break everything

Consider an agent asked to clean up a set of files. It identifies the correct files, determines they should be deleted, and generates what appears to be the right command. But a path expands differently than expected, a parameter points to the wrong environment, or the command returns an unexpected result that the agent interprets as success. The objective was correct. The execution wasn’t.

Adversa’s examples point to exactly this kind of failure, including shell quoting, tilde expansion, exit code parsing, and database configuration issues. These aren’t uniquely AI problems; humans make them too. The difference is that an agent can make the same mistake at machine speed and continue operating before anyone realizes what happened.

That becomes considerably more consequential when the system is working with financial data.

Finance doesn’t get a pass because the reasoning looks good

Imagine an AI workflow reviewing a spreadsheet, reconciling accounts, updating a financial model, or preparing a credit exposure analysis. The agent understands the task, identifies the right inputs, and produces a polished explanation. Somewhere in the workflow, however, a calculation is applied to the wrong range, a source is duplicated, or a tool call changes data in an unexpected way.

The final answer can look completely reasonable while the underlying number is wrong.

For financial teams, that means evaluating the final output isn’t enough. You need to understand what happened along the way: what data the agent accessed, which tools it called, what it changed, what those tools returned, and what the agent did next. The question isn’t simply whether the agent reached the right conclusion. It’s whether the system actually executed the work correctly.

The answer isn’t the whole story

This is where the traditional approach to AI evaluation starts to fall short. We can test whether an agent produces the right answer, but that doesn’t necessarily tell us what will happen when the agent interacts with a real environment.

A financial workflow can pass its evaluation and still fail in production because execution introduces variables that aren’t present in a benchmark. A spreadsheet has a different range than expected. A data source changes. A tool returns an unexpected value. A calculator succeeds technically but operates on the wrong inputs.

When that happens, the final output is only the end of the story. The useful artifact is the sequence that produced it.

Adversa makes a simple recommendation that applies well beyond coding agents: log the commands, not just the output. A record saying “analysis completed successfully” doesn’t help much when the analysis is wrong. You need the input, decision, tool call, action, result, and what happened next.

Without the chain, investigating an AI failure becomes an exercise in reconstructing the past from scattered logs and application records. And the longer that takes, the harder it becomes to fix the underlying problem.

The real bottleneck comes after the failure

I think this is where the conversation around AI reliability needs to shift. We shouldn’t expect agents to never fail. As they become more autonomous, failures are inevitable. The more important question is how quickly a team can understand and resolve one.

Can you determine exactly what the agent did? Can you distinguish a bad decision from a bad execution? Can you find out whether the same failure has happened before? And once you’ve identified the cause, can you actually fix it without starting the investigation from scratch?

That’s the difference between visibility and remediation. Visibility tells you something happened. A trace helps explain why. Remediation gets you from that explanation to a fix.

For financial workflows, that distinction is especially important. When an agent produces the wrong number, the goal isn’t to spend a day proving that the number is wrong. It’s to understand how it became wrong, correct the underlying issue, and make sure the same failure doesn’t happen again.

AI doesn’t have to be perfect. It has to be fixable

The promise of agents isn’t that they’ll never make mistakes. It’s that they’ll be able to do more work without requiring a human to supervise every step.

That only works if teams can recover quickly when something goes wrong.

The incidents we’re seeing now suggest that the next challenge isn’t simply making agents smarter. It’s making their actions understandable after the fact. An agent doesn’t need to hallucinate or go rogue to cause damage. Sometimes it just needs to execute one reasonable instruction in an unexpected way.

When that happens, the most valuable record isn’t the final answer. It’s the chain of actions that explains how the system got there and gives someone a clear path to fix it.